bf: BB-419 increase notification default concurrency

[jonathan-gramain]

Replaces https://github.com/scality/backbeat/pull/2421

• increase default notification BackbeatConsumer concurrency

  Notification QueueProcessor was limited by a concurrency of 10, which meant it could only wait for 10 delivery reports from the producer before consuming more entries.

  Increased the default concurrency to 1000 to improve throughput and add a test to show that it behaves well with this number of pending reports (the processing does not involve anything else asynchronous than waiting for delivery reports), as well as a few other extra tests.

  Also fix and improve logging related to processing errors and error reporting from delivery reports.

• [cleanup] BackbeatProducer.pollIntervalMs default as constant

  Use a code constant for pollIntervalMs, and add a JSDoc about it (and messageMaxBytes)

• [impr] configurable notif Kafka poll interval

  Make the external Kafka producer poll interval configurable: while the default is 2000ms, it could be helpful to reduce its value, e.g. to reduce latency of delivery in case the external Kafka cluster cannot receive all notifications in time.

  It's not clear at this point what would be a better value due to lack of testing to the limits, and 2000ms doesn't look unreasonable as a trade-off between latency and polling overhead, so leaving the default value as it is. It is however important to have a concurrency value high enough to not limit the possible throughput per notification processor, which is limited by "concurrency / poll_interval_seconds" messages per second. The concurrency has been raised to 1000 by default which therefore allows 500 messages per second to go through per processor with a 2000ms poll interval.

[bert-e]

Hello jonathan-gramain,

My role is to assist you with the merge of this pull request. Please type @bert-e help to get information on this process, or consult the user documentation.

Status report is not available.

[bert-e]

Conflict

A conflict has been raised during the creation of integration branch w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency with contents from bugfix/BB-419-increaseNotificationDefaultConcurrency and development/7.70.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency origin/development/7.70
 $ git merge origin/bugfix/BB-419-increaseNotificationDefaultConcurrency
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency

[bert-e]

Incorrect fix version

The Fix Version/s in issue BB-419 contains:

• 7.10.12

• 7.70.3

• 8.4.26

• 8.5.5

• 8.6.23

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

• 7.10.12

• 7.70.3

• 8.5.5

• 8.6.26

• 8.7.0

Please check the Fix Version/s of BB-419, or the target branch of this pull request.

[bert-e]

Conflict

A conflict has been raised during the creation of integration branch w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency with contents from bugfix/BB-419-increaseNotificationDefaultConcurrency and development/7.70.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency origin/development/7.70
 $ git merge origin/bugfix/BB-419-increaseNotificationDefaultConcurrency
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency

[bert-e]

Conflict

A conflict has been raised during the creation of integration branch w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency with contents from w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency and development/8.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency origin/development/8.5
 $ git merge origin/w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency

[bert-e]

Integration data created

I have created the integration data for the additional destination branches.

• this pull request will merge bugfix/BB-419-increaseNotificationDefaultConcurrency into development/7.10
• w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency will be merged into development/7.70
• w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency will be merged into development/8.5
• w/8.6/bugfix/BB-419-increaseNotificationDefaultConcurrency will be merged into development/8.6
• w/8.7/bugfix/BB-419-increaseNotificationDefaultConcurrency will be merged into development/8.7

The following branches will NOT be impacted:

• development/7.4

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[bert-e]

History mismatch

Merge commit #1e5dae878ab6d5cf220e8460791d515480d132c1 on the integration branch w/8.6/bugfix/BB-419-increaseNotificationDefaultConcurrency is merging a branch which is neither the current branch bugfix/BB-419-increaseNotificationDefaultConcurrency nor the development branch development/8.6.

It is likely due to a rebase of the branch bugfix/BB-419-increaseNotificationDefaultConcurrency and the merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[jonathan-gramain]

/approve

[bert-e]

In the queue

The changeset has received all authorizations and has been added to the relevant queue(s). The queue(s) will be merged in the target development branch(es) as soon as builds have passed.

The changeset will be merged in:

• :heavy_check_mark: development/7.10

• :heavy_check_mark: development/7.70

• :heavy_check_mark: development/8.5

• :heavy_check_mark: development/8.6

• :heavy_check_mark: development/8.7

The following branches will NOT be impacted:

• development/7.4

There is no action required on your side. You will be notified here once the changeset has been merged. In the unlikely event that the changeset fails permanently on the queue, a member of the admin team will contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

• Any commit you add on the source branch will trigger a new cycle after the current queue is merged.
• Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a member of the admin team now.

The following options are set: approve

[bert-e]

I have successfully merged the changeset of this pull request into targetted development branches:

• :heavy_check_mark: development/7.10

• :heavy_check_mark: development/7.70

• :heavy_check_mark: development/8.5

• :heavy_check_mark: development/8.6

• :heavy_check_mark: development/8.7

The following branches have NOT changed:

• development/7.4

Please check the status of the associated issue BB-419.

Goodbye jonathan-gramain.