Closed jonathan-gramain closed 1 year ago
My role is to assist you with the merge of this
pull request. Please type @bert-e help
to get information
on this process, or consult the user documentation.
Status report is not available.
A conflict has been raised during the creation of
integration branch w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
with contents from bugfix/BB-419-increaseNotificationDefaultConcurrency
and development/7.70
.
I have not created the integration branch.
Here are the steps to resolve this conflict:
$ git fetch
$ git checkout -B w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency origin/development/7.70
$ git merge origin/bugfix/BB-419-increaseNotificationDefaultConcurrency
$ # <intense conflict resolution>
$ git commit
$ git push -u origin w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
The Fix Version/s
in issue BB-419 contains:
7.10.12
7.70.3
8.4.26
8.5.5
8.6.23
Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:
7.10.12
7.70.3
8.5.5
8.6.26
8.7.0
Please check the Fix Version/s
of BB-419, or the target
branch of this pull request.
A conflict has been raised during the creation of
integration branch w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
with contents from bugfix/BB-419-increaseNotificationDefaultConcurrency
and development/7.70
.
I have not created the integration branch.
Here are the steps to resolve this conflict:
$ git fetch
$ git checkout -B w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency origin/development/7.70
$ git merge origin/bugfix/BB-419-increaseNotificationDefaultConcurrency
$ # <intense conflict resolution>
$ git commit
$ git push -u origin w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
A conflict has been raised during the creation of
integration branch w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency
with contents from w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
and development/8.5
.
I have not created the integration branch.
Here are the steps to resolve this conflict:
$ git fetch
$ git checkout -B w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency origin/development/8.5
$ git merge origin/w/7.70/bugfix/BB-419-increaseNotificationDefaultConcurrency
$ # <intense conflict resolution>
$ git commit
$ git push -u origin w/8.5/bugfix/BB-419-increaseNotificationDefaultConcurrency
I have created the integration data for the additional destination branches.
bugfix/BB-419-increaseNotificationDefaultConcurrency
into
development/7.10
development/7.70
development/8.5
development/8.6
development/8.7
The following branches will NOT be impacted:
development/7.4
You can set option create_pull_requests
if you need me to create
integration pull requests in addition to integration branches, with:
@bert-e create_pull_requests
The following approvals are needed before I can proceed with the merge:
the author
2 peers
Merge commit #1e5dae878ab6d5cf220e8460791d515480d132c1 on the integration branch
w/8.6/bugfix/BB-419-increaseNotificationDefaultConcurrency
is merging a branch which is neither the current
branch bugfix/BB-419-increaseNotificationDefaultConcurrency
nor the development branch
development/8.6
.
It is likely due to a rebase of the branch bugfix/BB-419-increaseNotificationDefaultConcurrency
and the
merge is not possible until all related w/*
branches are deleted or updated.
Please use the reset
command to have me reinitialize these branches.
The following approvals are needed before I can proceed with the merge:
the author
2 peers
/approve
The changeset has received all authorizations and has been added to the relevant queue(s). The queue(s) will be merged in the target development branch(es) as soon as builds have passed.
The changeset will be merged in:
:heavy_check_mark: development/7.10
:heavy_check_mark: development/7.70
:heavy_check_mark: development/8.5
:heavy_check_mark: development/8.6
:heavy_check_mark: development/8.7
The following branches will NOT be impacted:
development/7.4
There is no action required on your side. You will be notified here once the changeset has been merged. In the unlikely event that the changeset fails permanently on the queue, a member of the admin team will contact you to help resolve the matter.
IMPORTANT
Please do not attempt to modify this pull request.
If you need this pull request to be removed from the queue, please contact a member of the admin team now.
The following options are set: approve
I have successfully merged the changeset of this pull request into targetted development branches:
:heavy_check_mark: development/7.10
:heavy_check_mark: development/7.70
:heavy_check_mark: development/8.5
:heavy_check_mark: development/8.6
:heavy_check_mark: development/8.7
The following branches have NOT changed:
development/7.4
Please check the status of the associated issue BB-419.
Goodbye jonathan-gramain.
Replaces https://github.com/scality/backbeat/pull/2421
increase default notification BackbeatConsumer concurrency
Notification QueueProcessor was limited by a concurrency of 10, which meant it could only wait for 10 delivery reports from the producer before consuming more entries.
Increased the default concurrency to 1000 to improve throughput and add a test to show that it behaves well with this number of pending reports (the processing does not involve anything else asynchronous than waiting for delivery reports), as well as a few other extra tests.
Also fix and improve logging related to processing errors and error reporting from delivery reports.
[cleanup] BackbeatProducer.pollIntervalMs default as constant
Use a code constant for
pollIntervalMs
, and add a JSDoc about it (andmessageMaxBytes
)[impr] configurable notif Kafka poll interval
Make the external Kafka producer poll interval configurable: while the default is 2000ms, it could be helpful to reduce its value, e.g. to reduce latency of delivery in case the external Kafka cluster cannot receive all notifications in time.
It's not clear at this point what would be a better value due to lack of testing to the limits, and 2000ms doesn't look unreasonable as a trade-off between latency and polling overhead, so leaving the default value as it is. It is however important to have a concurrency value high enough to not limit the possible throughput per notification processor, which is limited by "concurrency / poll_interval_seconds" messages per second. The concurrency has been raised to 1000 by default which therefore allows 500 messages per second to go through per processor with a 2000ms poll interval.