scality / backbeat

Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.
https://www.zenko.io
Apache License 2.0
53 stars 19 forks source link

simplify handling of replication failure notification #2502

Closed Kerkesni closed 7 months ago

Kerkesni commented 7 months ago

Instead of duplicating the notification logic we set the correct originOp when updating the metadata to trigger the replication failure notification. (Bucket notification listens to the oplog and sends notification based on the originOp field in the metadata)

At the end of a failed replication we update the metadata with the failure status that ends up in the oplog topic. So this does not add any overhead in terms of extra Kafka messages being pushed to any of the topics.

This also fixes the duplicate notifications bug where we send as many notifications as destinations we have in case they use the same internal topic (bug already fixed in notification code).

Issue: BB-507

bert-e commented 7 months ago

Hello kerkesni,

My role is to assist you with the merge of this pull request. Please type @bert-e help to get information on this process, or consult the user documentation.

Available options name | description | privileged | authored ------ | ------------ | ------------ |---------- `/after_pull_request` | Wait for the given pull request id to be merged before continuing with the current one. | | `/bypass_author_approval` | Bypass the pull request author's approval | :star: | `/bypass_build_status` | Bypass the build and test status | :star: | `/bypass_commit_size` | Bypass the check on the size of the changeset ```TBA``` | :star: | `/bypass_incompatible_branch` | Bypass the check on the source branch prefix | :star: | `/bypass_jira_check` | Bypass the Jira issue check | :star: | `/bypass_peer_approval` | Bypass the pull request peers' approval | :star: | `/bypass_leader_approval` | Bypass the pull request leaders' approval | :star: | `/approve` | Instruct Bert-E that the author has approved the pull request. | | :writing_hand: `/create_pull_requests` | Allow the creation of integration pull requests. | | `/create_integration_branches` | Allow the creation of integration branches. | | `/no_octopus` | Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead | | `/unanimity` | Change review acceptance criteria from `one reviewer at least` to `all reviewers` | | `/wait` | Instruct Bert-E not to run until further notice. | |
Available commands name | description | privileged ------ | ------------ | ------------ `/help` | Print Bert-E's manual in the pull request. | `/status` | Print Bert-E's current status in the pull request ```TBA``` | `/clear` | Remove all comments from Bert-E from the history ```TBA``` | `/retry` | Re-start a fresh build ```TBA``` | `/build` | Re-start a fresh build ```TBA``` | `/force_reset` | Delete integration branches & pull requests, and restart merge process from the beginning. | `/reset` | Try to remove integration branches unless there are commits on them which do not appear on the source branch. |

Status report is not available.

bert-e commented 7 months ago

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically create the integration branches.

codecov[bot] commented 7 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 63.70%. Comparing base (1e4a5c0) to head (accbde0).

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/scality/backbeat/pull/2502/graphs/tree.svg?width=650&height=150&src=pr&token=LlNB6ANurP&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality)](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | [Files](https://app.codecov.io/gh/scality/backbeat/pull/2502?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | Coverage Δ | | |---|---|---| | [...ensions/notification/NotificationQueuePopulator.js](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#diff-ZXh0ZW5zaW9ucy9ub3RpZmljYXRpb24vTm90aWZpY2F0aW9uUXVldWVQb3B1bGF0b3IuanM=) | `97.22% <100.00%> (+0.12%)` | :arrow_up: | | [extensions/notification/constants.js](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#diff-ZXh0ZW5zaW9ucy9ub3RpZmljYXRpb24vY29uc3RhbnRzLmpz) | `100.00% <ø> (ø)` | | | [...ationStatusProcessor/ReplicationStatusProcessor.js](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#diff-ZXh0ZW5zaW9ucy9yZXBsaWNhdGlvbi9yZXBsaWNhdGlvblN0YXR1c1Byb2Nlc3Nvci9SZXBsaWNhdGlvblN0YXR1c1Byb2Nlc3Nvci5qcw==) | `67.72% <ø> (+4.96%)` | :arrow_up: | | [...ons/replication/replicationStatusProcessor/task.js](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#diff-ZXh0ZW5zaW9ucy9yZXBsaWNhdGlvbi9yZXBsaWNhdGlvblN0YXR1c1Byb2Nlc3Nvci90YXNrLmpz) | `0.00% <ø> (ø)` | | | [...sions/replication/tasks/UpdateReplicationStatus.js](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#diff-ZXh0ZW5zaW9ucy9yZXBsaWNhdGlvbi90YXNrcy9VcGRhdGVSZXBsaWNhdGlvblN0YXR1cy5qcw==) | `78.37% <ø> (-2.16%)` | :arrow_down: | | [lib/models/ObjectQueueEntry.js](https://app.codecov.io/gh/scality/backbeat/pull/2502?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#diff-bGliL21vZGVscy9PYmplY3RRdWV1ZUVudHJ5Lmpz) | `89.85% <ø> (+1.44%)` | :arrow_up: | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/scality/backbeat/pull/2502/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | [Components](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | Coverage Δ | | |---|---|---| | [Bucket Notification](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `66.11% <100.00%> (+0.21%)` | :arrow_up: | | [Core Library](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `62.50% <ø> (+0.03%)` | :arrow_up: | | [Ingestion](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `68.61% <ø> (-0.63%)` | :arrow_down: | | [Lifecycle](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `73.15% <ø> (ø)` | | | [Oplog Populator](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `75.90% <ø> (ø)` | | | [Replication](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `57.22% <ø> (-0.26%)` | :arrow_down: | | [Bucket Scanner](https://app.codecov.io/gh/scality/backbeat/pull/2502/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `85.60% <ø> (ø)` | | ```diff @@ Coverage Diff @@ ## development/8.6 #2502 +/- ## =================================================== - Coverage 63.74% 63.70% -0.05% =================================================== Files 192 192 Lines 12821 12743 -78 =================================================== - Hits 8173 8118 -55 + Misses 4638 4615 -23 Partials 10 10 ``` | [Flag](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | Coverage Δ | | |---|---|---| | [bucket-scanner](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `85.60% <ø> (ø)` | | | [ingestion](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `12.57% <0.00%> (+0.02%)` | :arrow_up: | | [lib](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `7.62% <0.00%> (+0.04%)` | :arrow_up: | | [lifecycle](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `19.51% <0.00%> (+0.12%)` | :arrow_up: | | [notification](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `0.89% <0.00%> (+<0.01%)` | :arrow_up: | | [replication](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `19.15% <0.00%> (-0.32%)` | :arrow_down: | | [unit](https://app.codecov.io/gh/scality/backbeat/pull/2502/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality) | `41.74% <100.00%> (-0.06%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scality#carryforward-flags-in-the-pull-request-comment) to find out more.
bert-e commented 7 months ago

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e commented 7 months ago

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically create the integration branches.

Kerkesni commented 7 months ago

/approve

bert-e commented 7 months ago

History mismatch

Merge commit #7bf30e15e1c854a162e1feeb593d4fce0841588c on the integration branch w/8.7/bugfix/BB-507 is merging a branch which is neither the current branch bugfix/BB-507 nor the development branch development/8.7.

It is likely due to a rebase of the branch bugfix/BB-507 and the merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: approve

Kerkesni commented 7 months ago

/reset

bert-e commented 7 months ago

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: approve

bert-e commented 7 months ago

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e commented 7 months ago

I have successfully merged the changeset of this pull request into targetted development branches:

The following branches have NOT changed:

Please check the status of the associated issue BB-507.

Goodbye kerkesni.

The following options are set: approve