search

scality / backbeat

Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.
https://www.zenko.io
Apache License 2.0
53 stars 19 forks source link

BB-355 use updateDescription field to get object metadata in kafka lo… #2360

Closed Kerkesni closed 1 year ago

Kerkesni commented 1 year ago

…g consumer

Issue: BB-355

Problem: Some times oplog update events we get when using the Kafka log consumer don't have any metadata. This mostly happens when deleting the object right after the update.

The issue happens because we use the fullDocument field to retrieve the metadata, when reading the mongodb docs we see that the field represents a copy of the entire updated document at a point in time after the update, which means that in some cases this value can be unavailable like when deleting right after the update.

Fix: Use the updateDescription field to get object metadata instead of fullDocument. Values from updateDescription represent the exact changes that occurred to the object. It is possible to have non complete metadata in this field but this shouldn't be an issue as we update the full object each time in the mongoClient. One key updates like we do in the update we perform before each deletion will be ignored as we don't even have the value field that's supposed to contain the metadata.

bert-e commented 1 year ago

Hello kerkesni,

My role is to assist you with the merge of this pull request. Please type @bert-e help to get information on this process, or consult the user documentation.

Status report is not available.

bert-e commented 1 year ago

Incorrect fix version

The Fix Version/s in issue BB-355 contains:

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

Please check the Fix Version/s of BB-355, or the target branch of this pull request.

bert-e commented 1 year ago

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests
bert-e commented 1 year ago

Waiting for approval

The following approvals are needed before I can proceed with the merge:

bert-e commented 1 year ago

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e commented 1 year ago

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests
bert-e commented 1 year ago

Waiting for approval

The following approvals are needed before I can proceed with the merge:

Kerkesni commented 1 year ago

/reset

bert-e commented 1 year ago

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e commented 1 year ago

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests
bert-e commented 1 year ago

Waiting for approval

The following approvals are needed before I can proceed with the merge:

bert-e commented 1 year ago

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e commented 1 year ago

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

You can set option create_pull_requests if you need me to create integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests
bert-e commented 1 year ago

Waiting for approval

The following approvals are needed before I can proceed with the merge:

Kerkesni commented 1 year ago

/approve

bert-e commented 1 year ago

In the queue

The changeset has received all authorizations and has been added to the relevant queue(s). The queue(s) will be merged in the target development branch(es) as soon as builds have passed.

The changeset will be merged in:

The following branches will NOT be impacted:

There is no action required on your side. You will be notified here once the changeset has been merged. In the unlikely event that the changeset fails permanently on the queue, a member of the admin team will contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

If you need this pull request to be removed from the queue, please contact a member of the admin team now.

The following options are set: approve

bert-e commented 1 year ago

I have successfully merged the changeset of this pull request into targetted development branches:

The following branches have NOT changed:

Please check the status of the associated issue BB-355.

Goodbye kerkesni.