linkedin / ambry

Distributed object store
https://github.com/linkedin/ambry/wiki
Apache License 2.0
1.74k stars 275 forks source link

Issue force delete for 503 delete operations #2825

Closed justinlin-linkedin closed 1 month ago

justinlin-linkedin commented 1 month ago

Summary

Issue a force delete request when the delete request is getting 503.

Details

When originating replicas returns more than 1 unavailable to frontend for delete operations, we have to return 503 back to client since we can't be sure about the status of the blob. If we have two unavailable hosts, (due to any reason, server busy, server unavailable, etc), it's impossible to know if this blob exists on these two hosts or not. In this case, we have to return 503 so client can retry later.

However, when executing delete, in most cases, we know it would be fine to place a delete tombstone to available replicas and just return 200 back to client, since the delete is the final state.

There are some catches here. If the cluster enables undelete, we might have some unavailable replicas that have blob at certain lifeVersion. A force delete would place a delete tombstone with life version 0, which will be overritten by the higher life version later.

Test

Unit tests.

codecov-commenter commented 1 month ago

Codecov Report

Attention: Patch coverage is 80.55556% with 7 lines in your changes missing coverage. Please review.

Project coverage is 70.24%. Comparing base (52ba813) to head (a65d74e). Report is 51 commits behind head on master.

Files Patch % Lines
.../java/com/github/ambry/router/DeleteOperation.java 81.48% 0 Missing and 5 partials :warning:
...ava/com/github/ambry/router/NonBlockingRouter.java 60.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #2825 +/- ## ============================================ + Coverage 64.24% 70.24% +5.99% - Complexity 10398 11783 +1385 ============================================ Files 840 844 +4 Lines 71755 72444 +689 Branches 8611 8720 +109 ============================================ + Hits 46099 50885 +4786 + Misses 23004 18889 -4115 - Partials 2652 2670 +18 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.