Re-upload files if they are missing in storage

rzvoncek commented 9 months ago

Fixes #709 Fixes #368

In short, the way we do this is to list all files in storage, in the data folder for the given prefix and node, after doing a snapshot but before we start backing up individual tables. We we then group the listed files into a dict->dict->set keyed by keyspace->table, so that we can look up any given file efficiently.

Tested manually on a node with ~11k LCS SSTables. Did not observe a particular increase in (differential) backup duration. There probably is an increased memory consumption, but I'm struggling a bit to quantify how much.

codecov[bot] commented 9 months ago

Codecov Report

Attention: Patch coverage is 97.34513% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 80.02%. Comparing base (49f88ea) to head (64f9eb6).

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716/graphs/tree.svg?width=650&height=150&src=pr&token=KTDCRD82NU&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) ```diff @@ Coverage Diff @@ ## master #716 +/- ## ========================================== - Coverage 80.17% 80.02% -0.15% ========================================== Files 52 52 Lines 4726 4751 +25 Branches 961 970 +9 ========================================== + Hits 3789 3802 +13 - Misses 908 920 +12 Partials 29 29 ``` | [Files](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle) | Coverage Δ | | |---|---|---| | [medusa/backup\_node.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL2JhY2t1cF9ub2RlLnB5) | `87.00% <100.00%> (-1.08%)` | :arrow_down: | | [medusa/storage/abstract\_storage.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvYWJzdHJhY3Rfc3RvcmFnZS5weQ==) | `92.36% <100.00%> (+0.03%)` | :arrow_up: | | [medusa/storage/azure\_storage.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvYXp1cmVfc3RvcmFnZS5weQ==) | `99.15% <100.00%> (+<0.01%)` | :arrow_up: | | [medusa/storage/google\_storage.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvZ29vZ2xlX3N0b3JhZ2UucHk=) | `89.55% <100.00%> (-4.44%)` | :arrow_down: | | [medusa/storage/local\_storage.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvbG9jYWxfc3RvcmFnZS5weQ==) | `98.03% <100.00%> (+0.01%)` | :arrow_up: | | [medusa/storage/s3\_base\_storage.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvczNfYmFzZV9zdG9yYWdlLnB5) | `93.85% <100.00%> (+0.02%)` | :arrow_up: | | [medusa/storage/s3\_rgw.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvczNfcmd3LnB5) | `75.00% <75.00%> (+3.57%)` | :arrow_up: | | [medusa/storage/\_\_init\_\_.py](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle#diff-bWVkdXNhL3N0b3JhZ2UvX19pbml0X18ucHk=) | `94.56% <94.44%> (-0.02%)` | :arrow_down: | ... and [4 files with indirect coverage changes](https://app.codecov.io/gh/thelastpickle/cassandra-medusa/pull/716/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=thelastpickle)

rzvoncek commented 8 months ago

Finalised the reimplementation to not even look at the manifests. DId a basic manual test with corrupting a file (rm then touch it).

rzvoncek commented 8 months ago

Pushed a commit with the fixes. Also rebased on master to fix merge conflicts.