palantir / atlasdb

Transactional Distributed Database Layer
https://palantir.github.io/atlasdb/
Apache License 2.0
55 stars 14 forks source link

Track and kill background sweep after targeted is enabled #3787

Open tpetracca opened 5 years ago

tpetracca commented 5 years ago

My understanding of sweep things:

FR: Actually track the above and disable background sweep on tables who have successfully [background] swept at least once since TS was enabled*

* I guess there are some edge cases here. Like if you transitions from CONSERVATIVE to THOROUGH you'd want to re-enable TS and let it do 1 run.

sandorw commented 5 years ago

I believe that we no longer update the priority tables for background sweep, meaning that once it finishes one final pass through all tables, it will never sweep them again. We could certainly do the final step here (although it's more work) to persist that fact and no longer spin up background sweep threads, but we're most of the way there already.

This won't prevent problems where there are large tables that have never been successfully fully swept - as is now, background sweep will always make one final pass through them (which is desirable in the abstract).