borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
11.24k stars 744 forks source link

repo-compress: reduce memory consumption #8543

Closed ThomasWaldmann closed 1 week ago

ThomasWaldmann commented 1 week ago

Worst (but frequent) case here is that all or most of the chunks in the repo need to get recompressed, thus storing all chunk ids in a python list would need significant amounts of memory for large repositories.

We already have all chunk ids stored in cache.chunks, so we now just flag the ones needing re-compression by setting the F_COMPRESS flag (that does not need any additional memory).

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 81.79%. Comparing base (295f4e3) to head (f7dea6e). Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
src/borg/archiver/repo_compress_cmd.py 91.66% 0 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #8543 +/- ## ======================================= Coverage 81.79% 81.79% ======================================= Files 74 74 Lines 13290 13291 +1 Branches 1956 1957 +1 ======================================= + Hits 10870 10871 +1 Misses 1756 1756 Partials 664 664 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.