p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
69 stars 5 forks source link

Garbage collect blobs from file system #571

Closed sandreae closed 11 months ago

sandreae commented 11 months ago

We already garbage collect unused blob views from the database, this PR implements garbage collecting blob views from the file system as well. This occurs both when a blob view is not the current view or pinned from another document, and for the current view as well when the blob document itself is not related to by any other document.

During implementation I did a little general renaming/refactoring in the garbage collection task to help code comprehension.

πŸ“‹ Checklist

codecov[bot] commented 11 months ago

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (3ac8b70) 92.43% compared to head (93ba503) 92.47%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #571 +/- ## ========================================== + Coverage 92.43% 92.47% +0.03% ========================================== Files 106 106 Lines 18164 18396 +232 ========================================== + Hits 16790 17011 +221 - Misses 1374 1385 +11 ``` | [Files](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda) | Coverage Ξ” | | |---|---|---| | [aquadoggo/src/db/stores/blob.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi9zdG9yZXMvYmxvYi5ycw==) | `98.18% <100.00%> (+<0.01%)` | :arrow_up: | | [aquadoggo/src/db/stores/document.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi9zdG9yZXMvZG9jdW1lbnQucnM=) | `98.76% <100.00%> (-0.07%)` | :arrow_down: | | [aquadoggo/src/materializer/tasks/reduce.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9tYXRlcmlhbGl6ZXIvdGFza3MvcmVkdWNlLnJz) | `94.92% <0.00%> (ΓΈ)` | | | [aquadoggo/src/test\_utils/node.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy90ZXN0X3V0aWxzL25vZGUucnM=) | `95.00% <97.29%> (+0.35%)` | :arrow_up: | | [...doggo/src/materializer/tasks/garbage\_collection.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9tYXRlcmlhbGl6ZXIvdGFza3MvZ2FyYmFnZV9jb2xsZWN0aW9uLnJz) | `97.80% <95.40%> (-0.83%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/p2panda/aquadoggo/pull/571/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

sandreae commented 11 months ago

Very concise useful Postgres error in the CI: Error: The operation was canceled.