p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
69 stars 5 forks source link

Stream blob data in chunks to files to not occupy too much memory #551

Closed adzialocha closed 1 year ago

adzialocha commented 1 year ago

Establishes a stream from reading binary blob data from the database (via pagination) into writing the file on the file system. This allows for (parallely) handling very large blobs without occupying too much of memory.

Closes: https://github.com/p2panda/aquadoggo/issues/548

📋 Checklist

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 91.28% and project coverage change: +0.04% :tada:

Comparison is base (a404d0c) 92.50% compared to head (77df667) 92.54%. Report is 3 commits behind head on development.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## development #551 +/- ## =============================================== + Coverage 92.50% 92.54% +0.04% =============================================== Files 105 105 Lines 17932 17988 +56 =============================================== + Hits 16588 16647 +59 + Misses 1344 1341 -3 ``` | [Files Changed](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda) | Coverage Δ | | |---|---|---| | [aquadoggo/src/db/stores/entry.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi9zdG9yZXMvZW50cnkucnM=) | `99.40% <ø> (ø)` | | | [aquadoggo/src/db/stores/operation.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi9zdG9yZXMvb3BlcmF0aW9uLnJz) | `89.63% <ø> (ø)` | | | [aquadoggo/src/db/stores/schema.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi9zdG9yZXMvc2NoZW1hLnJz) | `98.37% <ø> (ø)` | | | [aquadoggo/src/db/types/entry.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi90eXBlcy9lbnRyeS5ycw==) | `90.00% <ø> (ø)` | | | [aquadoggo/src/db/types/operation.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi90eXBlcy9vcGVyYXRpb24ucnM=) | `88.00% <ø> (ø)` | | | [aquadoggo/src/graphql/queries/collection.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9ncmFwaHFsL3F1ZXJpZXMvY29sbGVjdGlvbi5ycw==) | `99.60% <ø> (ø)` | | | [...ggo/src/graphql/scalars/document\_view\_id\_scalar.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9ncmFwaHFsL3NjYWxhcnMvZG9jdW1lbnRfdmlld19pZF9zY2FsYXIucnM=) | `78.57% <ø> (ø)` | | | [aquadoggo/src/materializer/tasks/dependency.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9tYXRlcmlhbGl6ZXIvdGFza3MvZGVwZW5kZW5jeS5ycw==) | `97.56% <ø> (ø)` | | | [...doggo/src/materializer/tasks/garbage\_collection.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9tYXRlcmlhbGl6ZXIvdGFza3MvZ2FyYmFnZV9jb2xsZWN0aW9uLnJz) | `98.59% <ø> (ø)` | | | [aquadoggo/src/network/service.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9uZXR3b3JrL3NlcnZpY2UucnM=) | `30.03% <ø> (ø)` | | | ... and [18 more](https://app.codecov.io/gh/p2panda/aquadoggo/pull/551?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda) | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.