p2panda / aquadoggo

Node for the p2panda network handling validation, storage, aggregation and replication
GNU Affero General Public License v3.0
70 stars 5 forks source link

Build a byte buffer over paginated pieces when assembling blobs #547

Closed adzialocha closed 1 year ago

adzialocha commented 1 year ago

This removes the maximum limit of allowed blob pieces (for now, we might want to have it back with some sort of configuration interface). This touched upon the pagination logic used to iterate over all pieces, so I decided to just paginate through it in a classic manner, page by page.

On top this uses now the BytesMut struct to efficiently assemble the blob pieces. For the future we might want to do this instead https://github.com/p2panda/aquadoggo/issues/548 to avoid keeping very large blobs in memory.

Closes: #546

📋 Checklist

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 90.62% and no project coverage change.

Comparison is base (edc0b30) 92.50% compared to head (bbec0eb) 92.50%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## http-blob-router #547 +/- ## ================================================= Coverage 92.50% 92.50% ================================================= Files 105 105 Lines 17932 17939 +7 ================================================= + Hits 16588 16595 +7 Misses 1344 1344 ``` | [Files Changed](https://app.codecov.io/gh/p2panda/aquadoggo/pull/547?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda) | Coverage Δ | | |---|---|---| | [aquadoggo/src/db/stores/blob.rs](https://app.codecov.io/gh/p2panda/aquadoggo/pull/547?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda#diff-YXF1YWRvZ2dvL3NyYy9kYi9zdG9yZXMvYmxvYi5ycw==) | `97.42% <90.62%> (+0.04%)` | :arrow_up: | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/p2panda/aquadoggo/pull/547/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=p2panda)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.