private-attribution / ipa

A raw implementation of Interoperable Private Attribution
MIT License
39 stars 23 forks source link

Split large request bodies into chunks of size not exceeding 1 MB #1169

Closed akoshelev closed 1 week ago

akoshelev commented 3 weeks ago

This fixes #1141 and adds a test that verifies that clients can submit inputs larger than 2GB ($2^{31}$ - 1). The test is quite slow (40 seconds), so I gated it behind an environment variable and only run it on Github. The probability of breaking it is quite low, so it is fine if it fails on CI only.

IPA uses two different stream types for in-memory and real-world infrastructure to submit bodies - one is Axum type and another one is just boxed byte stream. The fix must be applied to both because report_collector uses in-memory streams and helper binary and integration tests use Axum stream.

In order to have only one place that splits the incoming byte, both streams had to be hidden behind an umbrella type BodyStream that works for both.

akoshelev commented 2 weeks ago

Why do we go the route of having a BodyStream wrapper over BodyStreamInner?

Slicing must be done for both in-memory and axum streams, so BodyStream is where it is happening in one place. I could've implemented it twice separately, but I didn't see any benefit of duplicating the logic. Also, it is useful to have it done transparently for clients, so we don't need to worry about how the streams are constructed - chunking will apply regardless.

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 94.59459% with 4 lines in your changes missing coverage. Please review.

Project coverage is 92.03%. Comparing base (a734d04) to head (9c7c2b3). Report is 42 commits behind head on main.

Files Patch % Lines
ipa-core/src/helpers/transport/stream/mod.rs 95.52% 3 Missing :warning:
ipa-core/src/helpers/transport/stream/box_body.rs 50.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1169 +/- ## ========================================== + Coverage 91.89% 92.03% +0.13% ========================================== Files 195 196 +1 Lines 28939 29303 +364 ========================================== + Hits 26594 26968 +374 + Misses 2345 2335 -10 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.