Use 32 bit histogram values

akoshelev commented 1 month ago

This addresses #1100 by increasing number of bits in histogram values to 32, supporting up to ~500M inputs with 2 bit trigger values (or 200M with 3 bit values)

akoshelev commented 1 month ago

draft run (20M): https://draft-mpc.vercel.app/query/view/flip-value2024-06-14T1748

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 84.21053% with 3 lines in your changes missing coverage. Please review.

Project coverage is 92.05%. Comparing base (11ee19d) to head (b53caf9).

Files	Patch %	Lines
ipa-core/src/protocol/ipa_prf/aggregation/mod.rs	75.00%	2 Missing :warning:
ipa-core/src/query/executor.rs	85.71%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1140 +/- ## ========================================== + Coverage 92.04% 92.05% +0.01% ========================================== Files 196 196 Lines 29320 29326 +6 ========================================== + Hits 26987 26997 +10 + Misses 2333 2329 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

akoshelev commented 1 month ago

All right, I think it is working but takes 50% more time to finish, which is expected because we doubled the number of bits used in aggregation. We quickly get to 16 bits I think and after that we start paying the price of adding more bits while previously we just saturated at 16

But it is clearly working as histogram values are greater than $2^{16}$

41 | Some(421047) | Some(421047)

akoshelev commented 4 weeks ago

We are investigating an issue where 32 bit breakdowns may(?) cause protocols to be unnecessary slow. #1151. It does not seem to be the case, but I want to be absolutely sure before I can merge this PR

private-attribution / ipa

Use 32 bit histogram values #1140

Codecov Report