search

private-attribution / ipa

A raw implementation of Interoperable Private Attribution
MIT License
39 stars 23 forks source link

Report collector binary fails to submit a query request with 500M rows in it #1147

Open akoshelev opened 1 month ago

akoshelev commented 1 month ago 
     Running `target/release/report_collector --input-file /tmp/events-500000000.txt --network ../jun-2024-aws-cluster-network-config/network.toml oprf-ipa --max-breakdown-key 20 --plaintext-match-keys`
2024-06-17T17:39:06.254363Z  INFO ThreadId(01) ipa_core::cli::verbosity: Logging setup at level info
memory allocation of 14000000000 bytes failed

Fails to allocate 13Gb - the host has more than that, so I assume we make quite a bit extra copies before this input actually gets to the network buffer. Need to optimize this