search

private-attribution / ipa

A raw implementation of Interoperable Private Attribution
MIT License
42 stars 25 forks source link

Malicious IPA hangs in real world setup with 3 helpers #685

Closed akoshelev closed 1 year ago

akoshelev commented 1 year ago

I have set up 3 machines that represent helper parties and another one that plays the role of report collector (test_mpc). Helpers are using Docker to run IPA code, RC just straight calls cargo to run the test.

The results are disappointing. Semi-honest IPA scales up to 1M event with no issues. Malicious IPA breaks somewhere between 1000 and 5000 input reports.

This will be a tracking issue to fix this bug

akoshelev commented 1 year ago

@martinthomson suggested to use artificially high maximum number of HTTP2 streams (10k from the current 4k)

akoshelev commented 1 year ago

First thing I am going to try is to run this without Docker

akoshelev commented 1 year ago

fun stuff - forgot that you need to dance around permissions to allow binding to low numbered ports

./target/release/helper --network network.toml --identity 3 --tls-cert h3.pem --tls-key h3.key --port 443
2023-06-07T23:48:13.006296Z  INFO ipa::cli::verbosity: Logging setup at level info
thread 'main' panicked at 'Failed to bind server to a port', /home/ec2-user/workspace/ipa/src/net/server/mod.rs:189:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'Failed to serve: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }', /home/ec2-user/workspace/ipa/src/net/server/mod.rs:243:18
[ec2-user@ip-172-31-56-177 ipa]$ sudo setcap CAP_NET_BIND_SERVICE=+eip ./target/release/helper
akoshelev commented 1 year ago

having the same issue without docker as well. we hang at user capping

2023-06-07T23:58:10.857052Z  INFO ipa_query:attribute:apply_window: ipa::protocol::attribution::apply_attribution_window: close time.busy=501µs time.idle=18.9µs
2023-06-07T23:58:10.857068Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: new
2023-06-07T23:58:11.804931Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: close time.busy=527ms time.idle=421ms
2023-06-07T23:58:11.804955Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
...
nothing is happening after
akoshelev commented 1 year ago

next step is to try what Martin has suggested

akoshelev commented 1 year ago

changing this to 10k

https://github.com/private-attribution/ipa/blob/6292d3a3ae55bb1c54927196e0fa1c4f15c5e0d8/src/net/server/mod.rs#L237

akoshelev commented 1 year ago

I am not surprised but it worked for 5k input

2023-06-08T17:01:53.693025Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
2023-06-08T17:02:17.855622Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: close time.busy=13.0s time.idle=11.1s
2023-06-08T17:02:17.855654Z  INFO ipa_query:attribute:aggregate_credit: ipa::protocol::attribution::aggregate_credit: new
2023-06-08T17:02:20.147518Z  INFO ipa_query:attribute:aggregate_credit: ipa::protocol::attribution::aggregate_credit: close time.busy=1.20s time.idle=1.09s
2023-06-08T17:02:20.147554Z  INFO ipa_query:attribute:validate{step=protocol/after_convert_all_bits/validate}: ipa::protocol::context::validator: new
2023-06-08T17:02:20.170653Z  INFO ipa_query:attribute:validate{step=protocol/after_convert_all_bits/validate}: ipa::protocol::context::validator: close time.busy=127µs time.idle=23.0ms
2023-06-08T17:02:20.174874Z  INFO ipa_query:attribute: ipa::protocol::attribution: close time.busy=17.1s time.idle=13.7s
2023-06-08T17:02:20.175747Z  INFO ipa_query: ipa::query::runner::ipa: close time.busy=36.3s time.idle=34.5s
akoshelev commented 1 year ago

Scaling up to 50k

akoshelev commented 1 year ago

50k worked just fine

2023-06-08T17:14:52.968401Z  INFO ipa_query:attribute:validate{step=protocol/after_convert_all_bits/validate}: ipa::protocol::context::validator: close time.busy=156µs time.idle=13.2ms
2023-06-08T17:14:52.981242Z  INFO ipa_query:attribute: ipa::protocol::attribution: close time.busy=178s time.idle=122s
2023-06-08T17:14:52.989412Z  INFO ipa_query: ipa::query::runner::ipa: close time.busy=372s time.idle=281s
akoshelev commented 1 year ago

Overconfident me decided to run it with 500k input

akoshelev commented 1 year ago

It finished within 2hours which is good

akoshelev commented 1 year ago

However I still don't know the root cause. I strongly suspected https://github.com/hyperium/hyper/issues/2419 to be it because we were using hyper 0.14.20 that does not have the fix for it. However, after upgrading to 0.14.26 I am still seeing hangs in user capping

2023-06-08T21:21:03.936735Z  INFO ipa_query:attribute:validate{step=protocol/binary_validator/validate}: ipa::protocol::context::validator: new
2023-06-08T21:21:03.939183Z  INFO ipa_query:attribute:validate{step=protocol/binary_validator/validate}: ipa::protocol::context::validator: close time.busy=793µs time.idle=1.66ms
2023-06-08T21:21:04.321705Z  INFO ipa_query:attribute:apply_window: ipa::protocol::attribution::apply_attribution_window: new
2023-06-08T21:21:04.322332Z  INFO ipa_query:attribute:apply_window: ipa::protocol::attribution::apply_attribution_window: close time.busy=608µs time.idle=19.6µs
2023-06-08T21:21:04.322351Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: new
2023-06-08T21:21:05.375147Z  INFO ipa_query:attribute:accumulate_credit: ipa::protocol::attribution::accumulate_credit: close time.busy=533ms time.idle=520ms
2023-06-08T21:21:05.375175Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new

cargo.toml

hex = { version = "0.4", optional = true, features = ["serde"] }
hkdf = "0.12.3"
hpke = { version = "0.10.0", default-features = false, features = ["x25519-dalek"] }
hyper = { version = "0.14.26", optional = true, features = ["client", "server", "h2", "stream"] }
hyper-tls = { version = "0.5.0", optional = true }
akoshelev commented 1 year ago

HTTP2 settings negotiated between client and server (both are helpers)

Frame 15: 99 bytes on wire (792 bits), 99 bytes captured (792 bits) Ethernet II, Src: 0e:df:af:76:80:0d (0e:df:af:76:80:0d), Dst: 0e:be:e3:d5:6f:db (0e:be:e3:d5:6f:db) Internet Protocol Version 4, Src: 172.31.49.227, Dst: 172.31.49.57 Transmission Control Protocol, Src Port: 3000, Dst Port: 39880, Seq: 1, Ack: 271, Len: 33 HyperText Transfer Protocol 2 Stream: SETTINGS, Stream ID: 0, Length 24 Length: 24 Type: SETTINGS (4) Flags: 0x00 0... .... .... .... .... .... .... .... = Reserved: 0x0 .000 0000 0000 0000 0000 0000 0000 0000 = Stream Identifier: 0 Settings - Max concurrent streams : 256 Settings - Initial Windows size : 1048576 Settings - Max frame size : 16384 Settings - Max header list size : 16777216

akoshelev commented 1 year ago

Hmm we are genuinely use >200 streams in capping

I added a tracker to count number of active streams: https://github.com/private-attribution/ipa/commit/8660d8fbf10f5aef18684c98139298051e017080

Here is what I am seeing before user capping gets stuck:

2023-06-08T23:32:08.024202Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
2023-06-08T23:32:08.024355Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits]: 0
2023-06-08T23:32:08.024386Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits/duplicate_multiply]: 1
2023-06-08T23:32:08.127072Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits] closed: 2
2023-06-08T23:32:08.127104Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/mask_source_credits/duplicate_multiply] closed: 1
2023-06-08T23:32:08.137016Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit0/upgrade/upgrade_bit_triple0]: 0
2023-06-08T23:32:08.137063Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit0/upgrade/upgrade_bit_triple1]: 1
2023-06-08T23:32:08.137098Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit1/upgrade/upgrade_bit_triple0]: 2
2023-06-08T23:32:08.137135Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit1/upgrade/upgrade_bit_triple1]: 3
2023-06-08T23:32:08.137183Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/random_bits_for_comparison/random_bits/upgrade_bit_triples/bit2/upgrade/upgrade_bit_triple0]: 4
...

2023-06-08T23:32:09.671726Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit15]: 224
2023-06-08T23:32:09.671744Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit15/duplicate_multiply]: 225
2023-06-08T23:32:09.757334Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit16]: 226
2023-06-08T23:32:09.757377Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit16/duplicate_multiply]: 227
2023-06-08T23:32:09.757406Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit17]: 228
2023-06-08T23:32:09.757431Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/malicious_protocol/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/prefix_or/bit17/duplicate_multiply]: 229

I think my counter is sloppy but nevertheless we are dangerously close to the point where we can't really open a new stream because of 256 concurrent streams limit. Sad that our code is structured the way it can't make progress on the active steps without opening a new stream, @martinthomson work on streaming should help here as well

akoshelev commented 1 year ago

Semi-honest capping is much more frugal in streams usage


2023-06-08T23:40:44.468150Z  INFO ipa_query:attribute:user_capping: ipa::protocol::attribution::credit_capping: new
2023-06-08T23:40:44.468259Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/mask_source_credits]: 0
2023-06-08T23:40:44.557988Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: channel[H2,step=protocol/after_convert_all_bits/user_capping/mask_source_credits] closed: 1
2023-06-08T23:40:44.601742Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit0/xor1]: 0
2023-06-08T23:40:44.601799Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit1/xor1]: 1
2023-06-08T23:40:44.601833Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit2/xor1]: 2
2023-06-08T23:40:44.601870Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit3/xor1]: 3
2023-06-08T23:40:44.601902Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/random_bits_for_comparison/random_bits/convert_shares/bit4/xor1]: 4
...
2023-06-08T23:40:46.552108Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_lo/dot_product]: 96
2023-06-08T23:40:46.552154Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit0]: 97
2023-06-08T23:40:46.552182Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit1]: 98
2023-06-08T23:40:46.696108Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit2]: 99
2023-06-08T23:40:46.696159Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit3]: 100
2023-06-08T23:40:46.696189Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit4]: 101
2023-06-08T23:40:46.840098Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit5]: 102
2023-06-08T23:40:46.840156Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit6]: 103
2023-06-08T23:40:46.840189Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit7]: 104
2023-06-08T23:40:46.984126Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit8]: 105
2023-06-08T23:40:46.984338Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit9]: 106
2023-06-08T23:40:46.984375Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit10]: 107
2023-06-08T23:40:47.128113Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit11]: 108
2023-06-08T23:40:47.128172Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit12]: 109
2023-06-08T23:40:47.128210Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit13]: 110
2023-06-08T23:40:47.272115Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit14]: 111
2023-06-08T23:40:47.272175Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit15]: 112
2023-06-08T23:40:47.272205Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit16]: 113
2023-06-08T23:40:47.416108Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit17]: 114
2023-06-08T23:40:47.416163Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit18]: 115
2023-06-08T23:40:47.416203Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit19]: 116
2023-06-08T23:40:47.560107Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit20]: 117
2023-06-08T23:40:47.560155Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit21]: 118
2023-06-08T23:40:47.560183Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit22]: 119
2023-06-08T23:40:47.704098Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit23]: 120
2023-06-08T23:40:47.704163Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit24]: 121
2023-06-08T23:40:47.704201Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit25]: 122
2023-06-08T23:40:47.848144Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit26]: 123
2023-06-08T23:40:47.848218Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit27]: 124
2023-06-08T23:40:47.848243Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit28]: 125
2023-06-08T23:40:47.992102Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit29]: 126
2023-06-08T23:40:47.992165Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/prefix_or/bit30]: 127
2023-06-08T23:40:47.992222Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/compare_hi/dot_product]: 128
2023-06-08T23:40:48.205794Z  INFO ipa_query:attribute:user_capping: ipa::helpers::gateway::send: opening new stream: channel[H2,step=protocol/after_convert_all_bits/user_capping/report_level_capping/is_cap_less_than_current_contribution/and]: 129
martinthomson commented 1 year ago

I fear that work on streaming will make this worse not better, because it might allow for progress on more steps at a time.

akoshelev commented 1 year ago

I fear that work on streaming will make this worse not better, because it might allow for progress on more steps at a time.

hmm, true. I don't think I understand all the consequences, but maybe we can just remove the limit on number concurrent streams. There must be some bookkeeping cost on both client and server side but it seems we would need to pay it regardless if we really need 10k concurrent streams.

martinthomson commented 1 year ago

I doubt we'll get to that number, but as long as there is no cost for having that many streams possible, the cost of maintaining state for the streams we do use should be small.

akoshelev commented 1 year ago

fixed by #690