Open rklaehn opened 1 year ago
Let me know if you run into anything while testing it out. We also have quite a few performance-related tasks scheduled so the results are only going to get better from here.
@flub this is not really urgent, but if you want to try it out it would be nice.
I sometimes think that quic-rpc would also be a good option for connections between peers, but unlike for the RPC use case where hyper is fine, here we would really want QUIC.
yeah, had this on my "if i have time" list last weekend already :) but as you can tell... time... but i'd be keen to try it out sometime still
Here is a first attempt: https://github.com/n0-computer/quic-rpc/tree/s2n-quic
@camshaft I now got a very ugly but working impl in the above branch.
The API of quic-rpc was originally modeled after quinn, so in order to match the quic-rpc API I have to do so some seriously ugly stuff like call poll_whatever after grabbing a lock, because many of the s2n-quic methods require mutable references.
We might adapt the API later to avoid that, depending on how this experiment goes.
The bench works, however the bidirectional streaming is extremely slow. It seems like some buffer is not sent immediately but only after some fixed delay.
cargo test --release s2n_quic_channel_bench -- --nocapture
RPC seq 8_420 rps
RPC par 117_260 rps
bidi seq 37 rps
For comparison, this is the somewhat similar quinn based impl:
cargo test --release quinn_channel_bench -- --nocapture
RPC seq 16_243 rps
RPC par 201_385 rps
bidi seq 173_999 rps
I looked at this with good old println debugging, and it seems while the server gets the requests almost immediately, the responses only get to the client as a trickle. Revert https://github.com/n0-computer/quic-rpc/commit/98986b44b89c0885bbd13cb81336307dddf9c57d to see the debug output.
All the ugliness with the locks is done for establishing a connection. Once that is done, the code is extremely similar to the quinn code, so I am not sure this is on my side.
One notable thing is that we are not using the default crypto but rustlis. I guess I could try with the default crypto to make sure that is not the cause...
Tried out the built in crypto. Does not make a difference - same behaviour.
I have two branches where I wired up s2n_quic. One where there is questionable stuff like taking a lock to poll futures, and one that is pretty clean.
In both cases I am getting extremely bad performance, in particular for the bidi seq
test.
This would require a closer look before we proceed with this. In it's current form it is unusable because it is unbearably slow.
Can you open a draft PR with the clean one and I can add some comments
Also could you share a flamegraph of the benchmarks you're running. There might be something that stands out there
Will do. Just have to clean it up a bit.
Here it is: #40
There is basically a smoke test and a perf test for each transport. Here is the output of the perf tests:
❯ cargo test _channel_bench --all-features --release -- --nocapture
...
running 1 test
RPC seq 584_690 rps ....................
RPC par 642_932 rps
bidi seq 14_068_053 rps ....................
test flume_channel_bench ... ok
running 1 test
RPC seq 17_068 rps
RPC par 83_819 rps
bidi seq 587_368 rps
test hyper_channel_bench ... ok
running 1 test
RPC seq 21_120 rps
RPC par 190_103 rps
bidi seq 290_512 rps
test quinn_channel_bench ... ok
running 1 test
RPC seq 1_618 rps
RPC par 25_370 rps
bidi seq 35 rps
test s2n_quic_channel_bench ... ok
Maybe something about the config I need to change?
The quinn throughput numbers are disappointing. s2n-quic seems to have a pretty high level API, so it should not be that hard to wrap it.
https://github.com/aws/s2n-quic