Open huntharo opened 1 month ago
Wow, awesome analysis, thank you! 🚀
Thank you for reporting it.
I've been able to reproduce it. tested at AWS CloudShell in ap-northeast-1
$ time h2load -c 1 -n 100 -m 1 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
starting benchmark...
spawning thread #0: 1 total client(s). 100 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: X25519 253 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done
finished in 15.74s, 6.35 req/s, 15.59MB/s
requests: 100 total, 100 started, 100 done, 100 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 245.37MB (257291904) total, 7.97KB (8159) headers (space savings 82.86%), 245.23MB (257140500) data
min max mean sd +/- sd
time for request: 138.92ms 900.98ms 153.99ms 76.99ms 98.00%
time for connect: 337.22ms 337.22ms 337.22ms 0us 100.00%
time to 1st byte: 451.57ms 451.57ms 451.57ms 0us 100.00%
req/s : 6.35 6.35 6.35 0.00 100.00%
real 0m15.755s
user 0m0.318s
sys 0m0.198s
$ time ./oha-linux-amd64 -c 1 -p 1 -n 100 --http2 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
Summary:
Success rate: 100.00%
Total: 25.3859 secs
Slowest: 0.8901 secs
Fastest: 0.2342 secs
Average: 0.2505 secs
Requests/sec: 3.9392
Total data: 245.23 MiB
Size/request: 2.45 MiB
Size/sec: 9.66 MiB
Response time histogram:
0.234 [1] |
0.300 [98] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
0.365 [0] |
0.431 [0] |
0.497 [0] |
0.562 [0] |
0.628 [0] |
0.693 [0] |
0.759 [0] |
0.825 [0] |
0.890 [1] |
Response time distribution:
10.00% in 0.2351 secs
25.00% in 0.2396 secs
50.00% in 0.2456 secs
75.00% in 0.2461 secs
90.00% in 0.2485 secs
95.00% in 0.2543 secs
99.00% in 0.8901 secs
99.90% in 0.8901 secs
99.99% in 0.8901 secs
Details (average, fastest, slowest):
DNS+dialup: 0.0000 secs, 0.0000 secs, 0.0000 secs
DNS-lookup: 0.0000 secs, 0.0000 secs, 0.0000 secs
Status code distribution:
[200] 100 responses
real 0m25.459s
user 0m1.256s
sys 0m0.425s
And I've created a simple load generator to find whether the difference is came from oha
or hyper
which is backend of oha
.
https://github.com/hatoo/oha-538
$ time ./oha-538 https://media2.giphy.com/media/pE4zqXvpwpaUh67tkw/giphy.gif
....................................................................................................
real 0m24.733s
user 0m0.664s
sys 0m0.642s
I guess the difference came from hyper
.
I'll investigate further.
Could you check on your side?
Oh cool will try
Hmm... it's kind of tricky to do given that I'm on a Mac and CloudShell doesn't seem to have enough space to setup the Rust toolchain. I could do it a thousand other way but it would be so nice if the linux binary with that fix was available?
@hatoo On main builds (and even on branches) - What if we saved the built binaries as artifacts that we could download them when we need to test a specific version or pre-release. Thoughts?
540 solves on my side.
Could you check on your side?
@hatoo did you check the performance vs h2load
with the fix? What I see is:
oha
before fix: 25 seconds for 100 copies of fileoha
with fix: 18 secondsh2load
: 14.5 secondsThe fix has closed 64% of the gap with h2load
, which is GREAT!
But there is something else it seems? I wonder if Rustls is the remaining slowness or if we have to change any other parameters on hyper?
In my observation, #540 constantly did 100 downloads by around 15.5s which is about the same as h2load
But adaptive_window
may work vary on runtime (although I don't know what adaptive_window
means).
I've found that h2loads's default window size setting is larger than hyper
's default
I've added on #541. It takes time the same as h2load
in my tests.
Could you check it? Binary is attached in the PR.
@hatoo On main builds (and even on branches) - What if we saved the built binaries as artifacts that we could download them when we need to test a specific version or pre-release. Thoughts?
It looks good but I don't know how to do it without spamming https://github.com/hatoo/oha/releases page
Overview
I was testing the download speed characteristics from Japan (AWS CloudShell in
ap-northeast-1
) to Fastly's LAX PoP using both HTTP1.1 and HTTP2 usingoha
(this is about a 100 ms RTT).I observed that the steady state response time for a 2.5 MB file (non-compressible) stabilized as follows:
I reported this issue to Fastly and they asked that I retest with
h2load
fromnghttp2
's reference implementation.Steps to reproduce are below.
At first I thought maybe
h2load
had a reporting issue as it is more difficult to tell the steady state response times (the initial response takes 1.2 seconds due to their slow start and long RTT) given that they only reportmean
time and notmedian
. But running the commands withtime
showed the overall process foroha
taking 2x longer as well, which matched up.Commands for Testing
Results
✅ oha - http/1.1
❌ oha - http2
✅ h2load - http/1.1
✅ h2load - http2
Building
h2load
on AWS CloudShell