HTTP/2 download speeds / flow control settings

[tomchristie]

Which having an initial look at HTTP/3, @jlaine dug out that we seem to have signiifcantly slow uploading right now.

First thing to do would be to investigate and replicate, by eg. compare and contrast a simple upload from httpx vs requests/urllib3 - does it replicate trivially and reliably? This really shouldn't be an issue for us so will need some looking at.

Initial thoughts on where issues could be:

• We're sending too-small chunks, and always waiting for the network to fully drain on each chunk is negatively impacting performance.
• We're sending too-small chunks, and ending up with largish computation overhead relative to data transfer as a result.
• We've got some big unseen overhead in the sync-to-async marshalling.
• Something else?

[jlaine]

Actually the issue I reported was download speeds, and it does seem to be linked to HTTP/2.

Here is a script to reproduce the issue:

import httpx
import time
import requests

url = "https://quic.aiortc.org/50000000"

start = time.time()
r = requests.get(url)
print("requests %.1f s" % (time.time() - start))

for http_version in ["HTTP/1.1", "HTTP/2"]:
    client = httpx.Client(http_versions=http_version)

    start = time.time()
    r = client.get(url)
    print("httpx %s %.1f s" % (http_version, time.time() - start))

The results (the first two results vary somewhat, but are always close to each other):

requests 5.6 s
httpx HTTP/1.1 5.8 s
httpx HTTP/2 45.7 s

[tomchristie]

Okay, at this point it's a bit more complicated, since we've also got different performance on the server side too.

A sensible first thing to do might be to try to replicate against a non-python HTTP/1.1 and HTTP/2 server in order to try to help isolate if we've for a client issue or a server issue here (or both)?

[tomchristie]

Tried this against httpbin, because that'll be implemented as a WSGI HTTP/1.1 - I was expecting it to be fronted by something that supported both HTTP/1.1 and HTTP/2 (I think it's on heroku?)

That didn't actually progress things tho, since it turns out whatever it's hosted on is HTTP/1.1 only...

import httpx
import time
import requests

url = "https://httpbin.org/stream-bytes/50000000"

start = time.time()
r = requests.get(url)
print("requests %.1f s" % (time.time() - start))

for http_version in ["HTTP/1.1", "HTTP/2"]:
    client = httpx.Client(http_versions=http_version)

    start = time.time()
    r = client.get(url)
    print("httpx %s %.1f s" % (http_version, time.time() - start))
    print(r.http_version)

$ python ./example.py 
requests 0.8 s
httpx HTTP/1.1 0.8 s
HTTP/1.1
httpx HTTP/2 0.8 s
HTTP/1.1

Might want to try something similar but against some cloudflare content or something - would at least help us isolate things a bit more clearly.

[jlaine]

The server mentioned (quic.aiortc.org) above is serving HTTP/1.1 and HTTP/2 using nginx talking (over HTTP/1.1) to uvicorn.

Using cURL over HTTP/1.1 or HTTP/2 against the same server as above both give me times which are consistent with requests and httpx over HTTP/1.1 (i.e. around 6s). httpx over HTTP/2 really seems to be the outlier.

curl --http1.1 https://quic.aiortc.org/50000000 curl --http2 https://quic.aiortc.org/50000000

The httpbin test you gave above isn't super exciting, it only transfers ~100kB of data vs 50MB in mine :)

[tomchristie]

Okay, that's really helpful, thanks.

Something that'd be useful for us to do at some point would be crack open a tool like wireshark in cases like this, and verify if our HTTP messaging matches up properly with curl's implementation or not. (Ie. are we doing something differently in terms of the messaging that's causing it to be slower)

[tomchristie]

We've got a much cleaner HTTP/2 implementation now, and this still replicates.

I strongly suspect that it's a flow control issue. Perhaps something like our allowable window sizes being set too low by default, and having to send too many window updates.

In any case, we really need someone to try to dig into a proper comparision against some other tool (curl or a browser implementation) too see how our messaging differs.

I guess it's also possible that the messaging is correct, but there's a networking implementation issue (eg. perhaps we really need the writes to be on a different flow of control here.) I don't particularly expect that's the case, but it might be interesting to see if the issue persists on asyncio if we stop calling .drain() after every write?