Clarify HTTP/2 and HTTP/3 pseudo-headers interaction with ASGI

[jlaine]

In addition to "normal" HTTP headers, HTTP/2 and HTTP/3 use pseudo-headers starting with ":", some of which have natural mappings to ASGI scope variables :

• :method: => this maps to scope["method"]
• :path: => this maps to scope["raw_path"]
• :scheme: => this maps to scope["scheme"]

A more problematic pseudo-header is :authority which is the equivalent of HTTP/1.1's Host header.

It would be good to clarify in the ASGI specs:

• how should the :authority pseudo-header be conveyed by the server to the ASGI app: should servers append a synthetic Host header?

• my feeling is that pseudo-headers should be purge from what is passed into scope["headers"], do we want to specify this in the specs?

Another interesting pseudo-header is the :protocol pseudo-header which is used to handshake WebSockets over HTTP/2, and very likely also over HTTP/3.

[andrewgodwin]

I'd have to read up more on pseudo-headers to know how they need to be mapped, but I agree they should be stripped out of the scope's headers, as that's meant for traditional-style headers.

As for :authority, I suspect we should synthesise a Host header, but it would be nice to know if there's any precedent for this among existing HTTP/2 servers?

[jlaine]

In terms of Python servers, I think hypercorn is the only one which handles HTTP/2 internally. AFAICT hypercorn does not explicitly handle :authority and passes all the pseudo-headers through to the ASGI app.

I'll let @pgjones confirm this is the case, but running Django 2.2 (wrapped in WsgiToAsgi) behind hypercorn triggered a host validation error, which is what caused me to investigate the situation.

If you're using quart as your framework you're in luck (otherwise you are not!):

https://gitlab.com/pgjones/quart/blob/af7d7933b61e1855f6b970a09a108f254fd4368a/quart/wrappers/_base.py#L286

In my demo HTTP/3 server I synthesise a Host header and do not pass any pseudo-headers to the ASGI app:

https://github.com/aiortc/aioquic/blob/7e756f41d53bc63e445bc10788095c42942f7553/examples/http3_server.py#L189

Looking beyond the Python ecosystem, nginx synthesises a Host header and does not pass through pseudo-headers when acting as a proxy. If you are running any Django apps behind an http2-enabled nginx this is why it "just works".

[pgjones]

I've started to think of authority as much a part of the HTTP request semantics as path, method, and scheme. I think this is what HTTP/1.0 implicitly says (with a mandatory Host header) and HTTP/2 & HTTP/3 explicitly say. For this reason I think ASGI servers should extract it and put it into the scope as the authority field (better name in my view than host). This has the advantage of putting all the HTTP request semantics together, allows apps to ignore HTTP version differences (which should be a server problem), and finally makes it clear that apps shouldn't just look for a host header (which doesn't exist in 2 & 3). I'd rather the "fudge" of creating a host header be left up to applications to do.

I'm currently happy to pass the pseudo headers through as it doesn't seem likely to cause a problem, I'd be just as happy to filter them out.

[tomchristie]

I guess there's either:

• Have HTTP/2 servers fill in a Host header.
• Have all servers fill in an authority scope (although possibly making it optional/None-able, since it's being newly introduced.)

I think we should probably mandate that pseudo headers should not be included in the headers scope, since we'll steer usages towards better interoperability without them being present.

[jlaine]

I'm 100% for forbidding pseudo-headers in headers right now.

As for Host vs an authority scope, I'm (weakly) in favour of synthesising a Host header, primarily to give us instant compatibility with existing frameworks.

[andrewgodwin]

I've long held the position that ASGI's position is not to be a low-level HTTP library but a useful abstraction that works for 95% of apps, and so given that I agree we should ban pseudo-headers from going through and we should synthesise a Host header.

[pgjones]

See #123.