search

internetstandards / Internet.nl

Internet standards compliance test suite
https://internet.nl
166 stars 36 forks source link

Compression possible with Transfer-Encoding #1350

Open bwbroersma opened 3 months ago

bwbroersma commented 3 months ago

Thanks to:

I noticed currently the compression is only checked by adding a Accept-Encoding: https://github.com/internetstandards/Internet.nl/blob/a469e4c151c1740d3f69e36235bc854b0099004a/checks/tasks/http_headers.py#L733 To check if the server will return a Content-Encoding: https://github.com/internetstandards/Internet.nl/blob/a469e4c151c1740d3f69e36235bc854b0099004a/checks/tasks/http_headers.py#L24-L39

However it's also possible to have compression with Transfer-Encoding (hop-by-hop), see HTTP/1.1 - 7.2. Transfer Codings for Compression - RFC 9112. Note that in HTTP/2, Transfer-Encoding MUST not be present, see HTTP/2 - 8.2.2. Connection-Specific Header Fields - RFC 9113.

The IANA registered values that should be checked for in HTTP/1.1 are: (x-)compress, deflate and (x)-gzip.

BTW 'officially' Content-Encoding shouldn't even be used for on-the-fly compression, but reality is different:

The HTTP syntax error has been fixed in trunk, but the problem motivating this report is a no-win situation no matter how it is "fixed". The only good answer is "don't use mod_deflate" because changing content-encoding on the fly in an inconsistent manner (neither "never" nor "always) makes it impossible for later requests regarding that content (e.g., PUT or conditional GET) to be handled correctly. This is, of course, why performing on-the-fly content-encoding is a stupid idea, and why I added Transfer-Encoding to HTTP as the proper way to do on-the-fly encoding without changing the resource.

See the 2009-02-12 comment by Roy T. Fielding (one of the HTTP authors).

It could be tested by (in case of HTTP/1.1) by also sending the header:

TE: compress, deflate, gzip, x-compress, x-gzip

And then check for any compression value (chunked is okay) in Transport-Encoding, see HTTP/1.1 - 7.4. Negotiating Transfer Codings - RFC 9112.