http-compression prefers zopfli over brotli

GuillaumeRossolini commented 6 years ago

Hi,

[It was suggested to me on Twitter that I open an issue for this.]

At my company, we compress our assets different ways: first gzip, then brotli and lzma, and finally we overwrite the gzip file with a zopfli version. Our compression pipeline was broken for a while, so gzip and brotli worked fine but zopfli and lzma didn't happen. That was sort of ok since zopfli was just an enhancement over gzip, and lzma is used by nobody these days because of brotli taking off. And I don't think we have a rule to actually send any lzma files anyway...

Now about my issue: the http-compression hint asked for gzip and got that encoding (not zopfli), but it also could have asked for brotli and I was lead to understand that this should have been the case here: https://webhint.io/scanner/ad6908fd-a4e2-4c70-99f0-2ec2ca0c9e7b

To be clear, at the time of the report above, the gzip compression was indeed gzip when it could have been zopfli. That's now fixed on our end. That's not the current issue, it's about gzip-compatible VS brotli.

Examples:

$ curl -I https://assets.instantluxe.com/static/InstantLuxe.180.js

HTTP/2 200
server: nginx
date: Fri, 17 Aug 2018 08:57:14 GMT
content-type: text/javascript; charset=utf-8
content-length: 141967
last-modified: Mon, 02 Jul 2018 13:12:01 GMT
vary: Accept-Encoding
etag: "5b3a24a1-22a8f"
expires: Thu, 31 Dec 2037 23:55:55 GMT
cache-control: max-age=315360000
referrer-policy: strict-origin-when-cross-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
access-control-allow-origin: https://www.instantluxe.com
cache-control: public, immutable
x-content-type-options: nosniff
accept-ranges: bytes

$ curl -H "Accept-Encoding: gzip" -I https://assets.instantluxe.com/static/InstantLuxe.180.js

HTTP/2 200
server: nginx
date: Fri, 17 Aug 2018 08:57:30 GMT
content-type: text/javascript; charset=utf-8
content-length: 36841
last-modified: Tue, 14 Aug 2018 07:29:01 GMT
vary: Accept-Encoding
etag: "5b7284bd-8fe9"
content-encoding: gzip
expires: Thu, 31 Dec 2037 23:55:55 GMT
cache-control: max-age=315360000
referrer-policy: strict-origin-when-cross-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
access-control-allow-origin: https://www.instantluxe.com
cache-control: public, immutable
x-content-type-options: nosniff

$ curl -H "Accept-Encoding: br" -I https://assets.instantluxe.com/static/InstantLuxe.180.js

HTTP/2 200
server: nginx
date: Fri, 17 Aug 2018 08:57:38 GMT
content-type: text/javascript; charset=utf-8
content-length: 31846
last-modified: Mon, 06 Aug 2018 14:53:00 GMT
vary: Accept-Encoding
etag: "5b6860cc-7c66"
content-encoding: br
expires: Thu, 31 Dec 2037 23:55:55 GMT
cache-control: max-age=315360000
referrer-policy: strict-origin-when-cross-origin
strict-transport-security: max-age=31536000; includeSubDomains; preload
access-control-allow-origin: https://www.instantluxe.com
cache-control: public, immutable
x-content-type-options: nosniff

Here are the contents on disk (size in bytes, filename):

141967 static/InstantLuxe.min.js
31846 static/InstantLuxe.min.js.br
36841 static/InstantLuxe.min.js.gz
33277 static/InstantLuxe.min.js.lzma

(Yes the "last-modified" dates are not optimal, it's just rsync having lazy rules)

alrra commented 6 years ago

[It was suggested to me on Twitter that I open an issue for this.]

@GuillaumeRossolini Thank you for taking the time to open this issue.

Now about my issue: the http-compression hint asked for gzip and got that encoding (not zopfli), but it also could have asked for brotli ... it's about gzip-compatible VS brotli.

The http-compression hint does a bunch of tests, but in essence it will recommend Zopfli over HTTP , and depending on the targeted browsers and Brotli support, Brotli over HTTPS.

However, you should also keep in mind that servers should respect the request headers. If the user agent advertises that it only supports gzip (even over HTTPS), then the server shouldn't respond with something that is not gzip compatible (such as Brotli).

In your case, it wasn't that it preferred gzip/Zopfli over Brotli, it probably just saw that:

when gzip compression was requested it didn't get Zopfli, so it complained
when Brotli compression was requested over HTTP it did not get Brotli, so it didn't complain
when Brotli compression was requested over HTTPS it did get Brotli, so it didn't complain
....

Hope that that makes things clear.

https://webhint.io/scanner/da70cbb9-a1c5-4f8f-9911-436607af5cd4

@GuillaumeRossolini Good work on reducing the issues! 🎉

GuillaumeRossolini commented 6 years ago

I forgot to say that

the test was done over HTTPS, everything is scheme aware and we have HSTS in place
our domains are all completely redirected to HTTPS if anything is requested over plaintext, but I don't see how that can be (full HTTPS and HSTS predate the "assets" subdomain)

The part that makes sense is:

when gzip compression was requested it didn't get Zopfli, so it complained

alrra commented 6 years ago

I forgot to say that ... The part that makes sense is:

@GuillaumeRossolini I do apologize, but I don't really understand how to interpret your last comment. Are things clear now, or do you still think something is not recommended correctly?

GuillaumeRossolini commented 6 years ago

Yes, your first comment explained why I got this warning. Thank you.

webhintio / hint

http-compression prefers zopfli over brotli #1244