search

ipshipyard / waterworks-community

Discussion and documentation concerning the operation of the IPFS HTTP Gateway at https://ipfs.io/ipfs.
MIT License
0 stars 0 forks source link

Compression enabled without client asking for it via Accept-Encoding #13

Closed lidel closed 4 months ago

lidel commented 4 months ago

Compression should only be used if client explicitly asked for it via Accept-Encoding.

How to reproduce

🖥️ Click to show Request-response example ```console $ curl https://delegated-ipfs.dev/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v * Host delegated-ipfs.dev:443 was resolved. * IPv6: 2604:1380:45e3:7000::1 * IPv4: 147.75.49.247 * Trying 147.75.49.247:443... * Connected to delegated-ipfs.dev (147.75.49.247) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / id-ecPublicKey * ALPN: server accepted h2 * Server certificate: * subject: CN=*.delegated-ipfs.dev * start date: Jan 4 00:00:00 2024 GMT * expire date: Apr 3 23:59:59 2024 GMT * subjectAltName: host "delegated-ipfs.dev" matched cert's "delegated-ipfs.dev" * issuer: C=AT; O=ZeroSSL; CN=ZeroSSL ECC Domain Secure Site CA * SSL certificate verify ok. * Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384 * Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384 * Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384 * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://delegated-ipfs.dev/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: delegated-ipfs.dev] * [HTTP/2] [1] [:path: /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy] * [HTTP/2] [1] [user-agent: curl/8.5.0] * [HTTP/2] [1] [accept: */*] > GET /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy HTTP/2 > Host: delegated-ipfs.dev > User-Agent: curl/8.5.0 > Accept: */* > < HTTP/2 200 < content-encoding: br < content-type: application/json < vary: Accept-Encoding < vary: Origin < date: Sun, 11 Feb 2024 23:35:57 GMT < content-length: 347 < age: 53 < strict-transport-security: max-age=31536000; includeSubDomains; preload < cache-control: max-age=60 < x-cache-status: HIT < Warning: Binary output can mess up your terminal. Use "--output -" to tell Warning: curl to output it to your terminal anyway, or consider "--output Warning: " to save to a file. ```

Why this is a problem

Among other things:

Is this a someguy bug?

No, compression is opt-in and works correctly when using it directly:

🖥️ Click to show Request-response example ```console # without compression: plaintext works fine $ curl http://127.0.0.1:8091/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v * Trying 127.0.0.1:8091... * Connected to 127.0.0.1 (127.0.0.1) port 8091 > GET /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy HTTP/1.1 > Host: 127.0.0.1:8091 > User-Agent: curl/8.5.0 > Accept: */* > < HTTP/1.1 200 OK < Content-Type: application/json < Vary: Accept-Encoding < Vary: Origin < Date: Mon, 12 Feb 2024 00:06:12 GMT < Content-Length: 841 < * Connection #0 to host 127.0.0.1 left intact {"Providers":[{"Addrs":["/ip6/::1/udp/4001/quic-v1","/ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/127.0.0.1/tcp/4001","/ip4/127.0.0.1/udp/4001/quic-v1","/ip4/192.168.50.102/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/192.168.50.102/udp/4001/quic-v1","/ip6/::1/tcp/4001","/ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/192.168.50.102/tcp/4001"],"ID":"12D3KooWBdmLJjhpgJ9KZgLM3f894ff9xyBfPvPjFNn7MKJpyrC2","Schema":"peer"},{"ID":"12D3KooWQgrExcg6dkCdiTER3G3ARe14PZ4cLhinKtRcLHsvnk1Q","Schema":"peer"}]}% # opt-in compression: also works fine $ curl http://127.0.0.1:8091/routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy -v --compressed * Trying 127.0.0.1:8091... * Connected to 127.0.0.1 (127.0.0.1) port 8091 > GET /routing/v1/providers/QmRhzWfzRHHmMirAQrBimyuiaCRFbPprWLuyw3cBqmHUQy HTTP/1.1 > Host: 127.0.0.1:8091 > User-Agent: curl/8.5.0 > Accept: */* > Accept-Encoding: deflate, gzip, zstd > < HTTP/1.1 200 OK < Content-Encoding: zstd < Content-Type: application/json < Vary: Accept-Encoding < Vary: Origin < Date: Mon, 12 Feb 2024 00:06:31 GMT < Content-Length: 389 < {"Providers":[{"Addrs":["/ip4/192.168.50.102/udp/4001/quic-v1","/ip6/::1/tcp/4001","/ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/46.21.210.45/tcp/27897","/ip4/46.21.210.45/udp/27897/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/127.0.0.1/udp/4001/quic-v1","/ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip4/192.168.50.102/tcp/4001","/ip4/192.168.50.102/udp/4001/quic-v1/webtransport/certhash/uEiCcjA-nIDHEwi2_2bERdoChm571Ghi0Ou494c9U7l1fPQ/certhash/uEiDoxVVQS7XLV3G3kq3tTVKwSyBszVzAcamd3PvySQpxGg","/ip6/::1/udp/4001/quic-v1","/ip4/46.21.210.45/udp/27897/quic-v1","/ip4/127.0.0.1/tcp/4001"],"ID":"12D3KooWBdmLJjhpgJ9KZgLM3f894ff9xyBfPvPjFNn7MKJpyrC2","Schema":"peer"},{"ID":"12D3KooWQgrExcg6dkCdiT* Connection #0 to host 127.0.0.1 left intact ER3G3ARe14PZ4cLhinKtRcLHsvnk1Q","Schema":"peer"}]}% ```
lidel commented 4 months ago

@ns4plabs assigning to you because I confirmed this is not a problem with someguy, this seems to be an infra thing (either reverse proxy is ignoring vary: Accept-Encoding, adding own compression, or maybe we had old responses cached and need to purge them?)

ns4plabs commented 4 months ago

Confirmed that this is indeed an issue with the reverse proxy. It wasn't processing the content-encoding headers correctly despite having the option turned on to take into account vary: Accept-Encoding (Ref.: https://docs.haproxy.org/2.9/configuration.html#6.2.1-process-vary)

For now I disabled caching in HAProxy until a better solution is put in place - https://github.com/ipshipyard/waterworks-infra/commit/da3cc18bdbd349a3008405bb355ce50e02168688