search

storacha / w3link

🪐 The IPFS gateway for web3.storage is not "another gateway", but a caching layer that sits on top of existing IPFS public gateways.
Other
24 stars 9 forks source link

Sporadic "Bad Gateway" response content known to be in R2 #22

Closed olizilla closed 1 year ago

olizilla commented 1 year ago

While testing the fix for https://github.com/web3-storage/freeway/issues/19 after deploying an update to freeway I am seeing (an unrelated) sporadic "Bad Gateway" text/plain response for https://bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link/profile/svg.svg

but expected response is an svg with image/svg+xml content-type from x-freeway-version: 1.5.3

I'm also occasionally seeing a response from an older version with the header freeway-version 1.5.2

example Bad Gatway HAR ```json { "log": { "version": "1.2", "creator": { "name": "WebInspector", "version": "537.36" }, "pages": [ { "startedDateTime": "2022-11-11T12:22:24.572Z", "id": "page_17", "title": "https://bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link/profile/svg.svg", "pageTimings": { "onContentLoad": 305.24599994532764, "onLoad": 304.9180000089109 } } ], "entries": [ { "_initiator": { "type": "other" }, "_priority": "VeryHigh", "_resourceType": "document", "cache": {}, "connection": "23641", "pageref": "page_17", "request": { "method": "GET", "url": "https://bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link/profile/svg.svg", "httpVersion": "http/2.0", "headers": [ { "name": ":authority", "value": "bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link" }, { "name": ":method", "value": "GET" }, { "name": ":path", "value": "/profile/svg.svg" }, { "name": ":scheme", "value": "https" }, { "name": "accept", "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9" }, { "name": "accept-encoding", "value": "gzip, deflate, br" }, { "name": "accept-language", "value": "en-GB,en-US;q=0.9,en;q=0.8" }, { "name": "cache-control", "value": "max-age=0" }, { "name": "if-none-match", "value": "W/\"bafkreid4oicd27wudwwynrhpiokbk5uucdxv3kekrlxnll2utfuaxynqsq\"" }, { "name": "sec-ch-ua", "value": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"" }, { "name": "sec-ch-ua-mobile", "value": "?0" }, { "name": "sec-ch-ua-platform", "value": "\"macOS\"" }, { "name": "sec-fetch-dest", "value": "document" }, { "name": "sec-fetch-mode", "value": "navigate" }, { "name": "sec-fetch-site", "value": "none" }, { "name": "sec-fetch-user", "value": "?1" }, { "name": "upgrade-insecure-requests", "value": "1" }, { "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" } ], "queryString": [], "cookies": [], "headersSize": -1, "bodySize": 0 }, "response": { "status": 502, "statusText": "", "httpVersion": "http/2.0", "headers": [ { "name": "access-control-allow-origin", "value": "*" }, { "name": "access-control-expose-headers", "value": "Link" }, { "name": "cf-ray", "value": "7686fe43a8b488c2-LHR" }, { "name": "content-length", "value": "11" }, { "name": "content-type", "value": "text/html" }, { "name": "date", "value": "Fri, 11 Nov 2022 12:22:24 GMT" }, { "name": "server", "value": "cloudflare" }, { "name": "server-timing", "value": "request;dur=260" }, { "name": "vary", "value": "Accept-Encoding" } ], "cookies": [], "content": { "size": 11, "mimeType": "text/html", "text": "Bad Gateway" }, "redirectURL": "", "headersSize": -1, "bodySize": -1, "_transferSize": 101, "_error": null }, "serverIPAddress": "104.18.41.121", "startedDateTime": "2022-11-11T12:22:24.572Z", "time": 299.14999986067414, "timings": { "blocked": 2.3519999258220197, "dns": -1, "ssl": -1, "connect": -1, "send": 0.1499999999999999, "wait": 295.0229999013245, "receive": 1.6250000335276127, "_blocked_queueing": 0.8769999258220196 } }, { "_initiator": { "type": "other" }, "_priority": "High", "_resourceType": "other", "cache": {}, "connection": "23641", "pageref": "page_17", "request": { "method": "GET", "url": "https://bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link/favicon.ico", "httpVersion": "http/2.0", "headers": [ { "name": ":authority", "value": "bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link" }, { "name": ":method", "value": "GET" }, { "name": ":path", "value": "/favicon.ico" }, { "name": ":scheme", "value": "https" }, { "name": "accept", "value": "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8" }, { "name": "accept-encoding", "value": "gzip, deflate, br" }, { "name": "accept-language", "value": "en-GB,en-US;q=0.9,en;q=0.8" }, { "name": "referer", "value": "https://bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link/profile/svg.svg" }, { "name": "sec-ch-ua", "value": "\"Google Chrome\";v=\"107\", \"Chromium\";v=\"107\", \"Not=A?Brand\";v=\"24\"" }, { "name": "sec-ch-ua-mobile", "value": "?0" }, { "name": "sec-ch-ua-platform", "value": "\"macOS\"" }, { "name": "sec-fetch-dest", "value": "image" }, { "name": "sec-fetch-mode", "value": "no-cors" }, { "name": "sec-fetch-site", "value": "same-origin" }, { "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36" } ], "queryString": [], "cookies": [], "headersSize": -1, "bodySize": 0 }, "response": { "status": 502, "statusText": "", "httpVersion": "http/2.0", "headers": [ { "name": "access-control-allow-origin", "value": "*" }, { "name": "access-control-expose-headers", "value": "Link" }, { "name": "cf-ray", "value": "7686fe45ac4e88c2-LHR" }, { "name": "content-length", "value": "11" }, { "name": "content-type", "value": "text/html" }, { "name": "date", "value": "Fri, 11 Nov 2022 12:22:25 GMT" }, { "name": "server", "value": "cloudflare" }, { "name": "server-timing", "value": "request;dur=373" }, { "name": "vary", "value": "Accept-Encoding" } ], "cookies": [], "content": { "size": 11, "mimeType": "text/html", "text": "Bad Gateway" }, "redirectURL": "", "headersSize": -1, "bodySize": -1, "_transferSize": 102, "_error": null }, "serverIPAddress": "104.18.41.121", "startedDateTime": "2022-11-11T12:22:24.881Z", "time": 415.8930000849068, "timings": { "blocked": 3.9730001674592494, "dns": -1, "ssl": -1, "connect": -1, "send": 0.119, "wait": 411.06899993157384, "receive": 0.7319999858736992, "_blocked_queueing": 3.7830001674592495 } } ] } } ```
olizilla commented 1 year ago

Can reproduce if i send the if-none-match header on the request, as my browser does when the request fails

$ curl -I -X GET 'https://bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini.ipfs.w3s.link/profile/svg.svg' -H 'if-none-match: W/"bafkreid4oicd27wudwwynrhpiokbk5uucdxv3kekrlxnll2utfuaxynqsq"'
HTTP/2 502 
date: Fri, 11 Nov 2022 13:42:53 GMT
content-type: text/html
content-length: 11
access-control-allow-origin: *
access-control-expose-headers: Link
server-timing: request;dur=986
server: cloudflare
cf-ray: 7687741f1a7272e5-LHR
olizilla commented 1 year ago

NOICE! @alanshaw and @olizilla have a lead! The error only occurs when the browser sends the if-none-match header.

If we send the same request direct to freeway with the if-none-match header, it responds with a 304 not modified and no body, which is good an proper https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/304

curl -I -X GET 'https://freeway.dag.haus/ipfs/bafybeie27f5vgj6m75htnf3mdkdaxagsibygyepnkiauxlge35u7vkeini/profile/svg.svg' \
       -H 'if-none-match: W/"bafkreid4oicd27wudwwynrhpiokbk5uucdxv3kekrlxnll2utfuaxynqsq"'
HTTP/2 304 
date: Fri, 11 Nov 2022 13:57:07 GMT
cf-ray: 768788ff485272b5-LHR
access-control-allow-origin: *
age: 175923
cache-control: public, max-age=29030400
etag: "bafkreid4oicd27wudwwynrhpiokbk5uucdxv3kekrlxnll2utfuaxynqsq"
expires: Fri, 13 Oct 2023 13:57:07 GMT
vary: Accept-Encoding
cf-cache-status: HIT
access-control-allow-methods: GET
access-control-expose-headers: Content-Length
x-freeway-version: 1.5.2
server: cloudflare
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

In our ipfs-gateway-race lib we filter down the responses to only ones where reponse.ok is true. Where the client has sent a good if-none-match header, all our upstream gateways are able to return a 304 not modified response, but according to the spec, reponse.ok is only true for 200-299 response codes. When all our upstreams return 304, we are filtering our all the responses, which leaves us with a gateway error. A browser would then not send the if-not-match on a subsequent request because the last response was a 500 with no cache headers, hence the sporadic part.

see: https://github.com/web3-storage/reads/blob/4a8915121cc459ecc81b0e7c20e7ab4cf790f105/packages/ipfs-gateway-race/lib/index.js#L69-L73