IPFS Gateway not exposing content-length header when receiving Accept-Encoding

[vasco-santos]

At the moment IPFS Gateway does not send content-length header in response when it receives Accept-encoding headers. This is problematic for caching that can only be performed when we know content-length and browsers fetch default to send such headers.

Taking into account https://ipfs.io/ipfs/bafybeib4zuumguq4cgkt7caddeukb4ysijnehvntekrtu5afmet5ujvlka/package.json , we get content-length with CURL, but not browser.

@lidel found out that Accept-Encoding request header was the root problem.

curl -sD - https://ipfs.io/ipfs/bafybeib4zuumguq4cgkt7caddeukb4ysijnehvntekrtu5afmet5ujvlka/package.json | grep -i content-length

curl -sD - -H 'Accept-Encoding: gzip' https://ipfs.io/ipfs/bafybeib4zuumguq4cgkt7caddeukb4ysijnehvntekrtu5afmet5ujvlka/package.json | grep -i content-length

References:

• https://github.com/protocol/bifrost-infra/issues/1868
• https://github.com/nftstorage/nftstorage.link/pull/98
• https://github.com/metacpan/metacpan-api/issues/240#issuecomment-10258159

cc @lidel @thattommyhall @gmasgras

[lidel]

@vasco-santos Mind elaborating on "caching that can only be performed when we know content-length" ?

I think that statement is inviting leaky abstractions, just to be sure we operate on the same terminology, quick reminder:

• "content length" in IPFS context can mean many things, see values returned by ipfs files stat /ipfs/{cid}:
  • Size – The size of raw data e.g. byte size of a raw block, or sum of Data fields in UnixFS File's DAG concatenated together
  • I assume this is what you focus on in https://github.com/nftstorage/nftstorage.link/pull/98
  • CumulativeSize The total size of entire DAG (raw data + metadata envelopes)
  • It may happen to be the same as Size if the File was small enough to be inlined as raw leave (CIDv1 with raw codec)
• Content-Length HTTP header informs HTTP clients about the length of the payload
  • IT happens to be the same as the Size of raw data when UnixFS file is sent without chunking, but that is not a solid foundation one should build on (content-length will change based on compression, transfer encoding etc)

My naive understanding is that:

• if you are fetching a file from gateway and then store it in some cache, you already have the data, and can count the bytes yourself, no need for any size header.
  • So nothing new is required here
• if you are doing this cache population lazily, or delegate parts of the process asynchronously and don't have all data at hand, your only option is to do additional ipfs.files.stat call, to learn the Size without fetching the entire thing.
  • If this is a hot code path in your use case, lmk, there are ways of solving for this
    • e.g. what we could do on Gateways, is to return X-Ipfs-DataSize and X-Ipfs-DagSize headers, so you don't need to do any additional ipfs.files.stat call to read the size

[vasco-santos]

caching that can only be performed when we know content-length

Yeah, the object storage cache we use (Cloudflare R2) requires content length to put data. From CF discord there were these related messages that explain the reason. It is also a limitation on their end:

• "The length of the ReadableStream must be known"
• "Our storage nodes currently force us to tell them the content-length up-front. That means we'd have to buffer the payload up-front. Eventually I'm hoping we can lift that requirement from our storage nodes but it's going to take a while (year+, not weeks/months of effort)"

This means that for us to perform a request to the gateway and proxy it to R2 does not work for mentioned content types, unless we read all the content and create a new Response to put to R2. This is not feasible in some environments, like CF Workers as we are limited to 128MB.

Per https://fetch.spec.whatwg.org/#forbidden-header-name , I think we won't be able to overwrite content-length in the client.

[thattommyhall]

Sounds like you need to request it uncompressed then?

[vasco-santos]

@thattommyhall Accept-Encoding is a forbidden header name https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding , I tried to set it to double check, but browser sadly does not pick it.

[thattommyhall]

Can you not do it in the CF worker though? Isnt that your runtime?

[vasco-santos]

Can you not do it in the CF worker though? Isnt that your runtime?

Just tried it, and also impossible to mutate that header, it will always become accept-encoding":"gzip". This should follow Fetch Spec