Fix HTTP Caching of Public Goods responses (remove Vary: Origin)

lidel commented 9 months ago

Right now, service at delegated-ipfs.dev/routing/v1 sends CORS header only when Origin header is present in the request, and when Origin is present, the response has Vary: Origin:

$ curl -H "Origin: https://example1.com" https://delegated-ipfs.dev/routing/v1/providers/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi -s -i | grep -i origin
access-control-allow-origin: *
vary: Origin

Problem

If the Vary header in response is set to value Origin, it indicates that the response may vary depending on the value of the Origin header in the request.

It means the response is reusable (cacheable) only as long value in Origin header matches, so responses for requests made from different websites (origins) won't benefit from caching.

IIUC this makes little sense with access-control-allow-origin: * because we want liberal access to public goods, and we don't have site-specific responses, so we want cache to be shared across websites that use public goods to maximize cache HIT rate:

:point_right: We want CID lookup done by JS running on https://one.example.com to be returned from cache when https://two.example.net asks for it.

Solution

Trustless public good services must have a global cache that is shared across websites (no matter what is in Origin), which means we don't want Vary: Origin at

delegated-ipfs.dev (needs fixing)
trustless-gateway.link (already ok, but mentioning here as it should share the config/setup/tests)

TODO

[x] adding CORS, Vary, and HTTP Caching to our HTTP specs is tracked in https://github.com/ipfs/specs/issues/423 (added /routing/v1 items there, consider out of scope here)
[ ] remove Vary: Origin from delegated-ipfs.dev responses
- Caveat: be careful to not remove Vary: Accept-Encoding which is used by compression
- I suspect delegated-ipfs.dev regression was introduced when we added github.com/rs/cors to someguy (https://github.com/ipfs-shipyard/someguy/pull/30). An easy fix may be to remove that library and instead hardcode liberal CORS headers on all response types.

lidel commented 9 months ago

@ns4plabs I am not sure how easy it will be to fix in someguy without having to modify the github.com/rs/cors library (imo it should not send Vary: Origin if CORS is effectively disabled/lifted by *).

I it helps, I think for this specific project (someguy) hardcoding liberal CORS on all GET|HEAD|POST|OPTIONS responses is fine. Are you able to add this to your queue and ping me or @hacdias for review?

lidel commented 3 months ago

Somehow related improvements landed in https://github.com/ipshipyard/waterworks-infra/issues/215 Remaining work is to remove Vary: Origin

ipshipyard / waterworks-community