Leverage Service Worker and WebTransport on Public Gateways

This is a stub/meta issue for tracking work to leverage Service Workers (SW) as a fallback on public gateways.

cc historical SW threads for discovery: #57 https://github.com/ipfs/in-web-browsers/issues/171 https://github.com/ipfs/in-web-browsers/issues/158 https://github.com/ipfs/kubo/issues/4025 https://github.com/ipfs/in-web-browsers/issues/55

Broad strokes idea

SW would intercept requests for specific content root (/ipfs/cid/ or /ipns/id) and load data using
- Helia
- delegated routing over HTTPs (caskadht from cid.contact, /routing/v1/providers/cid?cascade=ipfs-dht)
- bitswap over WebTransport transport (allowing direct fetch from Kubo)

Prior art

There are two ways one can fetch blocks:

Demo on js.ipfs.tech uses ancient js-ipfs not sure if it still works. But we could redo that with Helia, IPNI and WebTransport.
Saturn' browser client uses ?format=car instead, and fetches it from a single public gateway. Unsure if good prior art to look at tho.

I think the thing we want to create something robust that does do both, and use all public gateways as a fallback

Open questions

When SW is registered / used

First HTTP GET will always hit the gateway. Response could register worker and reload. Future requests will hit SW, as long they come from the same Origin (see "no foreign fetch" below)

No foreign fetch

:warning: before anyone gets too creative, read an understand fundamental limitation of ServiceWorkers: the lack of foreign fetch. tldr: this means SW is active only if the root document loaded in the browser comes from the same Origin.

Use case ideas

(A) Client-side recovery

When gateway returns error (list tbd, could be 429 and maybe also 502 503 and 504), and it knows request comes from web browser, return text/html response (https://github.com/ipfs/boxo/issues/262).

Augument text/html response to include an option to recover from page load error via a Service Worker IPFS node.

This is a good candidate for dipping our toes, does not impact "successful" flow, but allows for recovery from 429 Too Many Requests and Saturn CDN hiccups we will see in the near future.

Recovery could be based on Helia and WebTransport and/or verifiable HTTP block/car requests to public gateways

(B) Ability to register global worker

Giving user ability to prefer local JS for future requests, not only for errors, but all responses.

(C) Ability to isolate path gateway

We could do something related to https://github.com/ipfs/in-web-browsers/issues/157 where actual payload is loaded in sandboxed iframe

Nice! @lidel a few questions:

First HTTP GET will always hit the gateway. Response could register worker and reload.

I am not sure how this would work, serviceWorker.register() can only be called from the navigator context. Which means, calling gateway directly won't register a service worker. To register a service worker, the gateway needs to be loaded explicitly which can call navigator.serviceWorker.register().

Giving user ability to prefer local JS for future requests

I feel the users should be oblivious to the fact where the content is being loaded from. This should be local first by-default and recover from a public gateway if needed.

Other Thoughts

Helia can ship with a bootstrap service worker, which can register a when loaded with in a navigator context. All requests heading out to the gateway can then be handled here. This out of the box functionality.
If we want to explicitly have a service worker per gateway and not per consumer, we can think in terms of shipping this gateway registration mechanism with helia or a hosted esm, which loads the gateway in an iframe, but that needs to conform to this caveat. i.e. the page will need to provide respective headers to allow browsers to register the worker from the iframe-context.

[..] The gateway needs to be loaded explicitly which can call navigator.serviceWorker.register().

Yes, just like you noted, the idea is the gateway will return HTML that loads JS responsible for registering SW.

Initially, we would be doing that (A) on HTTP 502 and 504 error pages (show a button, which when clicked, registers worker, and reloads page). This allows us to figure out SW parts without having to implement full gateway (we only implement flat files).

I feel the users should be oblivious to the fact where the content is being loaded from. This should be local first by-default and recover from a public gateway if needed. [..] loads the gateway in an iframe [..]

Yes, that would be the endgame for same origin requests send by web browser for the root document. But I urge everyone to NOT focus on that, and instead prioritize MVP that will make (A) HTTP error recovery UX pleasant and useful.

Rationale: start with smaller scope, make things robust first, Reimplementing the full gateway spec in JS to support for anything other than flat deserialized files will be a lot of work, and we need to have a reliable way of testing (gateway tests are wip).

Dropping some thoughts on core pieces:

Fetching data
- Bitswap over libp2p connections
  - WebTransport transport
    - WT enabled by default since Kubo 0.18, currently ~30% of network , will grow over time
    - Direct fetch should work fine in Chromium, other browsers
- "Bitswap" over HTTP Gateways
  - this could be a fallback of last resort (e.g. no libp2p providers)
  - verifiable requests over HTTP
  - current gateway + https://github.com/ipfs/public-gateway-checker/blob/master/src/gateways.json
  - check if any gateway has data cached via HEAD request with Cache-Control: only-if-cached, if none, pick random one.
Delegated routing (for bitswap over libp2p)
- Providers routing: IPNI and DHT proxy at cid.contact (IPIP-337 with ?cascade=ipfs-dht)
  - IPNS and Peer routing
  - IPNS routing can be handled by any Gateway endpoint supporting IPIP-351 (shipped with Kubo 0.19-rc1 already) OR Kubo RPC at `https://node[0-3].delegate.ipfs.io/api/v0/dht/get
    - In the future, also IPNI endpoint after IPIP-379 is finalized (spec is will wip),
  - Peer routing on IPNI is missing, there may be IPIP for this in the future
    - For now we have to use /api/v0/dht/findpeer RPC at existing delegate nodes used by js-ipfs (https://node[0-3].delegate.ipfs.io) – this is what bifrost-gateway does for IPNS routing

2023-03-17 Discussion in weekly standup

serviceworker should be able to take request that was going to service worker and send somewhere else..
not hosting on gateway to start, simple html page that can be rendered locally in browser.

End Goal

basic plan

simple html page that can do content routing: accept URL/CID in input element, return cat jpg beneath it (result of IPFS.get(URL/CID))
accept input element info via URL to the html page, should auto-fill input element and submit; render cat jpg beneath it
- Then support video rendering?
migrate code to library and demo to using service-worker
service-worker is pluggable for different retrieval methods (bitswap, libp2p stream, libp2p graph...?)

potential showoff items

Implement a brand new (madeup) transport and implement in demo - show users they can do this without going through bureaucracy.
Allow graph based requests over http/whatever

workstreams:

Demo of app that fetch blocks over http/bitswap from providers (direct).. verify blocks, render content
Use same logic, but over libp2p stream.

initial action items

[ ] @aschmahmann to provide basic diagram of "IPFS components" necessary to make this work
[ ] @SgtPooki to build demo repo, html page and basic layout that we can start working in.
[ ] @MarcoPolo to build "fetch over libp2p"

open questions

[ ] can service worker intercept Libp2p requests? (foreign-fetch).
- Can we implement browser hash URL standard for translating a local request to libp2p so service worker intercepts it (http://mysite.com/ipfsBrowser#libp2p:stuff/here -> intercepted by service worker, does the things)

callouts

will bundle with webpack and build locally runnable site (npm start) at first. webpack should help building a single "deployable" html page that can simply be ran locally.

ipfs / in-web-browsers