feat: use blockstore sessions

achingbrain commented 5 months ago

Adds a configurable session cache that creates sessions based on the base URL of the requested resource.

E.g. https://Qmfoo.ipfs.gateway.com/foo.txt andhttps://Qmfoo.ipfs.gateway.com/bar.txt will be loaded from the same session.

Defaults to 100 sessions maximum with a TTL of one minute. These are arbitrary numbers that will require some tweaking.

Adds allowInsecure and allowLocal options to createVerifiedFetch factory - because we search for gateways in the routing we need to exclude those that have local or http addresses, otherwise we'll get mixed content/connection errors.

This makes testing against local gateways difficult so pass true for these options to not filter out private/insecure hosts.

Depends on:

Change checklist

[x] I have performed a self-review of my own code
[x] I have made corresponding changes to the documentation if necessary (this includes comments as well)
[x] I have added tests that prove my fix is effective or that my feature works

achingbrain commented 5 months ago

In draft because it needs the next Helia release to work and also because the abort-handling tests are skipped pending a refactor - they feature extensive stubbing of internals which have now changed.

These need refactoring to land nodejs/node#42 anyway so...

SgtPooki commented 5 months ago

FYI, i ran tests locally here and I get:

(node:47763) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 585 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit

Enabling debug messaging:

helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg session is ready +0ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg found new providers re-retrieving QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg +0ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg fetching BLOCK for QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg from https://cesginc.com/p2p/12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys +0ms
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 254 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 255 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 256 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 257 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 258 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 259 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 260 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 261 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 262 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 263 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 264 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 265 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
(node:48764) MaxListenersExceededWarning: Possible EventTarget memory leak detected. 266 abort listeners added to [AbortSignal]. Use events.setMaxListeners() to increase limit
  helia:trustless-gateway-block-broker:cesginc.com GET https://cesginc.com/ipfs/QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg?format=raw 401 +0ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg no session peers had block for for QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg, finding new providers +34ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg finding 1-5 new provider(s) for QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg +0ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg:error error retrieving session block for QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg Error: unable to fetch raw block for CID QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg
    at TrustlessGateway.getRawBlock (file:///Users/sgtpooki/code/work/protocol.ai/ipfs/helia-verified-fetch/node_modules/@helia/block-brokers/src/trustless-gateway/trustless-gateway.ts:90:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at TrustlessGatewaySession.queryProvider (file:///Users/sgtpooki/code/work/protocol.ai/ipfs/helia-verified-fetch/node_modules/@helia/block-brokers/src/trustless-gateway/session.ts:41:19)
    at raceSignal (file:///Users/sgtpooki/code/work/protocol.ai/ipfs/helia-verified-fetch/node_modules/race-signal/src/index.ts:46:12)
    at Job.run (file:///Users/sgtpooki/code/work/protocol.ai/ipfs/helia-verified-fetch/node_modules/@libp2p/utils/src/queue/job.ts:78:22) +237ms
  helia:trustless-gateway-block-broker:cesginc.com GET https://cesginc.com/ipfs/QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg?format=raw 429 +3ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg no session peers had block for for QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg, finding new providers +3ms
  helia:trustless-gateway:session:QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg finding 1-5 new provider(s) for QmVP2ip92jQuMDezVSzQBWDqWFbp9nyCHNQSiciRauPLDg +0ms

achingbrain commented 5 months ago

I think that might actually be a bug in node - https://github.com/nodejs/undici/issues/3157 - there's a workaround here: https://github.com/ipfs/helia/pull/511

achingbrain commented 5 months ago