ipfs-inactive / bifrost-gateway

Experimental gateway with delegated backend. No longer maintained, consider migrating to https://github.com/ipfs/rainbow/
Other
41 stars 20 forks source link

meta: GRAPH_BACKEND fixes and latency improvements #88

Closed lidel closed 1 year ago

lidel commented 1 year ago

This is a meta issue for tracking things we can do to improve correctness/latency.

Spec conformance

### Tasks
- [x]   [IPIP-402](https://github.com/ipfs/specs/pull/402) conformance  tests for `dag-scope=block|entity|all`  →  [gateway-conformance#75 added tests](https://github.com/ipfs/gateway-conformance/pull/56)
- [x]   [IPIP-402](https://github.com/ipfs/specs/pull/402) conformance tests for `entity-bytes=from:to` →  [gateway-conformance tests](https://github.com/ipfs/gateway-conformance/pull/56) 
- [x]   [IPIP-402](https://github.com/ipfs/specs/pull/402) conformance tests for `entity-bytes` with missing block scenario to confirm `/ipfs/cid?format=car&bytes=from:to` works fast for big UnixFS file, and does not block until the entire thing is prefetched to some L1 →  [gateway-conformance#85 added tests](https://github.com/ipfs/gateway-conformance/pull/85)
- [x] Run gateway-conformance on Saturn CI: https://github.com/filecoin-saturn/L1-node/issues/447
- [x]  Saturn L1 pass `dag-scope` tests  [PR](https://github.com/filecoin-saturn/L1-node/pull/449)
- [ ] **(TODO)** Saturn L1 DOES NOT pass `entity-bytes` range request tests  [CI fails here](https://github.com/filecoin-saturn/L1-node/pull/449)

Perf. Improvements

### Tasks
- [ ] https://github.com/ipfs/specs/pull/402
- [ ] https://github.com/ipfs/specs/pull/412
- [x] **(PRECONDITION)** @aschmahmann IPIP-402 implementation in Boxo (https://github.com/ipfs/boxo/issues/283) so can verify tests and performance improvement ideas without doing a Rhea bifrost-gateway deployment.
- [x] **(OPTIMIZATION)** @aschmahmann https://github.com/ipfs/bifrost-gateway/pull/160
- [x] **(OPTIMIZATION)** @aschmahmann  implement https://github.com/ipfs/bifrost-gateway/issues/62
- [x] **(FIX)** ensure we use HTTP/2 when possible (#98  or https://github.com/ipfs/bifrost-gateway/pull/95)
- [x] **(FIX)** @lidel correctness degrading over time (https://github.com/ipfs/bifrost-gateway/issues/92)
- [x] **(OPTIONAL LATENCY FIX)** @lidel  / @hacdias / @aschmahmann   30m timeout for CARs is not enough, implement timeout-since-last-bytes-received that drops request/response processing if no new bytes were sent by L1 for [tbd] amount of time – stub in https://github.com/ipfs/bifrost-gateway/issues/91
- [ ] **(OPTIONAL CORRECTNESS FIX)** @lidel /  @aarshkshah1992   (content routing optimization, may improve correctness) when fetching path  remainders or single blocks, pass content path affinity to L1 and forward to Lassie in Ipfs-Path-Affinity HTTP header as a hint to augument content routing in situations where internal blocks are not announced on DHT/IPNI
- [ ] **(OPTIONAL LATENCY FIX)** (perf optimization) make caboose do more [RAPIDE-like retrieval](https://www.notion.so/MTB-WG-Meeting-6-RAPIDE-4ceba2dfb2fc4ceaa49f1cd566ca104f) and split DAG into parallel fetches?

Note: these things aren't blocking for enabling GRAPH_BACKEND=true. The blocker is https://github.com/ipfs/bifrost-gateway/issues/92

BigLep commented 1 year ago

2023-05-09 conversation: @BigLep to create the issue for implementing IPIP in Boxo. Needed for:

lidel commented 1 year ago
BigLep commented 1 year ago

2023-06-22 maintainer conversation: @aschmahmann is going to update the content of this issue to describe what's remaining that we're going to do.

BigLep commented 1 year ago

2023-10-03 conversation: we did all the mandatory things.