search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.03k stars 3k forks source link

HTTP fetch of fs-migrations should use CAR #9159

Closed lidel closed 6 months ago

lidel commented 2 years ago

Version: 0.14.x

Fetching migration data from IPFS was added in https://github.com/ipfs/kubo/pull/8064, but we did not have https://github.com/ipfs/go-ipfs/pull/8758 at the time and HTTP fetch is still delegating trust to the gateway.

I consider that a bug: migrations should be fetched in trustless mode as a CAR by requesting them as ?format=car and verifying every block before applying the migration.

This allows us to use third-party gateways af fallback and/or in scenarios where ipfs.io is blocked by ISP etc.

BigLep commented 1 year ago

@Jorropo will tackle this after some related light-client work he is doing.

BigLep commented 1 year ago

@lidel : is this a duplicate of https://github.com/ipfs/kubo/issues/8851 ?

I see there is also this PR: https://github.com/ipfs/kubo/pull/9250

lidel commented 1 year ago

I've closed https://github.com/ipfs/kubo/issues/8851 as a duplicate (this one has more context).

hacdias commented 1 year ago

I'm marking this as blocked by the IPIP-402. Once we have that, our life will be easier. Also, I want to add that we need IPNS resolution to solve this too. Copy of my comment:

I just took a second look at this and indeed it'll be much easier once we have widespread IPIP-0402. There are two main problems here:

  1. We need to be able to resolve distPath. If it is a /ipfs path, we're good to go. If it's a /ipns path we have to: a. Check if it's regular IPNS. If so, we can fetch record from gateway via ?format=ipns-record, validate and get the /ipfs path. b. If it's DNSLink (most likely), setup a NameSys resolver and resolve it via DNS.
  2. With the distPath being an /ipfs path, we can now request a CAR for {distPath}{filePath}?dag-scope=entity and verify if the CAR and path resolves correctly. Use that data for the migration file.

I was hoping we could do something now, but it'd just be too complicated and we're already working on IPIP-402 so I would mark this as blocked for now.

lidel commented 1 year ago

IPIP-402 shipped. Once ipfs.io upgrades to Kubo 0.21 or later, we will be able to fetch /ipfs/cid/path/file as entity+path and verify it from any gateway.

We should make it work with all gateways, so if anything is missing in CAR response, fetch it block-by-block (we have prior art in bifrost-gateway).

I don't think we should block on IPNS record resolution. We do not use it for migrations, and for that we have hardcoded CID, we only care about /ipfs/ paths: