search

n0-computer / beetle

Other
34 stars 15 forks source link

raw block response fails hash verification #98

Closed Arqu closed 1 year ago

Arqu commented 2 years ago

When running as a racing gateway iroh fails the the block hash verification for raw. bafybeifffvqarrkavsdg664bluuci3ojmmda6st73icfjgzqbb7ffhlkwi

Arqu commented 1 year ago

@b5 think I got to this. To confirm, can you possibly recall or share the exact http_resolvers address you passed in?

Arqu commented 1 year ago

So, this is fun. When fetching from an iroh gateway, you always get the data and just the data. On ipfs.io for example if you do a ?format=raw which we use for racing gateways, the data seems prefixed with:

lidel commented 1 year ago

Unable to reproduce, gateway returns exactly the same bytes as getting raw block:

$ ipfs block get bafybeifffvqarrkavsdg664bluuci3ojmmda6st73icfjgzqbb7ffhlkwi > expected
$ curl "http://127.0.0.1:8080/ipfs/bafybeifffvqarrkavsdg664bluuci3ojmmda6st73icfjgzqbb7ffhlkwi?format=raw" > local_curl
$ curl "https://ipfs.io/ipfs/bafybeifffvqarrkavsdg664bluuci3ojmmda6st73icfjgzqbb7ffhlkwi?format=raw" > ipfs.io_curl

$ diff -s expected local_curl
Files expected and local_curl are identical
$ diff -s expected ipfs.io_curl
Files expected and ipfs.io_curl are identical

@Arqu perhaps you are comparing different things, such as dag-pb[Data].Data field with the entire dag-pb block? ?format=raw will return the entire block as opaque bytes, and for dag-pb block it means entire outer protobuf.

ps. You may want to look at https://github.com/ipfs/specs/pull/331 which explains internals of dag-pb blocks, but ?format=raw does not care about codec, it does not parse the block, it only return bytes that will hash to the multihash in the requested CID and that is all.