search

paritytech / substrate

Substrate: The platform for blockchain innovators
Apache License 2.0
8.39k stars 2.65k forks source link

IPFS protocol for querying block data #6075

Open gavofyork opened 4 years ago

gavofyork commented 4 years ago

Substrate should provide an IPFS implementation over its libp2p to allow for querying certain data from chains. A (the?) offchain worker local storage should be able to be given data to store which can be indexed quickly with a Blake2 hash. The hash should be able to be used to query via IPFS (or is it bitswap protocol?) over libp2p in order to retrieve its preimage.

Rough API endpoints:

Two new OCW APIs:

Preimage lookup over libp2p for anything published up until block number until and not since unpublished.

Usecase: it should be possible for any IPFS browser node to locate and retrieve unexpired data published through an OCW pubish call.

expenses commented 4 years ago

@dvc94ch How in-development is libp2p-bitswap? I have a branch here: https://github.com/paritytech/substrate/tree/ashley-bitswap that attempts to integrate it into substrate, is libp2p-bitswap stable enough for us to do this, or does it need more work?

dvc94ch commented 4 years ago

It works ok, although I do need to spend some time on it. There are optimizations in https://blog.ipfs.io/2020-02-14-improved-bitswap-for-container-distribution/ that aren't implemented, and there's there will likely be incompatibilities once you reach a high throughput as currently the messages aren't split into whatever the maximum message size is.