Fetch blockchain with Graphsync instead of Bitswap

anorth commented 5 years ago

Description

Graphsync is a protocol for exchanging arbitrary merkle-dag structures with minimal round-trips. It is well suited to fetching blockchain-structured graphs, capable of fetching a large number of blocks and linked structures with a single request. This has the potential to massively speed up the process of fetching the chain, when compared with the existing bitswap-based implementation.

Replace the Bitswap chain fetcher with one based on Graphsync.

Acceptance criteria

Graphsync is used instead of bitswap to fetch blockchain segments
Initial chain syncing involves many fewer round-trips than the current bitswap method

Risks + pitfalls

Graphsync is still in development. The IPLD selectors on which it depends is also still in development.
Graphsync requires explicit peer identification, so we need to maintain an explicit peer list #3054

Where to begin

See the net.BitswapFetcher and chain.Syncer.

FYI @hannahhoward @frrist @ZenGround0

ridewindx commented 5 years ago

👍

taoshengshi commented 5 years ago

Does the storage data fetching/swapping use the same graphsync mechanism? is the same IPFS network underneath the filecoin network(blockchain + storage) ? is it possible to separate the storage flow/plane from blockchain flow/plane?

anorth commented 5 years ago

This issue requests use of graphsync for exchanging blockchain data, but not storage piece data. Piece data is still exchanged using bitswap, like IPFS (although this is not a strict requirement and a client/miner are free to exchange data however they agree). So in this sense they will be separated, and I think that's a desirable feature. There are no plans to use graphsync for exchanging storage data at present.

taoshengshi commented 5 years ago

regarding to the separation feature, because of the storage nature of filecoin network, Piece data and Block data could race to use the underlay transport network and withdraw each other. But there are some deploy scenarios need to be took into considerations: 1, filecoin storage serve as bittorrent system across wide-area network 2, filecoin storage serve as enterprise storage solution in data centers(some datacenter connected across wide-area network by dedicated access),using IPFS-cluster/infra 3,filecoin serves as blockchain service,and there is a bridge connected filecoin with storage system (using IPFS-cluster/infra)

Scenario 2 & 3 support storage flow/plane from blockchain flow/plane by nature if doing by proper design. Scenario1 should support block data flow higher priority（QoS Class）than storage flow/plane.

anorth commented 5 years ago

Done in #3153, #3184, #3190

anorth commented 5 years ago

See #3232 for more testing required.

filecoin-project / venus