search

data-preservation-programs / singularity

Tool for onboarding data to the Filecoin Network
Other
18 stars 15 forks source link

Content Provider / GraphSync #16

Open xinaxu opened 1 year ago

xinaxu commented 1 year ago

Description

Make Singualrity act as boost that serves graphsync retrieval while offloading data to storage providers

The database backed block store can be used to serve blocks.

If the data source has been exported to CAR files, those blocks will be served directly from the CAR. Otherwise, block retrieval will be a range HTTP request to the original data source which may be inefficient as it will be translated to range request.

anjor commented 1 year ago

@xinaxu can you explain why we want to support graphsync retrievals here? I was under the impression we were moving away from supporting graphsync in general?

xinaxu commented 1 year ago

To me this is a low-hanging fruit because it is easy to implement given we already have the blockstore that backs the bitswap retrieval. We don't have to to prioritize this item.