Blockstore on all dagstore cids

filecoin-project / dagstore

a sharded store to hold large IPLD graphs efficiently, packaged as location-transparent attachable CAR files, with mechanical sympathy

Other

42 stars 24 forks source link

Blockstore on all dagstore cids #116

Closed aarshkshah1992 closed 2 years ago

aarshkshah1992 commented 2 years ago

For IPFS <-> Filecoin interop.

A blockstore that can serve retrieval for any cid across all shards that the dagstore has.

TODO

[ ] Add fuzzed test.
[x] GetSize should use the underlying blockstore and not load the entire block in memory.
[ ] When a blockstore is removed from the cache it is closed. Use reference counting to ensure that the blockstore is not closed while there is still an outstanding operation against the blockstore

raulk commented 2 years ago

Is there a reason that this needs to be a member of the DagStore?

aarshkshah1992 commented 2 years ago

@raulk

Have renamed it as IndexBackedBlockstore in a new indexbs package.
Yeah, I think it dosen't need to be a member of dagstore. Have moved it outside.
Shards are released when they are evicted from the lru blockstore cache. See https://github.com/filecoin-project/dagstore/blob/8d9286a428a423f85648493c514dd47dddad5d5f/indexbs/indexbacked_bs.go#L44.

aarshkshah1992 commented 2 years ago

@raulk @dirkmc

There was a race here where two Get calls for the same shard key create two shard accesors and only one gets added to the lru cache and eventually relased whereas we lose the reference to the other one without ever releasing it (because multiple add calls to the lru cache for the same key only updates the priority of the item).

I've fixed this with striped locking based on shard key to synchornize access to the lru cache.

aarshkshah1992 commented 2 years ago

@raulk Please can you take a look ?

MichaelMure commented 2 years ago

May I ask what's the status of this work?

Wondertan commented 2 years ago

Hello. We would like to use DAGStore together with a top-level index. We also need a Blockstore over the top-level index to serve sampling requests to our light clients, so we would like to have this merged not to reinvent the wheel:

What else needs to be done here to say it's done?
Can we somehow facilitate the process?

Kindly ping @raulk, @dirkmc, and @aarshkshah1992. I know you are busy with FVM and Boost things, but this should not take too much time from you. Thanks!

willscott commented 2 years ago

@hannahhoward probably has the most recent context here, and I think we are hoping to finish the work needed to get this finished.

dirkmc commented 2 years ago

@Wondertan @MichaelMure we are hoping to get it merged this week