ipld / go-car

A content addressible archive utility
Other
145 stars 44 forks source link

Add an `AbsoluteIndexOffsets` option for readonly blockstore #490

Closed willscott closed 10 months ago

willscott commented 10 months ago

Allow opening a blockstore with an index that has offsets from the beginning of the car file, rather than from the beginning of the carv1 contained file.

Absolute indexes of this type are generated by boost. Once the car is stored to a remote location separate from the index, it's unclear if it was a v1 or v2 car. Using an absolute index means individual CIDs can be efficiently fetched without re-parses of the header. The carv2 blockstore is used over this situation by boost for cases where a blockstore interface is needed. in this case, we would like to be able to support a mode where the reader offset for index reads doesn't offset based on the indicated start of the data section in the carv2 header.

rvagg commented 10 months ago

@willscott did you close because you found a way around this or do we need to explore this further? there's probably a few options here, but if we can get by with a quick parse of the start of the file then that would be the easiest.

willscott commented 10 months ago

because it isn't worth blocking on a discussion at the car semantics level.

Here's my ugly work-around at the boost level: https://github.com/filecoin-project/boost/pull/1651/files#diff-701447f1a6bf0a69bba3d85f6e2cd6f23373748ea9ddfc7a6de95ebeb4dfc67fR663-R687