Add support for deals using a data segment index

[willscott]

A deal may, rather than a single car, be a series of concatinated car files with a segment index at the end containing an inclusion proof. When such an index is found, we will allow reading multiple cars and taking valid cids found in all of them as the contents of the overall deal.

ref: https://github.com/filecoin-project/boost/issues/1258

[willscott]

filing https://github.com/filecoin-project/go-data-segment/issues/9 to see if we can get a fixture to use for validation

[Wondertan]

This PR brings Filecoin implementation detail to Dagstore and multiple new dependencies. Is there a way to implement this using Dagstore's API on the layer above, rather then polluting its internals?

[willscott]

That's a good point, though carv2 indexes are in a way an equally opinionated existing equivalent. I'll see what it would look like for the 'get an index for this shard' to be fully handled by the caller

[raulk]

I think there's an implicit decision being taken here, which is to model segments as indexed entries over a single shard (= storage deal in the Filecoin context), instead of as dedicated shards. Establishing a shard for every segment was not possible in the past since there was no standard way of delineating logical units within a Filecoin deal. With PoDSI, this is now possible. Making shard == segment could let us keep the dagstore Filecoin-agnostic, but I suspect is not viable without additional complexity. @willscott could you walk us through the tradeoffs here?

[willscott]

Making shard == segment could let us keep the dagstore Filecoin-agnostic, but I suspect is not viable without additional complexity. @willscott could you walk us through the tradeoffs here?

shard == segment

• shard parse logic simplified as a single car file
• leads to caching / transfer at sub-deal level, better reflecting the potential for different usage patterns of different cars packed together in a deal.
• unsealed transfer of cars is no longer alone sufficient to re-create the deal - also need to to know order/offsets of them at least.

shard == deal

• no change in mental model of what files-on-disk are for SP operators
• the 'deal' is the object that is created / destroyed atomically
• unsealing is only possible at a deal granularity
• deals will remain a mix of single car vs segmented - so some file management will have to remain at a 'deal level' anyway

[raulk]

Thanks! I think the case for shard == segment isn't strong today, but it may strengthen over time as selective unsealing becomes available, PoDSI gains traction, and segment-based access patterns become a bottleneck and the next frontier for optimization. Agree with the implicit decision made here, but +1 to @Wondertan's request to invert the control of the index population, so that the API caller can feed the segments.

[willscott]

cc @Kubuxu