We’re storing very large files across multiple .car files. One of the .car files has a root node that describes all the chunks of the file and what other .car files the blocks can be retrieved at. But the .car files that just contain a bunch of raw blocks don’t have a logical root node.
Our options right now are to either add every block’s CID as a root, which seems unnecessary and a potential performance issue on read, or to create an unnecessary vector for all the links and put that in there as well, which is really just a bunch of unnecessary data we’re shoving into the file.
It would be nice if .car files could optionally not contain any root nodes at all.
Current loading semantics here require roots to do a dag.Walk() so this is:
a request to change the implied specification of .car files to allow an empty array of roots in the header (by removing the 3 lines in question above), as we may be generating them in JavaScript too without the dag.Walk(); and
a request for feedback for changing this code to allow alternate interfaces for .car creation rather than from a DAG tree extraction from a Blockstore -- potentially allowing go-car to act as an append-and-write-only Blockstore itself
Heya,
We’re using .car files as a block store format in a few new use cases and this bit here is giving us some trouble.
https://github.com/ipfs/go-car/blob/6bca8656ee37dcc2ed011117958f64427c2ef2ab/car.go#L109-L111
We’re storing very large files across multiple .car files. One of the .car files has a root node that describes all the chunks of the file and what other .car files the blocks can be retrieved at. But the .car files that just contain a bunch of raw blocks don’t have a logical root node.
Our options right now are to either add every block’s CID as a root, which seems unnecessary and a potential performance issue on read, or to create an unnecessary vector for all the links and put that in there as well, which is really just a bunch of unnecessary data we’re shoving into the file.
It would be nice if .car files could optionally not contain any root nodes at all.