search

ipld / go-car

A content addressible archive utility
Other
152 stars 44 forks source link

feat: support .car files with no root nodes #19

Open mikeal opened 4 years ago

mikeal commented 4 years ago

Heya,

We’re using .car files as a block store format in a few new use cases and this bit here is giving us some trouble.

https://github.com/ipfs/go-car/blob/6bca8656ee37dcc2ed011117958f64427c2ef2ab/car.go#L109-L111

We’re storing very large files across multiple .car files. One of the .car files has a root node that describes all the chunks of the file and what other .car files the blocks can be retrieved at. But the .car files that just contain a bunch of raw blocks don’t have a logical root node.

Our options right now are to either add every block’s CID as a root, which seems unnecessary and a potential performance issue on read, or to create an unnecessary vector for all the links and put that in there as well, which is really just a bunch of unnecessary data we’re shoving into the file.

It would be nice if .car files could optionally not contain any root nodes at all.

rvagg commented 4 years ago

Current loading semantics here require roots to do a dag.Walk() so this is:

  1. a request to change the implied specification of .car files to allow an empty array of roots in the header (by removing the 3 lines in question above), as we may be generating them in JavaScript too without the dag.Walk(); and
  2. a request for feedback for changing this code to allow alternate interfaces for .car creation rather than from a DAG tree extraction from a Blockstore -- potentially allowing go-car to act as an append-and-write-only Blockstore itself