ipfs / go-unixfsnode

An ADL IPLD prime node that wraps go-codec-dagpb's implementation of protobuf to enable pathing
Other
11 stars 5 forks source link

Add function to determine the size of final unixfs DAG given raw bytes #58

Open masih opened 1 year ago

masih commented 1 year ago

It would be fantastic to have a function that given the size of raw bytes would cheaply and quickly calculate the size of the final unixfs DAG. This would be useful in cases where the final DAG will be written out into a fixed sized bucket and the user wants to guarantee that each bucket will have the full raw bytes, e.g., FileCoin pieces of size 32GiB.

Other uses of this include pre-allocation of resources or more efficient buffer management depending on the final DAG size for large-scale data intensive applications.

willscott commented 1 year ago

is what you want the size of the dag when serialized as a car? (including a carv1 header?)

masih commented 1 year ago

CAR sections only, for a given LinkPrototype please (though I see that's fixed?, which is fine).

willscott commented 1 year ago
masih commented 1 year ago

Is there a reason not to include the carv1 header?

In my specific usecase, a CAR may contain multiple unixfs DAGs.