ipfs / specs

Technical specifications for the IPFS protocol stack
https://specs.ipfs.tech
1.17k stars 232 forks source link

Update UnixFS specification #316

Open Jorropo opened 2 years ago

Jorropo commented 2 years ago

We need:

BigLep commented 2 years ago

Cc @rvagg @dignifiedquire that Kubo maintainers are going to take the first stab at getting this written in September. Feel free to watch or leave any notes.

b5 commented 2 years ago

Our latest set of trials & tribulations from Iroh: https://github.com/n0-computer/iroh/pull/198 and our running doc of papercuts: https://number-zero.notion.site/UnixFs-742339892d9c47d5b79f4f942e661bbf

Jorropo commented 2 years ago

@b5 about https://github.com/n0-computer/iroh/pull/198 I think balanced tree is not in the spec. Or at least, if someone really care about it, it's a non authoritative part of the spec.

As long as you get your file sizes rights, and the merkle dag is correct (mean that a correctly build decoder successfully rebuild the original content). You can use whatever scheme you like.

b5 commented 2 years ago

Sure, maybe not an authoritative part of the spec, but as Lidel pointed out in the implementers call yesterday, there are many things that would be good to suggest within spec documents that give implementers hints so the don't footgun themselves.

No one says the dag needs to be balanced. Everyone ends up implementing a balanced tree at some point.

lidel commented 2 years ago

Some additional asks, based on real world problems I've seen:

willscott commented 2 years ago

Another ask is having a recommendation for how to add non-canonical / extensions to the spec e.g. systematization for extra metadata

Jorropo commented 2 years ago

I want to do this, but as a followup IPIP.

First I start by describing what Kubo do without any new inovation, then we see what we can improve. It will be impossible to review if we mix and match old and new stuff.

BigLep commented 2 years ago

Here is the in-progress PR: https://github.com/ipfs/specs/pull/331