unixfs: add file (extended) attributes, metadata

[mildred]

I think we should have the ability to add metadata to unixfs files. This could be served as (extended) attributes on fuse mounts and as headers (such as Content-Type) for the HTTP server. Tis would be an extension in unixfs.proto. We should discuss what form they take : string key, string value or more complex.

[whyrusleeping]

The way we have discussed to accomplish this is to make a 'metadata object' that wraps the file you wish to add metadata to

example (in json):

{
    "filetype":"mp3",
    "mp3 headers": {...},
    "data" : "<hash of actual file>",
}

[mildred]

But how do you know that you are dealing with a metadata object, and not a unixfs entry. and what's the advantage of introducing another level of indirection, this will only make it slower to fetch a file, don't you think?

[jbenet]

But how do you know that you are dealing with a metadata object, and not a unixfs entry

The metadata object can have its own type identifier (i.e. letting you know -- for example -- that it's a MIME Type Object, which embeds a link and a MIME type.) This is very tricky to get right but the important thing here is to ensure that the model remains extremely flexible to application use cases, and easy to use in the general case (i.e. being explicit with types and including lots of commonly used data structures with standard ipfs distros). Note that the "type" of an object can itself be a link to an object in ipfs defining a grammar (or something equivalent). The point is to make it possible for objects to be truly self-describing.

this will only make it slower to fetch a file, don't you think?

Yeah, but we can actually do a lot speeding up how we replicate files.

[wking]

On Fri, Jan 23, 2015 at 08:18:42AM -0800, Juan Batiz-Benet wrote:

But how do you know that you are dealing with a metadata object, and not a unixfs entry

The metadata object can have its own type identifier…

More discussion on “how do you know what type of object you're looking at?” in ipfs/ipfs#36.

[zignig]

+1 mime type.

[joamaki]

As a +1, I would be interested in seeing symbolic links and file permissions supported. This would allow using IPFS as a root filesystem and would be quite neat, for example for mounting a snapshot /nix/store in NixOS for doing immutable deployments. If it makes sense running root filesystem off of a FUSE mount is another subject...

[whyrusleeping]

@joamaki i'd be interested in finding someone with experience using fuse in a 'mission critical' scenario (such as using it for a rootfs).

Permissions and symlinks are definitely one the TODO list. permissions just waiting on implementation, and symlinks waiting on a bit of discussion for how to implement them.

[jbenet]

I have wanted this for so long!!! yes, let's get these supported asap. @whyrusleeping let's talk about this tomorrow.

[whyrusleeping]

@jbenet depends on your definition of tomorrow, but lets still talk about it

[xiaods]

any update?

[kevina]

@xiaods Please see https://github.com/ipfs/ipld-unixfs for the next version of unixfs, which will eventually have support for metadata.

[xiaods]

@kevina thanks for your hints. it rocks.

[xiaods]

@kevina there is not found code in ipld-unixfs repo. do you know the reason?

[kevina]

@xiaods the link works for me

[Kubuxu]

@xiaods work there is in issues and/or PRs where we are trying to draft a spec.

[xiaods]

@kevina i mean there are not some code on the repo. https://github.com/ipfs/ipld-unixfs

[ec1oud]

It's been 7 years. I see the new idea is to make kubo into a wasm container and give up on common standards like this. In the big picture that may be worthwhile, but I'm skeptical too: I think it may result in atomization of ideas and silo-ization of applications, and result in more years of delays, rather than getting us the ability to build real applications now. UnixFS v2 seems to have never been implemented. I'm looking through go-unixfsnode go-unixfs go-ipfs-files and I don't see any hints about how to add file attributes or metadata so far. Something concrete needs to be done: enough hand-waving. We need to be able to augment files with extended metadata somehow. Ideally, arbitrary IPLD. How are we going to get there? Or am I missing something about how it's already possible?

[grexe]

this is really sad and a missed opportunity, as I'm looking for a decentralised filesystem like IPFS but need custom filesystem attributes support for my project...