search

CINPLA / exdir

Directory structure standard for experimental pipelines.
http://exdir.rtfd.io
MIT License
72 stars 13 forks source link

Add support for internal and external links #1

Open dragly opened 7 years ago

dragly commented 7 years ago

Similar to HDF5 soft, hard and external links. Should be saved in meta.yml similar to this:

type: link
link: 
  type: hard/soft/external/internal
  target: /some_group/some_dataset
neuromusic commented 6 years ago

any work on this front, @dragly?

dragly commented 6 years ago

Hi, @neuromusic!

Thanks for bringing this issue up again. We have not yet implemented links in exdir, but have been discussing it since this issue was created. We would be happy to prioritize implementing links soon if you and others find it useful.

Soft links, both internal and external, should be fairly easy to implement. This will likely look something like the above in the exdir.yaml file and only requires us to update the Group.__getitem__ to handle links and return the appropriate SoftLink or ExternalLink objects.

Hard links are harder to implement because there is no concept of a central object registry in exdir. This means that we currently have no elegant way to keep track of the number of references to a given object, and a way to update those references if the object is moved. While we could create a central registry, the hard problem is to design an elegant way to maintain this outside the programming APIs. The goal of exdir has been to make the structure viewable and editable also from a file browser. This will be tricky to do with hard links. I think the most appropriate way of implementing hard links would be by moving objects that are referenced by a hard link into a hidden repository, similar to how Git handles objects, and leave references to the objects in the file tree. This has the disadvantage that it will no longer be easy to explore the data and metadata in a file browser, although the structure would still be viewable. In addition, we would need to add some mechanism to clean up objects that are no longer referenced by any links. I'm reluctant to adding hard links this way unless there is a really strong need for them. However, there might be better ways to implement them and if anyone has a good idea, we would love to hear it!

Feel free to share how you would like to use links in exdir so that we can try to take those considerations into account.