search

dandi / dandi-cli

DANDI command line client to facilitate common operations
https://dandi.readthedocs.io/
Apache License 2.0
21 stars 25 forks source link

schema: AssetMeta.contentUrl - move to BareAsset? #573

Open yarikoptic opened 3 years ago

yarikoptic commented 3 years ago

"Inspired" by @rly post in slack about working on an analytics demo case + ongoing discussion about future Python API (#249 ) dandi-cli design.

contentUrl can be file:///<PATH> for a local asset. Then (if pynwb/hdmf can take those as well, may be already does) analytics code could just use Dandiset abstraction to ask for a location (local file:// or e.g. url on s3 for ros3) and interact with the asset without needing any change to analytics code, and just by swapping which Dandiset (local or remote) it would use.

WDYT @satra @jwodder ?

satra commented 3 years ago

the only issue i would consider would be privacy. but the abstraction piece sounds good. i think we should still aim towards data -> nwb -> dandiset -> write scripts on top of this. then the scripts could work on data through the Dandiset python api (at least for python scripts).

yarikoptic commented 3 years ago

sorry -- what "privacy" aspect? those should be anyways replaced by the server with its own contentUrl. Or should we aim for full git-annex implementation here on tracking URLs? ;)

satra commented 3 years ago

i thought you wanted to support any local path not just relative to the dandiset root. file:/// is absolute no?

yarikoptic commented 3 years ago

yes, but those will be provided only for local dandisets