yarikoptic
commented
7 months ago Current solution approach: development of efficient WEBDAV interface which could then be used as an already supported "standard" protocol to implement desired by OSDF caching etc. Current webdav "frontend" is developed at https://github.com/dandi/dandidav and deployed at https://dandi.centerforopenneuroscience.org
Proposed change
yet to be figured out, but a user asked
Please clarify whether data saved in DANDI can be reached and used through OSDF. OSDF: https://osg-htc.org/services/osdf
From https://osg-htc.org/docs/data/stashcache/overview/: The OSG operates the Open Science Data Federation (OSDF), which provides organizations with a method to distribute their data in a scalable manner to thousands of jobs without needing to pre-stage data at each site.
Some caching solutions are part of the framework: https://osg-htc.org/docs/data/stashcache/install-cache/
Overall -- it sounds like a solution allowing making (public or private ) local data resources available to remote compute resources through unified interface with some caching to make it efficient. So, if someone tackles it I guess OSDF DANDI users would benefit form
Wasn't sure where to seek clarification yet... or what it would entail for a DANDI resource to expose itself as part of OSDF. If there was some other example of a public resource exposed to OSDF, might have provided a template to follow. Searching github got to https://github.com/PerilousApricot/docs/blob/de67fc125efb334ca67b011bde179689f035ffda/docs/data/stashcache/vo-data.md?plain=1#L3 so I will email
help@opensciencegrid.orgfor either such integration makes sense if there are examplesAlternative options
regular means to access from S3 directly, via DANDI API, via datalad/git-annex, dandi-cli
Who would use this feature?
Users of OSDF
(Optional): Suggest a solution
yet to be figured out if there is a problem to be solved ;)