search

ORNL / DataFed

A Federated Scientific Data Management System
https://ornl.github.io/DataFed/
Other
18 stars 14 forks source link

Data Refs: Allow linking to external raw data via URI/path #605

Closed dvstans closed 3 years ago

dvstans commented 3 years ago

A potential user has expressed interest in using DataFed's metadata capabilities with existing data available on external Globus servers. There is also interest in DataFed reflecting access control permissions to these external endpoints. Need to investigate feasibility of this.

dvstans commented 3 years ago

This now a required feature and critical path. Systems needs to distinguish between internal and external raw data. It should allow any path type (Globus, HTTP, filesystem) since DataFed will not ever access the data (i.e. for information only). However, if it is Globus or HTTP, the web UI could open the link in a separate window. Also consider eventual ingest of the external data (if Globus).

dvstans commented 3 years ago

Update - Only full Globus paths to external data will be supported. DataFed will not attempt to control access to raw data (it can't) and will enable upload/download functionality to all users who can read the associated record. If users do not have access, then the Globus transfer will fail. Data records will support an "external data" flag such that the source field will be used for up/download rather than an allocation. The UI will be updated to allow specification of external data and the data movement code will be updated to support all operations on such data (delete, move, change owner, etc.)