Register external files and datasets - Githubissues

clowder-framework / clowder2

Clowder v2 (in development)

Apache License 2.0

12 stars 6 forks source link

Register external files and datasets #665

Open lmarini opened 1 year ago

lmarini commented 1 year ago

File:

A file is an external URL. Store credentials in the backend. On file/instance/source basis?
A file is in another clowder instance.
A file is in an s3 bucket different from the clowder one. Credentials? Presigned URLs?
Local/mounted filesystem
Globus data transfer

One term we could use for some of those is "shortcuts."

Datasets:

A folder in an s3 bucket/local filesystem becomes a dataset in clowder. Does folder structure come from that? how do sync?
```
### Tasks
```
[x] Local/mounted filesystem
[ ] remote URLs
[ ] remote APIs
[ ] S3 bucket

max-zilla commented 11 months ago

I created a draft README with some ideas here: https://github.com/clowder-framework/clowder2/blob/Register-local-files/docs/source/storage.md

Covers:

local filesystems
sync'd folders
storing URLs
S3 storage

Priority will be local filesystems, then remote URLs.