Design data storage - Githubissues

bhklab / ORCESTRA-api

https://bhklab.github.io/ORCESTRA-api/

2 stars 0 forks source link

Open jjjermiah opened 5 months ago

jjjermiah commented 5 months ago

[ ] need to figure out what to do with private data
[ ] need to figure out what to do with processed (now can be public) data
[ ] design of GCS storage
[ ] feat: programmatically get access to GCS Storage blobs used in each pipeline

Nice To Have:

being able to "containerize" or "code-ocean" an entire workflow after it is complete combined with the pipeline code to be downloaded via Zenodo alongside the final object.

jjjermiah commented 5 months ago

re: processed (public-use), creating a bucket at gs://orcestra-rawdata (can re-name if we come up with something better)

uploading data processed on H4H for gCSI there under gs://orcestra-rawdata/gCSI

Will then download files from here in pipelines.

bucket will be have public read permissions but only authenticated write permissions so anyone can download.

My only concern really is what if someone just downloads the same file like 1000 times and destroys our egress lol.

jjjermiah commented 5 months ago