srivarra
commented
1 year ago @JLrumberger @ngreenwald
Options
Open Science Framework (OSF)
Main Features
- Simple to upload and store data.
Pros:
- Free
Cons:
- No data explorer view, can only look at file names
- Most of the data here is on the smaller scale, not sure how it'd work with TB-sized datasets.
- 5GB storage cap per file, 50GB per component
- We'd have to chunk the dataset out.
Takeaway: It's free, but we pay the price with less than seller functionality.
XetHub
Main Features
- Git like command interface designed for large datasets, allows branching, merging, diffs, etc...
- Some more advanced features include built in data visualizations
Pros:
- AWS S3 levels of scaling.
- Works with any file formats
Cons:
- Not free, $249/month for 250 GB of storage, and an extra 5 cents per GB
- A 3TB dataset would cost $386.5/month just for hosting
- Data Transfer is also not free, 500 GB of transfer with the $249/month storage plan
- We would have to develop our own portal of some sorts to allow others to view, explore and download the data, it's only for storage basically.
Takeaway: Powerful, isn't really a batteries included product. Also pretty expensive.
Open Microscopy - Image Data Resource
Main Features
- Backed by Open Microscopy
- Ability to add Jupyter notebooks in conjunction with the datasets to allow for exploratory analysis (runs on Binder)
- Includes an image viewer built in to the website.
Pros:
- Free
- Well tested, already contains 300+ TB of imaging datasets
Cons:
- Has an approval process / has to meet certain criteria
- A HuBMAP like procedure
Takeaway: Seems to be good, but it's more complicated than "zip up the data and upload to a server".
Zenodo
Main Features
- Contains data, and publication information associated with each dataset.
- Uses data versioning
Pros:
- Used by many ("it just works")
Cons:
- Datasets are limited to 50GB chunks (therefore we would have to break apart each dataset, maintain, and users would have to keep track of which is which).
- Can request for higher quotas.
Takeaway: The dataset size limitation is extremely unfortunate, as everything else sounds pretty good.
We host it ourselves and maintain it
Main Features:
- Full control of upload, ingestion, scale, etc...
Pros:
- Opens us up to a plethora of storage solutions such as using databases like Planetscale, Weaviate.
- Allows us to serve the data on the edge, close to anyone who may potentially want to access it making transfers faster.
- Or we can take advantage of Stanford's Resources here, such as Virtual Servers
Cons:
- Depending on the number of services we end up using, this can be very expensive.
- We have to maintain all of it, this would be a significant time investment.
Takeaway: Lots options to try here, we can host the data and see what works, also means a lot of time and effort will be needed to get something up and running.
ngreenwald
We have a number of projects in the works in the lab that will have very large accompanying datasets (300-3,000GB). We'll want to figure out what the best way to make this data available for download is, for people who decide they want access to the raw data.
The Bodenmiller lab does Zenodo for hosting their data. The Schuerch dataset is available via TCIA. It would be great to do a survey of what options are available, what are the pros and cons, etc so we can figure out a standardized lab plan.