search

nasa-fornax / fornax-documentation

Documentation for Fornax Science Console.
https://nasa-fornax.github.io/fornax-demo-notebooks/#user-documentation
BSD 3-Clause "New" or "Revised" License
2 stars 3 forks source link

Make suggestions for where users can store data #25

Open jkrick opened 6 months ago

jkrick commented 6 months ago

Users will need more than the currently allocated 10GB of storage. What are the options for users to store data?

Yes, one answer is to not download data, but while we are working on changing workflows to be cloud based, what are interim solutions for needing to store data.

zoghbi-a commented 6 months ago

I agree, 10GB is not enough. From your experience, is the large disk space needed for final product data? temporary intermediate products? or software?

troyraen commented 6 months ago

@zoghbi-a all of the above, but I think the largest is probably final product and the most frequent largish thing is intermediate products. We are trying to put large final products in the shared /efs directory.

troyraen commented 6 months ago

We've tried a few options so far. I think they all have benefits and pain points.

I'd like to hear from others what they personally prefer, and from infrastructure folks what their recommendations are.

jkrick commented 6 months ago

Is the shared /efs directory also open for "general" users? I had assumed that was just for us to provide necessary files for the users.

I don't know how to access Box from the SP. Dave Shupe tried with our local SP before he moved to a different project and he didn't have any luck. Things might have changed, but that seemed a dead end at the time.

For google drive I don't know of a way to write a file directly to google drive (and thereby not need to save it first followed by moving it to google drive and thereby not achieving the goal of needing less space). Is that possible in python?

zoghbi-a commented 6 months ago

One solution could one that is similar to what we have on Sciserver. Users are given 20GB of persistent storage for code and data products (e.g. final data, plots, spectra etc), and temporary large disk space (~tens of TB) that can be used for intermediate data. This storage is available during a session (we haven't defined what a session is for Fornax), and may persist for 1-3 months depending on demand.

We may have to experiment with something like this and then decide based on that.

@sawdog?