search

princeton-ddss / blackfish

Machine learning as a service (MLaaS) for busy researchers
https://princeton-ddss.github.io/blackfish/
0 stars 0 forks source link

Storage locations? #49

Open cswaney opened 1 month ago

cswaney commented 1 month ago

In general, services will benefit from, or even require, access to a file system/server.

One approach is to include an storage_location parameter with service requests. This location would be automatically mounted to containers and allow users to access files in the specified location. If they decide they want to use a file elsewhere, they would need to create a new service.

An alternative approach is to add "storage locations" to profiles and automatically mount all storage locations. For example,

[default]
type=slurm
host=della.princeton.edu
home=/home/<user>/.blackfish
cache=/scratch/gpfs/<shared>/.blackfish
storage_locations=[/home/<user>, /scratch/gpfs/<user>, /scratch/gpfs/<lab>/data]

Now, then a service is launched, we mount all these locations:

docker run -v { profile.storage_locations[0] }:/data/ext0 -v { profile.storage_locations[1] }:/data/ext1 ...

Users then have access to all of their storage locations from within the container by using the appropriate container path.