Create persistent per-user folder on HPC and mount into singularity containers

For each user, we create a persistent folder at a location other than scratch (configurable in json), folder name could be jupyter_username>@<jupyter_instance_url. This per-user folder will be mounted into each container at the same local path, such as /compute_persistent.

The use case is, user submits a job A, which dumps some data into the persistent folder; and later user could submit job B to read data from the persistent folder.

Background: current core support specifying additional folders to be mounted to containers on HPC, this is configurable in json. but this is not a per-user mount. it is a shared mount. all users see the same folder in their singularity containers (and could change them as current mount type is RW). this ticket is to make the folder on a per-user basis

This would overlap with some of the problems being addressed by the "reusable data" work. But I think the above might be a quick "hack" to enable "reusable data" in a short term.

cybergis / cybergis-compute-core

Create persistent per-user folder on HPC and mount into singularity containers #58