Add an oem_mode for mm_jobman

[danielnachun]

After the successful deployment of our pixi/micromamba-based interactive sessions, we'd like to now create a similar system for batch jobs, where we use our ability to mount EFS volumes with pre-installed software instead of loading a larger container, or installing the software from scratch for every job.

The way we are envisioning this is that we will have an "OEM" EFS volume containing pixi and micromamba folders that will be mounted read-only as $HOME/.pixi and $HOME/micromamba. These read-only .pixi and micromamba folders will contain all of the software we use throughout xQTL in a static state, where batch jobs will not be able to modify the software at all. In some sense we can think of this a replacement for our attempts at an xqtl_full container with all the software we use preinstalled.

This approach has several key advantages over what we've done before:

• No need to build different containers for different batch jobs - all the software we need is available, and we don't have to deal with long wait times for a container image to decompress
• Very low startup times for individual jobs - in this mode all we're doing is mounting some folders and then starting the job
• Jobs don't need a large EBS volume - all the jobs will be mounted to the same EFS volume for their software and we already supposed to read and write our data to an S3 bucket, so the EBS volume only needs to be large enough to fit our fairly small tmate-minimal container and maybe a bit of scratch space for temp files

We'll also need a --oem-admin mode for mm_interactive that will allow us to mount this "OEM" EFS volume in read-write mode to make changes or updates to this volume. This mode will need to be used with care, as changes made here will affect all batch jobs. For now we'll have to simply say that users need to ask permission before using this. Eventually we may want a way to restrict this only to certain users. Importantly, --oem-admin will only be available for interactive modes. We do not want to provide any way for batch jobs submitted with mm_jobman to make changes to this "OEM" EFS volume.

Initially I will populate this volume with all the software we use, and once we've tested it, other users can start adding software as needed.

Please let me know what questions and clarifications are needed!

[gaow]

Proposed interface that may work for both mm_jobman.sh and mm_interactive.sh, and to also address #117.

Currently there is --mount-packages in mm_interactive.sh with boolean value. What we may do is:

1. Change --mount-packages behavior in mm_interactive.sh as --mount-packages /mnt/efs/$FLOAT_USER:/home/ubuntu /mnt/efs/shared:/opt by default. The /mnt/efs/$FLOAT_USER:/home/ubuntu is the current behavior for --mount-packages set to true. We additionally support /mnt/efs/shared:/opt so we will eg make ln from /mnt/efs/shared/.pixi to /opt/.pixi, and similarly for micromamba. When we make our start script to additionally consider software under /opt/.pixi we will be able to support #117.
2. Suppose that we have OEM packages configured under /mnt/efs/admin then interactive session users can have a lazy / default OEM package setup via setting --mount-packages /mnt/efs/admin:/home/ubuntu /mnt/efs/shared:/opt. By doing so, because all packages under /mnt/efs/admin and mnt/efs/shared are rw to admin and ro to regular users, when users load packages this way they will be able to use everything available but cannot install anything to them.
3. To use packages from the container VM not mounting anything, we can add a --no-mount-packages switch to override --mount-packages regardless of the setting of --mount-packages
4. We can add --mount-packages similarly to mm_jobman.sh and make default to --mount-packages /mnt/efs/$FLOAT_USER:/home/ubuntu. Then by default, users will run batch jobs with their own software setup rather than from the image. By using --mount-packages /mnt/efs/admin:/home/ubuntu the batch jobs will be using the OEM packages we install. Of course --no-mount-packages will be supported if users run images such as the current xqtl-full docker image where all software are available. This will impelment #116
5. For admin, if they set --mount-packages /mnt/efs/admin:/home/ubuntu /mnt/efs/shared:/opt in an interactive session, since PIXI_HOME is already set to /home/ubuntu/.pixi and they have rw to /mnt/efs/admin they can make changes to this to maintain the OEM packages designed for batch jobs #116
6. For admin if they set --mount-packages /mnt/efs/shared:/home/ubuntu since PIXI_HOME is already set to /home/ubuntu/.pixi and they have rw to /shared they will be able to now make changes to /shared to maintain the shared packages designed for interactive analysis #117 .

[gaow]

@Ashley-Tung the above is summary of discussion between @gaow and @danielnachun on slack.