nsheff
commented
3 years ago The instructions in the guides thus far are oriented around a single user and their collection of software. If bulker is to be provided as something akin to "Environment Modules" for multiple users on a shared HPC system, how should it be configured?
Thanks for the question. Indeed, the docs are a bit sparse for that use case, I guess. The only thing written about it is here: https://bulker.databio.org/en/latest/multi_user_environment/
Not sure if you saw this, but here's a blog post I wrote that addresses this very issue: https://databio.org/posts/modules_to_containers.html
I will write a more detailed how-to guide on how to do this. It's really not that hard, though -- you set it up exactly as you would for a single user, and just have everyone point their BULKERCFG environment variable to a shared bulker configuration file. It works great, that's how we're doing it in my lab. We haven't had any problems.
And is there a way to combine per-user collections of containers (with Singularity - this makes less sense with Docker) and a read-only central cache?
Hmm...in my setup I've just had everyone use the same central area. I haven't thought about having some local tools with some shared central ones. At the moment, you could set this up with symlinks manually. In this case one way you could do this is to have each user have their own config file, but then symlink some of the manifest folders to a shared area. I can't think of a way to automate it off the top of my head. It should be fairly simple to add this to bulker, though -- there could be a priority list of SIMAGES directories, you'd put the shared one at the back of the list and make it read-only. bulker would install into the highest priority that it has write permissions.
pvanheus
The instructions in the guides thus far are oriented around a single user and their collection of software. If bulker is to be provided as something akin to "Environment Modules" for multiple users on a shared HPC system, how should it be configured?
And is there a way to combine per-user collections of containers (with Singularity - this makes less sense with Docker) and a read-only central cache?