Add support for a common environment with curated, centrally managed dependencies shared on a multi-user system

[whlavina]

Problem description

I work in Bioinformatics at the federal government, where we have need to supporting a large staff of scientific researchers that use High-Performance Computing (HPC) in a regulatory setting with constraints regarding Application Security. I think Pixi is very close to providing a solution that could replace existing tools and practices for making scientific software available to the userbase.

I would love to see support for a common environment with curated, centrally managed dependencies shared on a multi-user system. The requirements as I see them in our organization:

1. Essential: Ability for a system administrator to define a restricted set of software versions, such that resolution of dependencies by users will only consider the curated set of versions, so that AppSec policies of software review are honored.
2. Desirable: Ability for a system administrator to install a set of software versions in a shared network location, so that disk space is conserved across the large staff (hundreds of users).
3. Nice to have: Ability for Pixi to support modules (applications, perhaps also libraries) pre-installed via other means, such as an OS package manager, such that a dependency expressed by a user may be satisfied by the existing software installation, so that one may benefit from declarative dependencies in projects while allowing administrators to customize how some software is built and installed, outside of the Conda ecosystem.
4. Essential: Ability for a system administrator to upgrade the shared environment and have those updated packages reflected in all environments that use it, so that the maintenance effort of staying current for AppSec compliance can be the responsibility of the system administrators, reducing the burden on the main users.
5. Nice to have: Ability for workgroups other than the system administrators to provide all of the above, so that one may define team-level dependencies or shared dependencies across a pool of related projects.
6. Essential: Ensuring that stale software versions are promptly evicted from any disk caches, so that AppSec filesystem security scans are not triggered. This requirement is automatically addressed for the packages from the common environment, if (2) and (3) are supported, since the packages would be installed on the shared disk storage.
7. Desirable: System logging that includes at least the creation of new environments, so that AppSec may monitor software installations.

Giving some ideas for implementation:

• I believe (1) could be addressed by vendoring via local repositories, for example, by using a privately hosted Conda repository (e.g. Artifactory) with a curated copy of packages and/or a repository proxy with cache. However, that won't solve (4).
• I believe (2) and (6) could be solved by adding support for a shared read-only global cache, perhaps using content-addressable storage. A local Pixi environment could use symlinks and/or hardlinks into a shared cache, where the files are managed by the system administrators and are read-only by the users. If using symlinks, such as if the data crosses filesystem mounts, there is the problem of garbage collection and how to address dangling references: files do need to be deleted to avoid hits by AppSec scans. Perhaps it's sufficient but less than ideal for Pixi to emit a clear warning if an environment is impacted and require the user to invoke an explicit update step on their environment.
• I believe (7) could be handled by use of syslog, or a mechanism to configure plug-ins at the system level.
• "Chained" environments is a concept which has been used in other package managers, where an environment inherits the packages of another, and requests to fulfill a dependency are satisfied by the chained upstream environment.

I encourage looking at the solutions offered by these 2 tools, common in HPC environments:

• Lmod: A New Environment Module System
• Spack, with support for Chaining Spack Installations

For another take on the use case, see:

• Implementing a Common HPC Environment in a Multi-User Spack Instance
• https://github.com/spack/spack/pull/11871

[ruben-arts]

He @whlavina,

Thank you for this informative write-up, we've noticed some more interest from the HPC community which we love.

• (1) We're currently looking for partners to develop our prefix.dev platform further, one of the features we have been discussing is exactly what your are requesting here. If this interests you, contact us on hi@prefix.dev.
• (2) This is an interesting idea, we currently depend on a local cache which we (hard/ref)link into the users environment. As you describe this will be hard to manage over network. (6) Also removing the packages that are still in use by an environment is hard as removing the cache will still keep a copy in the environment so the current requirement is also removing all .pixi folders in the projects.
• (3) This is something I'm not sure we're going to support. Our biggest focus is reproducible environments, which means that depending on system installations is something we cannot do. We only install what we have in the lock-file and avoid using the system as much as we can. Defaulting to sys-admin accepted base dependencies is something we advice to be handled through a companies distribution of a channel. (What I'm talking about in (1)). This also avoids the implementation of chaining environments, as a project cannot control the previous link in a chain.
• (4) We believe in version control systems to take care of history and updates. Using tools like dependabot or renovate on the projects could help in the future to update the projects which intern automatically updates the environments. And in combination with the previously mentioned "distributions" you could force users into specific version of accepted dependencies. To force the distribution onto all environments the use of the coming mirror functionality might help: https://github.com/prefix-dev/pixi/pull/931.
• (5) Again the distributions could fix this request :smile:
• (6) Removing unused cache isn't a problem but removing cache that is in use by environments on the system is a harder problem. We specifically use (hard/ref)links to avoid cache removal breaking any environments.
• Edit: (7) We could look in to that, do you have examples of tools that do this? Then we might understand the requirements better.

These are some really organization specific requests, as mentioned we are looking for partners to design with. If you would be open for a call we would happily discuss a partnership where we could help each other. Email us at hi@prefix.dev. Or join our Discord and send a personal message to me.

[whlavina]

Thank you for the prompt and optimistic reply! I hope you don't mind that I edited my comment to add an item (7) for logging. :-)

Let me discuss within our organization a little more before the next step of reaching out to your team: I opened this issue given clear blockers that I saw for our adoption of Pixi, and your reply will help me promote the idea and gain support.

[ruben-arts]

Great, reacted to (7) in my initial reaction for completeness. Looking forward to your next steps ;). If you need anymore material or answers to questions that come up, let us know!

[chebee7i]

Hi, another related use case. We generally provide centrally managed environments which 99% of users use by default (stored on a shared filesystem).

pixi seems to make strong assumptions about the conda environment living inside of a project, which is not how we prefer to install our environments. Is there anyway that we can use pixi, or are we stuck on mamba? Is there a way to "activate" one of the environments, so that stuff just works without having to do pixi run?

[tdejager]

Hi, another related use case. We generally provide centrally managed environments which 99% of users use by default (stored on a shared filesystem).

pixi seems to make strong assumptions about the conda environment living inside of a project, which is not how we prefer to install our environments. Is there anyway that we can use pixi, or are we stuck on mamba? Is there a way to "activate" one of the environments, so that stuff just works without having to do pixi run?

I'm not sure about the use-case exactly but given a pixi.toml. You can activate a shell using pixi shell or pixi shell -e <env>. When navigating to other directories, even once containing another pixi.toml the original should be respected. This even works for tasks, although in the cases that you are using an activated shell in a directory with a different pixi manifest you will get a warning.

It's simple to add a command that uses this feature with the --manifest-path to add a kind-of activation that you are used to. However, we do want to support a different workflow with pixi, so if you want the more vanilla conda experience, conda or mamba are your best bet!