search

TACC / Lmod

Lmod: An Environment Module System based on Lua, Reads TCL Modules, Supports a Software Hierarchy
http://lmod.readthedocs.org
Other
503 stars 128 forks source link

RFC/Feature Request: Linux Namespace integration #387

Open azet opened 6 years ago

azet commented 6 years ago

Hi,

I've recently worked a lot with linux namespaces (see: https://lwn.net/Articles/531114/ and it's follow-up articles on the different flavors). They're massively useful in Container and Virtualization environments, but they're also in HPC, in fact because of that they're in use in some national labs and were originally developed and implemented back in the late 2000s at Google for seperation, isolation and security of applications run by different users/groups on their massive global-scale clusters. I think it would make sense to integrate at least User-, PID-, UTS- and Mount namespaces into Lmod. Network namespaces can also be useful but depend highly on the set-up of the system in question, interconnect in use and how storage systems and locations are used in practice.

Recently work has started on "time namespaces" (which are very interesting for checkpoint/restart in HPC and some development scenarios among other things), though they're far from being ready to get merged into upstream: https://criu.org/Time_namespace (https://lwn.net/Articles/766089/)

Another thing to think about is if we'd be interested in mangling with cgroups in Lmod too, since most major Linux distributions have switched to systemd, they're in place and easily accessible with decent tooling anyhow, so a wrapper may suffice there if, and where needed. But this is - again - something very dependent on the individual set-up on the HPC installation in question.

Implementation-wise: luajit/luaffi will make implementation of these features rather straight forward. I'm just interested if there's users that are interested in these features and would possibly even fund a project to develop those? I might work on them in my spare time, but since I have a lot of work and projects on the side at the moment, that may take far more time.

See also: http://man7.org/linux/man-pages/man7/namespaces.7.html http://man7.org/linux/man-pages/man7/pid_namespaces.7.html http://man7.org/linux/man-pages/man7/user_namespaces.7.html http://man7.org/linux/man-pages/man7/mount_namespaces.7.html

May label as: enhancement.

/CC @rtmclay @boegel @fgeorgatos @georg-rath @ebirn @pforai

wpoely86 commented 6 years ago

I'm not following, what should be integrated?

The only thing Lmod does is read some files and output something which can be interpreted by a shell. Start a new shell and you have a completely new and separate environment.