search

TACC / Lmod

Lmod: An Environment Module System based on Lua, Reads TCL Modules, Supports a Software Hierarchy
http://lmod.readthedocs.org
Other
499 stars 128 forks source link

Generate system cache without taking `$MODULEPATH` changes into account #708

Closed bedroge closed 5 months ago

bedroge commented 5 months ago

On our cluster we offer some top-level modules that give access to certain software stacks, e.g. to our own stack and the EESSI software stack. These top-level modules basically just prepend some path to $MODULEPATH and to $LMOD_RC, where the latter is used to point to an existing cache file of that stack.

Initially we didn't provide a cache for the (handful) top-level modules, but we noticed that Lmod would seem to generate one anyway (per user?), which made it walk through the entire stack; I suspect because it was finding the $MODULEPATH change in those module files. This caused issues where logging in could take a long time.

We then tried to work around it by providing a cache for the top-level modules, but we noticed that again the update_lmod_system_cache_files would do the same thing, i.e. find the $MODULEPATH change in the modulefile and hence scan the entire stacks for modules. As EESSI (and also our own stack) have procedures in place to automatically generate those caches every time new software is added, we would like to prevent this from happening. We now worked around it by temporarily removing the prepend_path("MODULEPATH", ...) line from the modulefile when generating that top-level cache, but is there a better way to do this? Or would it be possible to add some option to the script that disables this functionality?

rtmclay commented 5 months ago

Please read https://lmod.readthedocs.io/en/latest/130_spider_cache.html

ocaisa commented 5 months ago

I did look into that and the issue I saw is that we would need to talk to the hosting site to have the EESSI caches included in their local setup.

However, I also thought I saw recently that LMOD_RC can now be path-like. Does this mean that we can then dynamically build up the cache information without having to define everything in advance at the site?

This would allow us to have a complicated set of possibilities within EESSI (since we may want to deliver multiple different naming schemes possibilities) without having to have discussions with the hosting site.

rtmclay commented 5 months ago

Have you read: https://lmod.readthedocs.io/en/latest/350_community.html.

This might be a solution for what you are looking for.

bedroge commented 5 months ago

Have you read: https://lmod.readthedocs.io/en/latest/350_community.html.

This might be a solution for what you are looking for.

Thanks, that looks useful for this use case, I'll give that a try.

bedroge commented 5 months ago

I've tried the approach from https://lmod.readthedocs.io/en/latest/350_community.html, and that seems to work as expected. Thanks!