speed up module loading and purging

[dtenenba]

Not sure if this is the right place for this, but here goes.

I remember @fizwit telling me that there is no reason that module loading should not be instantaneous - all it's doing is setting environment variables, right?

But as we know it can be pretty slow. ml purge can be slow as well.

Aside from just being annoying, there a few use cases where this breaks things, for example, setting VS code to develop in R - if you try and load the modules, things break because it takes too long. The hacky workaround is to point VS code's R extension to a wrapper script that sets certain environment variables to the same value that ml fhR sets them to and then runs R.

It would be nice to avoid hacks like this. I would be happy to help investigate why this is happening. Maybe we could start by asking the easybuild / lmod community?

[fizwit]

Load time with --ingore-cache is the same as with CACHE enabled. It seems like something is wrong with the cache.
But the lua file for fhR has a two variables that are hundreds of lines long. By removing whatis([==[Extensions: removed 10K from the lua file and shortened load time from 9 seconds to 3 seconds. It seems like including the extension list in the Lua is consuming a lot of resources.

Kenneth has opened an issue with Robert about this in the past. It would seem like the solution is to configure EasyBuild too not create these variables in the Module files.

Before

time (export LMOD_IGNORE_CACHE=no; ml fhR/4.3.3-foss-2023b )

real    0m10.043s

After

time (ml fhR/4.3.3-foss-2023b )

real    0m2.598s