MODULEPATH modifications made in the load/unload hooks are not found by `module spider`

[AcerP-py]

Describe the bug I have a lmod hook that adds/removes items from the MODULEPATH when certain modules are loaded/unloaded. I know that the normal method for modifying MODULEPATH is to add a prepend_path('MODULEPATH', ...) to the module that you want the path added for. However, in our HPC cluster we don't control the compiler or mpi module files that we would want those paths added for. They are managed by the vendor. While our approch with the hook works it breaks the spider command. It appears that for module spider the load hooks are not called when evaluating a module file. I am guessing this was intentional to help with the speed? I was wondering if their might be a way to enable this. If you don't want it on all the time maybe make it optional via an environment variable.

To Reproduce Module Layout:

$ tree
.
├── addition
│   └── B
│       └── 1.0.0.lua
├── base
│   └── A
│       └── 1.0.0.lua
└── etc
    └── SitePackage.lua

SitePackage.lua

---
-- @module SitePackage

local Hook = require("Hook")

function l_load_hook(t)
    prepend_path('MODULEPATH', '/tmp/modules/addition')
end

function l_unload_hook(t)
    remove_path('MODULEPATH', '/tmp/modules/addition')
end

Hook.register('load', l_load_hook, 'append')
Hook.register('unload', l_unload_hook, 'append')

Add the base path and then run:

$ ml av

-------------------------------------------------------------------------------------------------------------------------------------------------- /tmp/modules/base --------------------------------------------------------------------------------------------------------------------------------------------------
   A/1.0.0

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

xjv@LAP0142212:/tmp/modules$ ml load A
xjv@LAP0142212:/tmp/modules$ ml av

------------------------------------------------------------------------------------------------------------------------------------------------ /tmp/modules/addition ------------------------------------------------------------------------------------------------------------------------------------------------
   B/1.0.0

-------------------------------------------------------------------------------------------------------------------------------------------------- /tmp/modules/base --------------------------------------------------------------------------------------------------------------------------------------------------
   A/1.0.0 (L)

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

$ ml unload A
$ ml av

-------------------------------------------------------------------------------------------------------------------------------------------------- /tmp/modules/base --------------------------------------------------------------------------------------------------------------------------------------------------
   A/1.0.0

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

$ ml spider B
Lmod has detected the following error:  Unable to find: "B".

$ 

Expected behavior I would expect module spider to be able to find anything you can by loading modules, but as explained above it seems it does not call the load hook when evaluating a file.

Debug Info See attached: miniconfig.txt spider_debug.txt

[rtmclay]

When Lmod is building the spider cache, it loops over all modules found in the $MODULEPATH. Lmod does evaluate changes to $MODULEPATH. But the load() function is ignored, so there is no way to run the load hooks.

[rtmclay]

I have not read in detail your setup. But a quick read of your situation, you might consider seeing if you can implement your own compiler and mpi modules and use the inherit() function to find the vendor compiler/mpi modulefiles.

[rtmclay]

My replies were a bit snarky. The problem is that when spider is evaluating module files it is not a true load. Also many sites, including ours, use the load hook to track module usage. So clearly having the spider evaluating all modules using the load_hook wouldn't work. Fortunately, Lmod already provides a solution, you can use the "load_spider" hook. You can reuse the same simple l_load_hook function. Just register it to the "load_spider" hook.

[AcerP-py]

First let me say thank you for the quick responses! I don't think your responses were snarky. They were short but I understand that.

In regards to the "load_spider" hook. I have read the documentation/code examples https://lmod.readthedocs.io/en/latest/170_hooks.html#hook-functions https://github.com/TACC/Lmod/blob/8d68c7d8e68f08e53d94da532098ce6930caa6d1/rt/prophook/SitePackage.lua However, I can't get my hook to be called:

---
-- @module SitePackage

local Hook = require("Hook")

function l_spider_hook(t)
    LmodMessage("Hello, World!")
end

Hook.register('load_spider', l_spider_hook)

Is there a variable or config somewhere that has to be set. I know for the "avail" hook you have to set 'LMOD_AVAIL_STYLE' and wondered if it might be the same for "load_spider". Because right now my hook prints nothing when module spider is called.

[rtmclay]

It is being called. It is just that LmodMessage() is set to be quiet during spider operations. You might be able to use io.stderr:write("Hello World!\n"). but that might be set to quiet as well. If you have to you can open a file and write to it.

[rtmclay]

O.K. to close this issue?

[AcerP-py]

For various reasons I am not sure of the viability of the spider hook or the inherit function for my particular situation but further discussion would probably be best suited for the monthly zoom meeting so I am good with closing this.