spider json fails to mark modules as hidden

[simonLeary42]

Consider the following module tree:

├── newdir
│   └── foobar
│       ├── .modulerc.lua
│       └── 1.2.3.lua
├── root
│   └── newdir
│       └── 1.0.lua

with the following contents:

$ cat root/newdir/1.0.lua
prepend_path("MODULEPATH", "/home/simon/lmod-bug-test/newdir")

$cat newdir/foobar/1.2.3.lua
prepend_path("PATH", "/opt/foobar")

$ cat newdir/foobar/.modulerc.lua
hide_version("foobar/1.2.3")

module avail shows foobar/1.2.3 as hidden:

$ ml av

----------- /home/simon/lmod-bug-test/root -----------
   newdir/1.0

$ ml newdir/1.0
$ ml av

----------- /home/simon/lmod-bug-test/root -----------
   newdir/1.0 (L)

$ ml --show_hidden av

----------- /home/simon/lmod-bug-test/newdir -----------
   foobar/1.2.3 (H)

----------- /home/simon/lmod-bug-test/root -----------
   newdir/1.0 (L)

spiderT also shows foobar/1.2.3 as hidden:

`$ /opt/lmod/lmod/lmod/libexec/spider -o spiderT /home/simon/lmod-bug-test/root` ```lua timestampFn = { false, } mrcT = { alias2modT = {}, hiddenT = {}, version2modT = {}, } mrcMpathT = { ["/home/simon/lmod-bug-test/newdir"] = { hiddenT = { ["foobar/1.2.3"] = true, <===================== HERE }, }, } spiderT = { ["/home/simon/lmod-bug-test/newdir"] = { foobar = { defaultA = { { barefn = ".modulerc.lua", defaultIdx = 2, fn = "/home/simon/lmod-bug-test/newdir/foobar/.modulerc.lua", fullName = "foobar/.modulerc", luaExt = 10, mpath = "/home/simon/lmod-bug-test/newdir", value = false, }, }, defaultT = { barefn = ".modulerc.lua", defaultIdx = 2, fn = "/home/simon/lmod-bug-test/newdir/foobar/.modulerc.lua", fullName = "foobar/.modulerc", luaExt = 10, mpath = "/home/simon/lmod-bug-test/newdir", value = false, }, dirT = {}, fileT = { ["foobar/1.2.3"] = { Version = "1.2.3", canonical = "1.2.3", fn = "/home/simon/lmod-bug-test/newdir/foobar/1.2.3.lua", luaExt = 6, mpath = "/home/simon/lmod-bug-test/newdir", pV = "000000001.000000002.000000003.*zfinal", pathA = { ["/opt/foobar"] = 1, }, wV = "000000001.000000002.000000003.*zfinal", }, }, }, }, ["/home/simon/lmod-bug-test/root"] = { newdir = { defaultA = {}, defaultT = {}, dirT = {}, fileT = { ["newdir/1.0"] = { Version = "1.0", canonical = "1.0", changeMPATH = true, fn = "/home/simon/lmod-bug-test/root/newdir/1.0.lua", luaExt = 4, mpath = "/home/simon/lmod-bug-test/root", pV = "000000001.*zfinal", wV = "000000001.*zfinal", }, }, }, }, version = 5, } mpathMapT = { ["/home/simon/lmod-bug-test/newdir"] = { ["newdir/1.0"] = "/home/simon/lmod-bug-test/root", }, } ```

but strangely, json spider cache does not mark foobar/1.2.3 as hidden:

$ /opt/lmod/lmod/lmod/libexec/spider -o spider-json /home/simon/lmod-bug-test/root | python -m json.tool

{
    "newdir": {
        "/home/simon/lmod-bug-test/root/newdir/1.0.lua": {
            "hidden": false,
            "fullName": "newdir/1.0",
            "pV": "000000001.*zfinal",
            "wV": "000000001.*zfinal",
            "mpath": "/home/simon/lmod-bug-test/root",
            "Version": "1.0"
        }
    },
    "foobar": {
        "/home/simon/lmod-bug-test/newdir/foobar/1.2.3.lua": {
            "hidden": false, <=========================== HERE
            "fullName": "foobar/1.2.3",
            "parentAA": [
                [
                    "newdir/1.0"
                ]
            ],
            "pathA": {
                "/opt/foobar": 1
            },
            "pV": "000000001.000000002.000000003.*zfinal",
            "wV": "000000001.000000002.000000003.*zfinal",
            "mpath": "/home/simon/lmod-bug-test/newdir",
            "Version": "1.2.3"
        }
    }
}

I thought spiderT and spider-json stored all the same information, just in a different layout. Is that not true?

$ module --version
Modules based on Lua: Version 8.7.34  2024-01-05 19:34 -07:00
    by Robert McLay mclay@tacc.utexas.edu

[rtmclay]

This was a subtle bug that took sometime to track down. Please try the PR684 branch to see if it works for you. Thanks very much for a test case that shows the issue!

[simonLeary42]

My test case is still showing the issue after using spider built from PR684

[simonLeary42]

I'm using Lua 5.1 and TCL 8.6 on Ubuntu 20.04

[rtmclay]

please run

$ $LMOD_DIR/spider -o dbT /home/simon/lmod-bug-test/root > dbT.lua

and paste the file dbT.lua here

[simonLeary42]

timestampFn = {
  false,
}
dbT = {
  foobar = {
    ["/home/simon/lmod-bug-test/newdir/foobar/1.2.3.lua"] = {
      Version = "1.2.3",
      fullName = "foobar/1.2.3",
      hidden = false,
      mpath = "/home/simon/lmod-bug-test/newdir",
      pV = "000000001.000000002.000000003.*zfinal",
      parentAA = {
        {
          "newdir/1.0",
        },
      },
      pathA = {
        ["/opt/foobar"] = 1,
      },
      wV = "000000001.000000002.000000003.*zfinal",
    },
  },
  newdir = {
    ["/home/simon/lmod-bug-test/root/newdir/1.0.lua"] = {
      Version = "1.0",
      fullName = "newdir/1.0",
      hidden = false,
      mpath = "/home/simon/lmod-bug-test/root",
      pV = "000000001.*zfinal",
      wV = "000000001.*zfinal",
    },
  },
}
provideByT = {}

[rtmclay]

Thanks! I'll look to see what happened

[rtmclay]

I found the issue and fixed it for me. This issue had to do with the way that the dbT table was computed. First the spiderT table is computed and this is converted into dbT. Since building spiderT involves walking the module tree, the current known directories in $MODULEPATH are known. However building dbT doesn't have to walk the module tree structure so $MODULEPATH wasn't correct. This has now been fixed in the PR684 branch. Please update and test this branch (PR684) to see if it works for you.

Thanks for submitting this bug as it was a subtle one!

[rtmclay]

O.K. to close this issue?

[rtmclay]

The PR684 branch has been merged into the main branch and has been released as Lmod 8.7.36