HenrikBengtsson commented 2 years ago

For example, in https://wynton.ucsf.edu/hpc/software/software-repositories.html#module_built-in_cuda, e.g. cuda and matlab is contaminated by ditto from the Sali lab repo.

HenrikBengtsson commented 2 years ago

$ cat lmod_data/etc/modulefiles.json | jq .
...
  {
    "package": "matlab",
    "versions": [
      {
        "wV": "000000009.000000005.000000000.000944444.*zfinal",
        "path": "/salilab/diva1/home/modules/matlab/9.5.0.944444",
        "versionName": "9.5.0.944444",
        "markedDefault": false,
        "parent": [
          [
            "Sali"
          ]
        ],
        "canonicalVersionString": "000000009.000000005.000000000.000944444.*zfinal",
        "full": "matlab/9.5.0.944444",
        "description": "Language for technical computing "
      },
      {
        "wV": "000002018.*b.*zfinal",
        "path": "/etc/modulefiles/matlab/2018b",
        "versionName": "2018b",
        "canonicalVersionString": "000002018.*b.*zfinal",
        "full": "matlab/2018b",
        "markedDefault": false
      },
...

vs

$ cat lmod_data/salilab/diva1/home/modules-py27.json | jq .
...
  {
    "description": "Language for technical computing ",
    "defaultVersionName": "9.5.0.944444",
    "package": "matlab",
    "url": "https://www.mathworks.com/products/matlab.html ",
    "versions": [
      {
        "wV": "000000009.000000005.000000000.000944444.*zfinal",
        "path": "/salilab/diva1/home/modules-py27/matlab/9.5.0.944444",
        "versionName": "9.5.0.944444",
        "markedDefault": false,
        "canonicalVersionString": "000000009.000000005.000000000.000944444.*zfinal",
        "full": "matlab/9.5.0.944444",
        "description": "Language for technical computing "
      }
    ]
  },
  {
...

HenrikBengtsson commented 2 years ago

The problem is that the built-in module folder holds CBI.lua and Sali.lua as well:

$ ls -l /etc/modulefiles/                                                                                                                                
total 20
drwxr-xr-x.   9 root root   137 May 11  2020 .
drwxr-xr-x. 133 root root 12288 May 13 14:11 ..
drwxr-xr-x.   2 root root    16 Feb 23  2021 amber
lrwxrwxrwx.   1 root root    49 Apr 19  2019 CBI.lua -> /wynton/home/cbi/shared/modulefiles/repos/CBI.lua
drwxr-xr-x.   2 root root    74 Oct 27  2021 cuda
drwxr-xr-x.   2 root root    32 Aug 12  2021 julia
drwxr-xr-x.   2 root root   110 Apr 27 14:05 matlab
drwxr-xr-x.   2 root root    71 Apr 27 14:06 matlab-runtime
drwxr-xr-x.   2 root root    51 May 27  2021 mpi
drwxr-xr-x.   2 root root    72 Feb  8 10:14 openjdk
-rw-r--r--.   1 root root    95 Feb 23  2021 Sali.lua

and those *.lua files are also parsed when doing:

/usr/share/lmod/lmod/libexec/spider --no_recursion -o jsonSoftwarePage /etc/modulefiles/

which (I think) is the same as:

/usr/share/lmod/lmod/libexec/spider --no_recursion -o jsonSoftwarePage /etc/modulefiles/*

Not sure what --no_recursion is meant for. I thought it was this.

Task

The solution is to figure out how to do the above without those /etc/modulefiles/*.lua files.

ucsf-wynton / wynton-website-hpc

BUG: 'built-in' modules includes same-name modules from the other repositories #54

Task