climbfuji
commented
2 years ago Mike,
we do this before we build the stack, you may want to do the same when building the app:
module purge module unuse /glade/u/apps/ch/modulefiles/default/compilers export MODULEPATH_ROOT=/glade/p/ral/jntp/GMTB/tools/compiler_mpi_modules module use /glade/p/ral/jntp/GMTB/tools/compiler_mpi_modules/compilers
Basically what this is doing is replacing CISL’s compiler modules with identical ones, except that the module tree doesn’t have the hierarchical modules that CISL provides (it only has compilers, mpi libs and the ncarcompilers modules). You do NOT want to load the default netCDF module from CISL if you use hpc-stack, which has its own netCDF (and the entire stack being built against it for consistency).
Dom
On Jan 17, 2022, at 5:42 PM, Michael Kavulich @.***> wrote:
Describe the bug
The short summary: using load(modulename) in Lua modulefiles will not necessarily get you the module you expect. This is currently causing failures of the Short Range Weather Application on Cheyenne, and could theoretically result in unexpected behavior on any platform, up to and including build and runtime failures
Many NCEPLIBS have dependencies on each other, this is obvious. Currently, however, there doesn't seem to be any uniformity in how these dependencies are handled; and some of the current methods are less than ideal. Here are the various methods used:
ATLAS simply uses prereq, which causes the module load command to fail if the prerequisites are not already loaded. This seems like a natural way to handle required modules: https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/atlas/atlas.lua#L15-L18 https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/atlas/atlas.lua#L15-L18 wgrib2 (in addition to others) uses a load command prior to the prereq. In addition to this being redundant (both load and prereq will fail if the module does not exist, and if it does exist, prereq will always succeed after a load of the same module), this has a dangerous side-effect of being able to overwrite the existing environment unexpectedly: if, for example, netcdf/4.7.4 is loaded from earlier in the hpc-stack process, but the system already has a built-in netcdf module, the load(netcdf) line will instead load that system default netcdf module, which may be a different version and incompatible with other hpc-stack modules! https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/wgrib2/wgrib2.lua#L15-L18 https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/wgrib2/wgrib2.lua#L15-L18 MADIS (in addition to others) uses always_load in addition to prereq. This appears to be an even worse option to use, since in addition to the above-mentioned problems, module unload will attempt to load netCDF rather than unloading it; there doesn't seem to be any reason to do this. https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/madis/madis.lua#L15-L18 https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/madis/madis.lua#L15-L18 To Reproduce On Cheyenne, run these commands and observe the environment changes:
module purge module load ncarenv/1.3 module load gnu/9.1.0 module load mpt/2.22 module load ncarcompilers/0.5.0 module use /glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/modulefiles/stack module load hpc/1.1.0 module load hpc-gnu/9.1.0 module load hpc-mpt/2.22 module load netcdf/4.7.4 module list
Currently Loaded Modules: 1) ncarenv/1.3 3) hpc/1.1.0 5) hpc-gnu/9.1.0 7) hpc-mpt/2.22 9) netcdf/4.7.4 2) ncarcompilers/0.5.0 4) gnu/9.1.0 6) mpt/2.22 8) hdf5/1.10.8
module show netcdf
/glade/p/ral/jntp/GMTB/tools/hpc-stack-v1.1.0/modulefiles/mpi/gnu/9.1.0/mpt/2.22/netcdf/4.7.4.lua:
module load wgrib2/2.0.8
The following have been reloaded with a version change: 1) netcdf/4.7.4 => netcdf/4.8.1
module list
Currently Loaded Modules: 1) ncarenv/1.3 3) hpc/1.1.0 5) hpc-gnu/9.1.0 7) hpc-mpt/2.22 9) wgrib2/2.0.8 2) ncarcompilers/0.5.0 4) gnu/9.1.0 6) mpt/2.22 8) netcdf/4.8.1
module show netcdf
/glade/u/apps/ch/modulefiles/default/gnu/9.1.0/netcdf/4.8.1.lua:
What is happening here is that wgrib2's load(netcdf) command has no consideration for if the user has already loaded a version of netcdf. So because a netcdf system default module exists on Cheyenne, the module rules for wgrib2 cause the existing netcdf/4.7.4 from hpc-stack to be unloaded, and replaced by the system default netcdf/4.8.1. This causes mismatches in libraries that ultimately results in runtime failures for the Short Range Weather Application; see that issue for more details: ufs-community/ufs-srweather-app#201 https://github.com/ufs-community/ufs-srweather-app/issues/201 Expected behavior One would expect that if you already have a module loaded, that choice will not be overwritten. If a subsequent module is not compatable with your currently loaded modules, it should fail with a descriptive message, not overwrite the current environment. This is especially problematic for systems that already have a built-in module with the same name of one used in hpc-stack: in the case of Cheyenne, this is netCDF, but the problem could manifest on any platform with different versions of the same libraries used in hpc-stack.
System: Cheyenne, but this issue may occur on any platform that has built-in modules with the same names as those found in hpc-stack.
Additional context As far as I can see there are two potential fixes to this.
Simply remove the load command from each .lua module that contains a prereq command, allowing the module to fail if the prerequisite modules are not loaded. This would give the user a descriptive error message if the prerequisites are not met: I would find this an acceptable solution, but I understand that it may be disruptive to some existing workflows.
If the load is required for some workflows, the "overwrite" behavior can be eliminated with a conditional statement:
if ( not isloaded("netcdf") ) then load("netcdf") end This will only load netcdf if there is not another netcdf module already loaded.
Apologies for the long-winded issue, but I wanted to be sure that the entire problem was covered. I am willing to open a PR to fix this issue, but as I mention above, I'm not sure what the best solution is, so I hope for some wider feedback. Let me know if anything else is needed.
— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/hpc-stack/issues/376, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5C2RJJNHAIIBYW534WUWDUWSZQPANCNFSM5MF323VA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.
kgerheiser
christopherwharrop-noaa
mkavulich
Describe the bug
The short summary: using
load(modulename)in Lua modulefiles will not necessarily get you the module you expect. This is currently causing failures of the Short Range Weather Application on Cheyenne, and could theoretically result in unexpected behavior on any platform, up to and including build and runtime failuresMany NCEPLIBS have dependencies on each other, this is obvious. Currently, however, there doesn't seem to be any uniformity in how these dependencies are handled; and some of the current methods are less than ideal. Here are the various methods used:
prereq, which causes themodule loadcommand to fail if the prerequisites are not already loaded. This seems like a natural way to handle required modules:https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/atlas/atlas.lua#L15-L18
loadcommand prior to theprereq. In addition to this being redundant (bothloadandprereqwill fail if the module does not exist, and if it does exist,prereqwill always succeed after aloadof the same module), this has a dangerous side-effect of being able to overwrite the existing environment unexpectedly: if, for example, netcdf/4.7.4 is loaded from earlier in the hpc-stack process, but the system already has a built-in netcdf module, theload(netcdf)line will instead load that system default netcdf module, which may be a different version and incompatible with other hpc-stack modules!https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/wgrib2/wgrib2.lua#L15-L18
always_loadin addition toprereq. This appears to be an even worse option to use, since in addition to the above-mentioned problems,module unloadwill attempt to load netCDF rather than unloading it; there doesn't seem to be any reason to do this.https://github.com/NOAA-EMC/hpc-stack/blob/44c8e077a21099fcc1883cd297783cdfa6c08b23/modulefiles/mpi/compilerName/compilerVersion/mpiName/mpiVersion/madis/madis.lua#L15-L18
To Reproduce On Cheyenne, run these commands and observe the environment changes:
What is happening here is that wgrib2's
load(netcdf)command has no consideration for if the user has already loaded a version of netcdf. So because a netcdf system default module exists on Cheyenne, the module rules for wgrib2 cause the existing netcdf/4.7.4 from hpc-stack to be unloaded, and replaced by the system default netcdf/4.8.1. This causes mismatches in libraries that ultimately results in runtime failures for the Short Range Weather Application; see that issue for more details: https://github.com/ufs-community/ufs-srweather-app/issues/201Expected behavior One would expect that if you already have a module loaded, that choice will not be overwritten. If a subsequent module is not compatable with your currently loaded modules, it should fail with a descriptive message, not overwrite the current environment. This is especially problematic for systems that already have a built-in module with the same name of one used in hpc-stack: in the case of Cheyenne, this is netCDF, but the problem could manifest on any platform with different versions of the same libraries used in hpc-stack.
System: Cheyenne, but this issue may occur on any platform that has built-in modules with the same names as those found in hpc-stack.
Additional context As far as I can see there are two potential fixes to this.
Simply remove the
loadcommand from each .lua module that contains aprereqcommand, allowing the module to fail if the prerequisite modules are not loaded. This would give the user a descriptive error message if the prerequisites are not met: I would find this an acceptable solution, but I understand that it may be disruptive to some existing workflows.If the
loadis required for some workflows, the "overwrite" behavior can be eliminated with a conditional statement:This will only load netcdf if there is not another netcdf module already loaded.
Apologies for the long-winded issue, but I wanted to be sure that the entire problem was covered. I am willing to open a PR to fix this issue, but as I mention above, I'm not sure what the best solution is, so I hope for some wider feedback. Let me know if anything else is needed.