MODULEPATH unset with `module stash`

BBR-U22 commented 9 months ago

Describe the bug

Modules Release 5.3.0 (2023-05-14) In UGE jobs MODULEPATH variable is cleared with module stash command and set back on module stashpop (__MODULES_LMINIT is not set in jobs). It is fine when ssh though.

To Reproduce

$ qlogin
$ module load ANYMODULE
$ module stash
$ module avail
ERROR: No module path defined
$ module stashpop
$ module avail 
< list of modules here >

Location and content of any modulerc or modulefile involved:

$  cat /usr/local/Modules/etc/initrc
module use --append {/usr/local/Modules/modulefiles}

$ cat /usr/local/Modules/etc/modulespath
/share/apps/environmentmodules/centos7/devtools
/share/apps/environmentmodules/centos7/general
/share/apps/environmentmodules/centos7/libraries
/usr/local/Modules/modulefiles

$ cat /usr/local/Modules/modulefiles/modules
#%Module1.0###################
## modules modulefile
proc ModulesHelp { } {
        global version prefix
        puts stderr "\tmodules - loads the modules software & application environment"
        puts stderr "\n\tThis adds $prefix/* to several of the"
        puts stderr "\tenvironment variables."
        puts stderr "\n\tVersion $version\n"
}
module-whatis   "loads the modules environment"
# for Tcl script use only
set     version         5.3.0
set     prefix          {/usr/local/Modules}

setenv          MODULESHOME     $prefix
prepend-path    PATH            {/usr/local/Modules/bin}
prepend-path    MANPATH         {/usr/local/Modules/share/man}

Expected behavior

List of modules after module statsh

Error and debugging information

$ module load pigz

$ module stash --debug
DEBUG setState: cmdline set to '/usr/local/Modules/libexec/modulecmd.tcl bash stash --debug'
DEBUG setConf: tcl_ext_lib set to '/usr/local/Modules/lib/libtclenvmodules.so'
DEBUG Load Tcl extension library (/usr/local/Modules/lib/libtclenvmodules.so)
DEBUG setState: tcl_ext_lib_loaded set to '1'
DEBUG setConf: siteconfig set to '/usr/local/Modules/etc/siteconfig.tcl'
DEBUG sourceSiteConfig: Source site configuration (/usr/local/Modules/etc/siteconfig.tcl)
DEBUG setState: siteconfig_loaded set to '1'
DEBUG setConf: locked_configs set to ''
DEBUG setState: supported_shells set to 'sh bash ksh zsh csh tcsh fish cmd tcl perl python ruby lisp cmake r'
DEBUG setState: shell set to 'bash'
DEBUG setState: subcmd set to 'stash'
DEBUG setState: subcmd_args set to ''
DEBUG setState: init_error_report set to '1'
DEBUG setConf: verbosity set to 'debug'
DEBUG setState: is_stderr_tty set to '1'
DEBUG setConf: term_background set to 'dark'
DEBUG setConf: colors set to 'hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44:kL=30;48;5;109'
DEBUG setConf: color set to '1'
DEBUG setConf: pager set to '/usr/bin/less -eFKRX'
DEBUG setState: paginate set to '1'
DEBUG setState: report_format set to 'regular'
DEBUG setState: reportfd set to 'file7'
DEBUG setState: timer set to '0'
DEBUG lappendState: modulefile appended with '{}'
DEBUG parseModuleCommandName: (command=stash, cmdvalid=1, cmdempty=0)
DEBUG setConf: avail_indepth set to '1'
DEBUG setConf: search_match set to 'starts_with'
DEBUG parseModuleCommandArgs: (show_oneperline=0, show_mtime=0, show_filter=, search_filter=, search_match=starts_with, dump_state=0, addpath_pos=prepend, not_req=0, tag_list=, otherargs=)
DEBUG setConf: advanced_version_spec set to '1'
DEBUG setConf: variant_shortcut set to ''
DEBUG lappendState: always_read_full_file appended with '1'
DEBUG lappendState: commandname appended with 'stash'
DEBUG setState: rc_running set to '1'
DEBUG setConf: ignore_user_rc set to '0'
DEBUG unsetState: rc_running unset
DEBUG setConf: reset_target_state set to '__init__'
DEBUG setState: is_win set to '0'
DEBUG setState: path_separator set to ':'
DEBUG setState: lm_info_cached set to '1'
DEBUG setState: sub1_separator set to '&'
DEBUG setState: sub2_separator set to '|'
DEBUG setModuleVersSpec: Set module 'pigz' (escglob 'pigz'),  module name '.' (re '\.'), module root 'pigz', version cmp 'eq', version(s) '', variant(s) '' and module name version spec 'pigz' for argument 'pigz' (raw 'pigz'), extra specifier(s) '' and matching available module(s) ''
DEBUG setConf: icase set to 'search'
DEBUG setConf: extended_default set to '1'
DEBUG setConf: unload_match_order set to 'returnlast'
DEBUG doesModuleConflict: 'pigz/2.6' conflicts with '' (declared as '')
DEBUG setModuleDependency: set requirements of 'pigz/2.6' to ''
DEBUG setModuleDependency: set NPO requirements of 'pigz/2.6' to ''
DEBUG cacheCurrentModules: 1 loaded
DEBUG setState: cwd set to '/data/home/user'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG setConf: collection_pin_tag set to '0'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getMovementBetweenList: from(pigz/2.6) to() with extfrom() extto()
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/devtools' resolved to '/share/apps/environmentmodules/centos7/devtools'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/general' resolved to '/share/apps/environmentmodules/centos7/general'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/libraries' resolved to '/share/apps/environmentmodules/centos7/libraries'
DEBUG resolvStringWithEnv: '/usr/local/Modules/modulefiles' resolved to '/usr/local/Modules/modulefiles'
DEBUG getPathToModule: finding 'pigz' in '/share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles' (report_issue=0, look_loaded=no, excdir='')
DEBUG setConf: implicit_default set to '1'
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/devtools (fetch_mtime=0, search=rc_defs_included resolve, filter=)
DEBUG findModules: finding 'pigz' in /share/apps/environmentmodules/centos7/devtools (depthlvl=0, fetch_mtime=0)
DEBUG setConf: ignore_cache set to '0'
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/devtools/.modulecache' cannot be found or read
DEBUG setConf: ignored_dirs set to 'CVS RCS SCCS .svn .git .SYNC .sos'
DEBUG findModules: found 
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/devtools:pigz:0:0'
DEBUG setState: inhibit_errreport set to '0'
Get modules: {} matching 'pigz' in '/share/apps/environmentmodules/centos7/devtools'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/devtools:pigz:0:rc_defs_included resolve:'
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/general (fetch_mtime=0, search=rc_defs_included resolve, filter=)
DEBUG findModules: finding 'pigz' in /share/apps/environmentmodules/centos7/general (depthlvl=0, fetch_mtime=0)
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/general/.modulecache' cannot be found or read
DEBUG setConf: mcookie_check set to 'always'
DEBUG findModules: found pigz/2.4 pigz/.version pigz/2.6
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/general:pigz:0:0'
DEBUG lappendState: modulefile appended with '/share/apps/environmentmodules/centos7/general/pigz/.version'
DEBUG lappendState: modulename appended with 'pigz/.version'
DEBUG lappendState: specifiedname appended with 'pigz/.version'
DEBUG [#1:pigz/.version] lappendState: debug_msg_prefix appended with '{[#1:pigz/.version] }'
Evaluate modulerc: '/share/apps/environmentmodules/centos7/general/pigz/.version'
DEBUG [#1:pigz/.version] execute-modulerc: creating interp __modrc_1
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_version
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_interactive
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var ModuleToolVersion
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var ModuleTool
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var auto_path
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_pkgPath
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_patchLevel
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_platform
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_library
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc proc list auto_load clock unknown tclLog auto_qualify auto_execok auto_import auto_load_index
DEBUG [#1:pigz/.version] resetInterpState: saving for __modrc_1 command list module-version tell socket subst open eof pwd glob list module-hide pid exec auto_load_index time unknown eval module-forbid lassign lrange module-virtual is-loaded fblocked lsearch auto_import gets case lappend proc break reportTraceExecEnter variable llength module-info auto_execok return linsert is-used error catch clock info split readModuleContent array if fconfigure concat join lreplace source fcopy global switch auto_qualify update close cd for auto_load file append lreverse format unload read package set binary namespace scan versioncmp apply module-tag trace seek while chan flush after vwait dict system continue uplevel foreach lset rename fileevent regexp module-alias lrepeat upvar reportInternalBug formatErrStackTrace encoding expr unset load regsub interp exit setModulesVersion puts incr lindex lsort tclLog uname string
DEBUG [#1:pigz/.version] execute-modulerc: readModuleContent /share/apps/environmentmodules/centos7/general/pigz/.version
DEBUG [#1:pigz/.version] execute-modulerc: setModulesVersion 2.6
DEBUG [#1:pigz/.version] setModuleResolution: pigz/default resolved to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: pigz resolved to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: set symbols 'default' to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: module-version pigz/default = pigz/2.6
DEBUG setState: debug_msg_prefix set to ''
DEBUG setState: specifiedname set to ''
DEBUG setState: modulename set to ''
DEBUG setState: modulefile set to '{}'
DEBUG setState: hiding_threshold set to '0'
DEBUG setModuleResolution: pigz/latest resolved to pigz/2.6
DEBUG setModuleResolution: define auto symbolic version 'pigz/latest' targeting pigz/2.6
Get modules: {pigz/2.4 pigz/default pigz pigz/latest pigz/2.6} matching 'pigz' in '/share/apps/environmentmodules/centos7/general'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/general:pigz:0:rc_defs_included resolve:'
DEBUG getEqArrayKey: key 'pigz' in array 'mod_list' matches 'pigz'
DEBUG getEqArrayKey: key 'pigz/2.6' in array 'mod_list' matches 'pigz/2.6'
Select module: 'pigz/2.6' (/share/apps/environmentmodules/centos7/general/pigz/2.6) matching 'pigz/2.6'
DEBUG getMovementBetweenList: from(/share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles) to() with extfrom() extto()
DEBUG setConf: collection_pin_version set to '0'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/devtools' resolved to '/share/apps/environmentmodules/centos7/devtools'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/general' resolved to '/share/apps/environmentmodules/centos7/general'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/libraries' resolved to '/share/apps/environmentmodules/centos7/libraries'
DEBUG resolvStringWithEnv: '/usr/local/Modules/modulefiles' resolved to '/usr/local/Modules/modulefiles'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG cmdModuleSave: Saving /data/home/user/.module/stash-1695207488863
DEBUG setConf: implicit_default set to '1'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getMovementBetweenList: from(pigz/2.6) to() with extfrom() extto()
DEBUG getMovementBetweenList: from(/share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles) to() with extfrom() extto()
DEBUG lappendState: evalid appended with 'restore-__init__-0'
DEBUG lappendState: unloading_sticky appended with 'pigz/2.6'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG cmdModuleUnload: unloading pigz/2.6 (context=unload, match=match, auto=0, force=0, onlyureq=0, onlyndep=0)
DEBUG lappendState: mode appended with 'unload'
DEBUG getPathToModule: finding 'pigz/2.6' in '' (report_issue=1, look_loaded=match, excdir='')
DEBUG getLoadedMatchingName: 'pigz/2.6' matches 'pigz/2.6'
Select module: 'pigz/2.6' (/share/apps/environmentmodules/centos7/general/pigz/2.6) matching 'pigz/2.6'
DEBUG lappendState: evalid appended with 'unload-pigz/2.6-0'
DEBUG lappendState: msgrecordid appended with 'unload-pigz/2.6-0'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG pushSettings: settings saved (#1)
DEBUG getDependentLoadedModuleList: get loaded mod dependent of 'pigz/2.6' (strong=1, direct=1, nporeq=0, loading=1, sat_constraint=0, being_unload=0)
DEBUG getDependentLoadedModuleList: got ''
DEBUG setConf: auto_handling set to '1'
DEBUG lappendState: modulefile appended with '/share/apps/environmentmodules/centos7/general/pigz/2.6'
DEBUG lappendState: modulename appended with 'pigz/2.6'
DEBUG lappendState: modulenamevr appended with 'pigz/2.6'
DEBUG lappendState: specifiedname appended with 'pigz/2.6'
DEBUG [#1:unload:pigz/2.6] lappendState: debug_msg_prefix appended with '{[#1:unload:pigz/2.6] }'
DEBUG [#1:unload:pigz/2.6] setState: inhibit_interp set to '0'
DEBUG [#1:unload:pigz/2.6] getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG [#1:unload:pigz/2.6] getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG [#1:unload:pigz/2.6] execute-modulefile: creating interp __modfile_unload_1_1
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var tcl_version
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var tcl_interactive
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var ModuleToolVersion
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var ModuleTool
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var auto_path
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var tcl_pkgPath
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var tcl_patchLevel
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var tcl_platform
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile var tcl_library
DEBUG [#1:unload:pigz/2.6] dumpInterpState: saving for __modfile proc list auto_load clock unknown tclLog auto_qualify auto_execok auto_import auto_load_index
DEBUG [#1:unload:pigz/2.6] resetInterpState: saving for __modfile_unload_1_1 command list module-version tell socket subst conflict open eof add-property pwd glob list module-hide pid exec auto_load_index time unknown eval module-forbid lassign lrange append-path module-user module-virtual reportWarning is-loaded fblocked lsearch auto_import gets report case lappend proc setenv break reportTraceExecEnter variable llength module-info set-function prereq auto_execok extensions return linsert is-used error catch clock info split variant readModuleContent array if fconfigure concat join lreplace source fcopy global switch isVerbosityLevel auto_qualify update prereq-any module-log reportError unset-alias close reportCmdTrace cd for auto_load file append lreverse source-sh getenv format unload read package set binary namespace scan always-load versioncmp apply module-tag raiseErrorCount trace remove-property unsetenv seek complete while isWin chan flush after vwait is-saved dict getvariant system continue uplevel is-avail require-fullname remove-path foreach lset rename pushenv chdir fileevent module-whatis regexp uncomplete module-alias x-resource lrepeat upvar formatErrStackTrace reportInternalBug encoding expr unset prepend-path load regsub module-trace module-verbosity unset-function exit family interp set-alias _puts puts incr lindex lsort prereq-all depends-on tclLog uname module string
DEBUG [#1:unload:pigz/2.6] execute-modulefile: readModuleContent /share/apps/environmentmodules/centos7/general/pigz/2.6 1
DEBUG [#1:unload:pigz/2.6] execute-modulefile: module-whatis {adds pigz 2.6 to your environment}
DEBUG [#1:unload:pigz/2.6] execute-modulefile: conflict pigz
DEBUG [#1:unload:pigz/2.6] execute-modulefile: module-info name
DEBUG [#1:unload:pigz/2.6] execute-modulefile: module-info specified
DEBUG [#1:unload:pigz/2.6] execute-modulefile: module-info mode
DEBUG [#1:unload:pigz/2.6] execute-modulefile: puts file6 {20/09/2023 11:58:08 user node 3101946 unload pigz/2.6 pigz/2.6}
DEBUG [#1:unload:pigz/2.6] execute-modulefile: prepend-path PATH /share/apps/centos7/pigz/gcc/4.8.5/2.6/bin
DEBUG [#1:unload:pigz/2.6] unload-path: (PATH /share/apps/centos7/pigz/gcc/4.8.5/2.6/bin) cmd=prepend-path, mode=unload, dflbhv=remove
DEBUG [#1:unload:pigz/2.6] parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=remove, var=PATH, val=/share/apps/centos7/pigz/gcc/4.8.5/2.6/bin, nbval=1)
DEBUG [#1:unload:pigz/2.6] unset-env: __MODULES_PUSHENV_PATH (internal=1, val=)
DEBUG [#1:unload:pigz/2.6] getReferenceCountArray: (var=PATH, delim=:) got '/data/ITSR-Scheduler/8.6.12b/bin/lx-amd64 1 /usr/bin 1 /share/admin/toolsbin 1 /usr/local/bin 1 /tmp/3101946.1.all.q 1 /data/home/user/MyUtils/bin 1 /bin 1 /opt/puppetlabs/bin 1 /share/admin/bin 1 /usr/local/Modules/bin 1 /usr/sbin 1 /share/apps/centos7/pigz/gcc/4.8.5/2.6/bin 1 /usr/local/sbin 1 /usr/lpp/mmfs/bin 1'
DEBUG [#1:unload:pigz/2.6] set-env: PATH=/data/ITSR-Scheduler/8.6.12b/bin/lx-amd64:/usr/local/Modules/bin:/tmp/3101946.1.all.q:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/puppetlabs/bin:/share/admin/bin:/share/admin/toolsbin:/data/home/user/MyUtils/bin
DEBUG [#1:unload:pigz/2.6] interp-sync-env: set var='::env(PATH)', val='/data/ITSR-Scheduler/8.6.12b/bin/lx-amd64:/usr/local/Modules/bin:/tmp/3101946.1.all.q:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/puppetlabs/bin:/share/admin/bin:/share/admin/toolsbin:/data/home/user/MyUtils/bin' on interp(s) __modfile_unload_1_1 __modrc_1
DEBUG [#1:unload:pigz/2.6] execute-modulefile: module-info mode
DEBUG [#1:unload:pigz/2.6] execute-modulefile: exiting /share/apps/environmentmodules/centos7/general/pigz/2.6
DEBUG setState: debug_msg_prefix set to ''
DEBUG setState: specifiedname set to ''
DEBUG setState: modulename set to ''
DEBUG setState: modulenamevr set to ''
DEBUG setState: modulefile set to '{}'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG unload-path: (LOADEDMODULES pigz/2.6) cmd=remove-path, mode=load, dflbhv=remove
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=remove, var=LOADEDMODULES, val=pigz/2.6, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV_LOADEDMODULES (internal=1, val=)
DEBUG unset-env: LOADEDMODULES (internal=0, val=)
DEBUG interp-sync-env: set var='::env(LOADEDMODULES)', val='' on interp(s) __modfile_unload_1_1 __modrc_1
DEBUG unload-path: (--index _LMFILES_ 0) cmd=remove-path, mode=load, dflbhv=remove
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=1, ign_refcount=0, bhv=remove, var=_LMFILES_, val=0, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV__LMFILES_ (internal=1, val=)
DEBUG unset-env: _LMFILES_ (internal=0, val=)
DEBUG interp-sync-env: set var='::env(_LMFILES_)', val='' on interp(s) __modfile_unload_1_1 __modrc_1
DEBUG unsetModuleDependency: unset requirements of 'pigz/2.6'
DEBUG unsetModuleDependency: unset NPO requirements of 'pigz/2.6'
DEBUG unload-path: (__MODULES_LMCONFLICT pigz/2.6&pigz) cmd=remove-path, mode=load, dflbhv=remove
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=remove, var=__MODULES_LMCONFLICT, val=pigz/2.6&pigz, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV___MODULES_LMCONFLICT (internal=1, val=)
DEBUG unset-env: __MODULES_LMCONFLICT (internal=0, val=)
DEBUG interp-sync-env: set var='::env(__MODULES_LMCONFLICT)', val='' on interp(s) __modfile_unload_1_1 __modrc_1
DEBUG unload-path: (__MODULES_LMALTNAME pigz/2.6&pigz/default&pigz&as|pigz/latest) cmd=remove-path, mode=load, dflbhv=remove
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=remove, var=__MODULES_LMALTNAME, val=pigz/2.6&pigz/default&pigz&as|pigz/latest, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV___MODULES_LMALTNAME (internal=1, val=)
DEBUG unset-env: __MODULES_LMALTNAME (internal=0, val=)
DEBUG interp-sync-env: set var='::env(__MODULES_LMALTNAME)', val='' on interp(s) __modfile_unload_1_1 __modrc_1
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG popSettings: previously saved settings flushed (#1)
DEBUG setConf: tag_abbrev set to 'auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS:keep-loaded=kL'
DEBUG setConf: tag_color_name set to ''
DEBUG setConf: term_width set to '0'
DEBUG setState: term_columns set to '213'
DEBUG unsetState: report_sep_next unset

DEBUG setState: report_sep_next set to '1'
Unloading pigz/2.6
  Evaluate modulefile: '/share/apps/environmentmodules/centos7/general/pigz/2.6' as 'pigz/2.6'
DEBUG unsetState: report_sep_next unset

DEBUG setState: report_sep_next set to '1'
DEBUG setState: evalid set to 'restore-__init__-0'
DEBUG setState: msgrecordid set to ''
DEBUG setState: mode set to ''
DEBUG setState: unloading_sticky set to ''
DEBUG lappendState: mode appended with 'load'
DEBUG unload-path: (--ignore-refcount MODULEPATH /usr/local/Modules/modulefiles /share/apps/environmentmodules/centos7/libraries /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/devtools) cmd=remove-path, mode=load, dflbhv=remove
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=1, bhv=remove, var=MODULEPATH, val=/usr/local/Modules/modulefiles /share/apps/environmentmodules/centos7/libraries /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/devtools, nbval=4)
DEBUG unset-env: __MODULES_PUSHENV_MODULEPATH (internal=1, val=)
DEBUG getReferenceCountArray: (var=MODULEPATH, delim=:) got '/share/apps/environmentmodules/centos7/libraries 1 /share/apps/environmentmodules/centos7/general 1 /usr/local/Modules/modulefiles 1 /share/apps/environmentmodules/centos7/devtools 1'
DEBUG unset-env: MODULEPATH (internal=0, val=)
DEBUG interp-sync-env: set var='::env(MODULEPATH)', val='' on interp(s) __modfile_unload_1_1 __modrc_1
DEBUG setState: mode set to ''
DEBUG getMovementBetweenList: from() to() with extfrom() extto()
DEBUG getMovementBetweenList: from() to() with extfrom() extto()
DEBUG unsetConf: implicit_default unset
DEBUG setState: evalid set to ''
DEBUG setState: commandname set to ''
DEBUG setState: always_read_full_file set to ''
DEBUG setState: autoinit set to '0'
DEBUG setState: shelltype set to 'sh'
DEBUG setState: error_count set to '0'
DEBUG setState: return_false set to '0'

$ module avail pigz --debug
DEBUG setState: cmdline set to '/usr/local/Modules/libexec/modulecmd.tcl bash avail pigz --debug'
DEBUG setConf: tcl_ext_lib set to '/usr/local/Modules/lib/libtclenvmodules.so'
DEBUG Load Tcl extension library (/usr/local/Modules/lib/libtclenvmodules.so)
DEBUG setState: tcl_ext_lib_loaded set to '1'
DEBUG setConf: siteconfig set to '/usr/local/Modules/etc/siteconfig.tcl'
DEBUG sourceSiteConfig: Source site configuration (/usr/local/Modules/etc/siteconfig.tcl)
DEBUG setState: siteconfig_loaded set to '1'
DEBUG setConf: locked_configs set to ''
DEBUG setState: supported_shells set to 'sh bash ksh zsh csh tcsh fish cmd tcl perl python ruby lisp cmake r'
DEBUG setState: shell set to 'bash'
DEBUG setState: subcmd set to 'avail'
DEBUG setState: subcmd_args set to 'pigz'
DEBUG setState: init_error_report set to '1'
DEBUG setConf: verbosity set to 'debug'
DEBUG setState: is_stderr_tty set to '1'
DEBUG setConf: term_background set to 'dark'
DEBUG setConf: colors set to 'hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44:kL=30;48;5;109'
DEBUG setConf: color set to '1'
DEBUG setConf: pager set to '/usr/bin/less -eFKRX'
DEBUG setState: paginate set to '1'
DEBUG setState: report_format set to 'regular'
DEBUG setState: reportfd set to 'file7'
DEBUG setState: timer set to '0'
DEBUG lappendState: modulefile appended with '{}'
DEBUG parseModuleCommandName: (command=avail, cmdvalid=1, cmdempty=0)
DEBUG setConf: avail_indepth set to '1'
DEBUG setConf: search_match set to 'starts_with'
DEBUG parseModuleCommandArgs: (show_oneperline=0, show_mtime=0, show_filter=, search_filter=, search_match=starts_with, dump_state=0, addpath_pos=prepend, not_req=0, tag_list=, otherargs=pigz)
DEBUG setConf: advanced_version_spec set to '1'
DEBUG setConf: variant_shortcut set to ''
DEBUG setModuleVersSpec: Set module 'pigz' (escglob 'pigz'),  module name '.' (re '\.'), module root 'pigz', version cmp 'eq', version(s) '', variant(s) '' and module name version spec 'pigz' for argument 'pigz' (raw 'pigz'), extra specifier(s) '' and matching available module(s) ''
DEBUG lappendState: always_read_full_file appended with '0'
DEBUG lappendState: commandname appended with 'avail'
DEBUG setState: rc_running set to '1'
DEBUG setConf: ignore_user_rc set to '0'
DEBUG unsetState: rc_running unset
DEBUG setConf: avail_output set to 'modulepath:alias:dirwsym:sym:tag:variantifspec:key'
DEBUG getModules: get 'pigz' in  (fetch_mtime=0, search=starts_with wild rc_alias_only, filter=noplaindir)
DEBUG setConf: implicit_default set to '1'
DEBUG setConf: icase set to 'search'
DEBUG setConf: extended_default set to '1'
DEBUG setState: lm_info_cached set to '1'
DEBUG setState: is_win set to '0'
DEBUG setState: path_separator set to ':'
DEBUG setState: sub1_separator set to '&'
DEBUG setState: sub2_separator set to '|'
DEBUG cacheCurrentModules: 0 loaded
DEBUG setState: inhibit_errreport set to '0'
Get modules: {} matching 'pigz' in ''
DEBUG getModules: create cache entry ':pigz:0:starts_with wild rc_alias_only:noplaindir'
DEBUG setConf: tag_abbrev set to 'auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS:keep-loaded=kL'
DEBUG setConf: tag_color_name set to ''
DEBUG displayElementList: header=global/user modulerc, sgrkey=hi, hstyle=sepline, elt_cnt=0, max_len=0, one_per_line=0, display_idx=0, start_idx=1
DEBUG setState: error_count set to '0'
DEBUG setState: error_count set to '1'
DEBUG setState: false_rendered set to '1'
DEBUG setState: shelltype set to 'sh'
DEBUG setState: inhibit_errreport set to '0'
DEBUG setState: error_count set to '2'
ERROR: No module path defined

$ module stashpop --debug
DEBUG setState: cmdline set to '/usr/local/Modules/libexec/modulecmd.tcl bash stashpop --debug'
DEBUG setConf: tcl_ext_lib set to '/usr/local/Modules/lib/libtclenvmodules.so'
DEBUG Load Tcl extension library (/usr/local/Modules/lib/libtclenvmodules.so)
DEBUG setState: tcl_ext_lib_loaded set to '1'
DEBUG setConf: siteconfig set to '/usr/local/Modules/etc/siteconfig.tcl'
DEBUG sourceSiteConfig: Source site configuration (/usr/local/Modules/etc/siteconfig.tcl)
DEBUG setState: siteconfig_loaded set to '1'
DEBUG setConf: locked_configs set to ''
DEBUG setState: supported_shells set to 'sh bash ksh zsh csh tcsh fish cmd tcl perl python ruby lisp cmake r'
DEBUG setState: shell set to 'bash'
DEBUG setState: subcmd set to 'stashpop'
DEBUG setState: subcmd_args set to ''
DEBUG setState: init_error_report set to '1'
DEBUG setConf: verbosity set to 'debug'
DEBUG setState: is_stderr_tty set to '1'
DEBUG setConf: term_background set to 'dark'
DEBUG setConf: colors set to 'hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44:kL=30;48;5;109'
DEBUG setConf: color set to '1'
DEBUG setConf: pager set to '/usr/bin/less -eFKRX'
DEBUG setState: paginate set to '1'
DEBUG setState: report_format set to 'regular'
DEBUG setState: reportfd set to 'file7'
DEBUG setState: timer set to '0'
DEBUG lappendState: modulefile appended with '{}'
DEBUG parseModuleCommandName: (command=stashpop, cmdvalid=1, cmdempty=0)
DEBUG setConf: avail_indepth set to '1'
DEBUG setConf: search_match set to 'starts_with'
DEBUG parseModuleCommandArgs: (show_oneperline=0, show_mtime=0, show_filter=, search_filter=, search_match=starts_with, dump_state=0, addpath_pos=prepend, not_req=0, tag_list=, otherargs=)
DEBUG setConf: advanced_version_spec set to '1'
DEBUG setConf: variant_shortcut set to ''
DEBUG lappendState: always_read_full_file appended with '1'
DEBUG lappendState: commandname appended with 'stashpop'
DEBUG setState: rc_running set to '1'
DEBUG setConf: ignore_user_rc set to '0'
DEBUG unsetState: rc_running unset
DEBUG setConf: implicit_default set to '1'
DEBUG setState: cwd set to '/data/home/user'
DEBUG setModuleVersSpec: Set module 'pigz' (escglob 'pigz'),  module name '.' (re '\.'), module root 'pigz', version cmp 'eq', version(s) '', variant(s) '' and module name version spec 'pigz' for argument 'pigz' (raw 'pigz'), extra specifier(s) '' and matching available module(s) ''
DEBUG setState: lm_info_cached set to '1'
DEBUG setState: is_win set to '0'
DEBUG setState: path_separator set to ':'
DEBUG setState: sub1_separator set to '&'
DEBUG setState: sub2_separator set to '|'
DEBUG cacheCurrentModules: 0 loaded
DEBUG setConf: icase set to 'search'
DEBUG setConf: extended_default set to '1'
DEBUG getMovementBetweenList: from() to(pigz) with extfrom() extto()
DEBUG getMovementBetweenList: from() to(/share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles) with extfrom() extto()
DEBUG lappendState: evalid appended with 'restore-stash-1695207488863-0'
DEBUG getMovementBetweenList: from() to(pigz) with extfrom() extto()
DEBUG getMovementBetweenList: from() to(/share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles) with extfrom() extto()
DEBUG unsetConf: implicit_default unset
DEBUG lappendState: mode appended with 'load'
DEBUG add-path: (--ignore-refcount MODULEPATH /share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles) cmd=append-path, mode=load, dflbhv=append
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=1, bhv=append, var=MODULEPATH, val=/share/apps/environmentmodules/centos7/devtools /share/apps/environmentmodules/centos7/general /share/apps/environmentmodules/centos7/libraries /usr/local/Modules/modulefiles, nbval=4)
DEBUG unset-env: __MODULES_PUSHENV_MODULEPATH (internal=1, val=)
DEBUG getReferenceCountArray: (var=MODULEPATH, delim=:) got ''
DEBUG set-env: MODULEPATH=/share/apps/environmentmodules/centos7/devtools:/share/apps/environmentmodules/centos7/general:/share/apps/environmentmodules/centos7/libraries:/usr/local/Modules/modulefiles
DEBUG setState: mode set to ''
DEBUG cmdModuleLoad: loading pigz (context=load, uasked=1, tryload=0, loadany=0)
DEBUG lappendState: mode appended with 'load'
DEBUG getPathToModule: finding 'pigz' in '' (report_issue=1, look_loaded=no, excdir='')
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/devtools' resolved to '/share/apps/environmentmodules/centos7/devtools'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/general' resolved to '/share/apps/environmentmodules/centos7/general'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/libraries' resolved to '/share/apps/environmentmodules/centos7/libraries'
DEBUG resolvStringWithEnv: '/usr/local/Modules/modulefiles' resolved to '/usr/local/Modules/modulefiles'
DEBUG setConf: implicit_default set to '1'
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/devtools (fetch_mtime=0, search=rc_defs_included resolve, filter=)
DEBUG findModules: finding 'pigz' in /share/apps/environmentmodules/centos7/devtools (depthlvl=0, fetch_mtime=0)
DEBUG setConf: ignore_cache set to '0'
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/devtools/.modulecache' cannot be found or read
DEBUG setConf: ignored_dirs set to 'CVS RCS SCCS .svn .git .SYNC .sos'
DEBUG findModules: found 
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/devtools:pigz:0:0'
DEBUG setState: inhibit_errreport set to '0'
Get modules: {} matching 'pigz' in '/share/apps/environmentmodules/centos7/devtools'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/devtools:pigz:0:rc_defs_included resolve:'
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/general (fetch_mtime=0, search=rc_defs_included resolve, filter=)
DEBUG findModules: finding 'pigz' in /share/apps/environmentmodules/centos7/general (depthlvl=0, fetch_mtime=0)
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/general/.modulecache' cannot be found or read
DEBUG setConf: mcookie_check set to 'always'
DEBUG findModules: found pigz/2.4 pigz/.version pigz/2.6
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/general:pigz:0:0'
DEBUG lappendState: modulefile appended with '/share/apps/environmentmodules/centos7/general/pigz/.version'
DEBUG lappendState: modulename appended with 'pigz/.version'
DEBUG lappendState: specifiedname appended with 'pigz/.version'
DEBUG [#1:pigz/.version] lappendState: debug_msg_prefix appended with '{[#1:pigz/.version] }'
Evaluate modulerc: '/share/apps/environmentmodules/centos7/general/pigz/.version'
DEBUG [#1:pigz/.version] execute-modulerc: creating interp __modrc_1
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_version
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_interactive
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var ModuleToolVersion
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var ModuleTool
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var auto_path
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_pkgPath
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_patchLevel
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_platform
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_library
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc proc list auto_load clock unknown tclLog auto_qualify auto_execok auto_import auto_load_index
DEBUG [#1:pigz/.version] resetInterpState: saving for __modrc_1 command list module-version tell socket subst open eof pwd glob list module-hide pid exec auto_load_index time unknown eval module-forbid lassign lrange module-virtual is-loaded fblocked lsearch auto_import gets case lappend proc break reportTraceExecEnter variable llength module-info auto_execok return linsert is-used error catch clock info split readModuleContent array if fconfigure concat join lreplace source fcopy global switch auto_qualify update close cd for auto_load file append lreverse format unload read package set binary namespace scan versioncmp apply module-tag trace seek while chan flush after vwait dict system continue uplevel foreach lset rename fileevent regexp module-alias lrepeat upvar reportInternalBug formatErrStackTrace encoding expr unset load regsub interp exit setModulesVersion puts incr lindex lsort tclLog uname string
DEBUG [#1:pigz/.version] execute-modulerc: readModuleContent /share/apps/environmentmodules/centos7/general/pigz/.version
DEBUG [#1:pigz/.version] execute-modulerc: setModulesVersion 2.6
DEBUG [#1:pigz/.version] setModuleResolution: pigz/default resolved to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: pigz resolved to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: set symbols 'default' to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: module-version pigz/default = pigz/2.6
DEBUG setState: debug_msg_prefix set to ''
DEBUG setState: specifiedname set to ''
DEBUG setState: modulename set to ''
DEBUG setState: modulefile set to '{}'
DEBUG setState: hiding_threshold set to '0'
DEBUG setModuleResolution: pigz/latest resolved to pigz/2.6
DEBUG setModuleResolution: define auto symbolic version 'pigz/latest' targeting pigz/2.6
Get modules: {pigz/2.4 pigz/default pigz pigz/latest pigz/2.6} matching 'pigz' in '/share/apps/environmentmodules/centos7/general'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/general:pigz:0:rc_defs_included resolve:'
DEBUG getEqArrayKey: key 'pigz' in array 'mod_list' matches 'pigz'
DEBUG getEqArrayKey: key 'pigz/2.6' in array 'mod_list' matches 'pigz/2.6'
Select module: 'pigz/2.6' (/share/apps/environmentmodules/centos7/general/pigz/2.6) matching 'pigz/2.6'
DEBUG lappendState: evalid appended with 'load-pigz/2.6-0'
DEBUG lappendState: msgrecordid appended with 'load-pigz/2.6-0'
DEBUG setConf: unload_match_order set to 'returnlast'
DEBUG getLoadedMatchingName: '' matches 'pigz/2.6'
DEBUG pushSettings: settings saved (#1)
DEBUG setConf: auto_handling set to '1'
DEBUG doesModuleConflict: 'pigz/2.6' conflicts with '' (declared as '')
DEBUG getDependentLoadedModuleList: get loaded mod dependent of '' (strong=0, direct=0, nporeq=0, loading=0, sat_constraint=1, being_unload=0)
DEBUG getDependentLoadedModuleList: got ''
DEBUG getUnmetDependentLoadedModuleList: got ''
DEBUG cmdModuleLoad: depre mod list is ''
DEBUG lappendState: modulefile appended with '/share/apps/environmentmodules/centos7/general/pigz/2.6'
DEBUG lappendState: modulename appended with 'pigz/2.6'
DEBUG lappendState: modulenamevr appended with 'pigz/2.6'
DEBUG lappendState: specifiedname appended with 'pigz'
DEBUG [#1:load:pigz/2.6] lappendState: debug_msg_prefix appended with '{[#1:load:pigz/2.6] }'
DEBUG [#1:load:pigz/2.6] setState: inhibit_interp set to '0'
DEBUG [#1:load:pigz/2.6] execute-modulefile: creating interp __modfile_load_1_1
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var tcl_version
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var tcl_interactive
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var ModuleToolVersion
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var ModuleTool
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var auto_path
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var tcl_pkgPath
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var tcl_patchLevel
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var tcl_platform
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile var tcl_library
DEBUG [#1:load:pigz/2.6] dumpInterpState: saving for __modfile proc list auto_load clock unknown tclLog auto_qualify auto_execok auto_import auto_load_index
DEBUG [#1:load:pigz/2.6] resetInterpState: saving for __modfile_load_1_1 command list module-version tell socket subst conflict open eof add-property pwd glob list module-hide pid exec auto_load_index time unknown eval module-forbid lassign lrange append-path module-user module-virtual reportWarning is-loaded fblocked lsearch auto_import gets report case lappend proc setenv break reportTraceExecEnter variable llength module-info set-function prereq auto_execok extensions return linsert is-used error catch clock info split variant readModuleContent array if fconfigure concat join lreplace source fcopy global switch isVerbosityLevel auto_qualify update prereq-any module-log reportError unset-alias close reportCmdTrace cd for auto_load file append lreverse source-sh getenv format unload read package set binary namespace scan always-load versioncmp apply module-tag raiseErrorCount trace remove-property unsetenv seek complete while isWin chan flush after vwait is-saved dict getvariant system continue uplevel is-avail require-fullname remove-path foreach lset rename pushenv chdir fileevent module-whatis regexp uncomplete module-alias x-resource lrepeat upvar formatErrStackTrace reportInternalBug encoding expr unset prepend-path load regsub module-trace module-verbosity unset-function exit family interp set-alias _puts puts incr lindex lsort prereq-all depends-on tclLog uname module string
DEBUG [#1:load:pigz/2.6] execute-modulefile: readModuleContent /share/apps/environmentmodules/centos7/general/pigz/2.6 1
DEBUG [#1:load:pigz/2.6] execute-modulefile: module-whatis {adds pigz 2.6 to your environment}
DEBUG [#1:load:pigz/2.6] execute-modulefile: conflict pigz
DEBUG [#1:load:pigz/2.6] getLoadedMatchingName: '' matches 'pigz'
DEBUG [#1:load:pigz/2.6] execute-modulefile: module-info name
DEBUG [#1:load:pigz/2.6] execute-modulefile: module-info specified
DEBUG [#1:load:pigz/2.6] execute-modulefile: module-info mode
DEBUG [#1:load:pigz/2.6] execute-modulefile: puts file6 {20/09/2023 11:58:45 user node 3101946 load pigz pigz/2.6}
DEBUG [#1:load:pigz/2.6] execute-modulefile: prepend-path PATH /share/apps/centos7/pigz/gcc/4.8.5/2.6/bin
DEBUG [#1:load:pigz/2.6] add-path: (PATH /share/apps/centos7/pigz/gcc/4.8.5/2.6/bin) cmd=prepend-path, mode=load, dflbhv=prepend
DEBUG [#1:load:pigz/2.6] parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=prepend, var=PATH, val=/share/apps/centos7/pigz/gcc/4.8.5/2.6/bin, nbval=1)
DEBUG [#1:load:pigz/2.6] unset-env: __MODULES_PUSHENV_PATH (internal=1, val=)
DEBUG [#1:load:pigz/2.6] getReferenceCountArray: (var=PATH, delim=:) got '/data/ITSR-Scheduler/8.6.12b/bin/lx-amd64 1 /usr/bin 1 /usr/local/bin 1 /share/admin/toolsbin 1 /tmp/3101946.1.all.q 1 /data/home/user/MyUtils/bin 1 /bin 1 /opt/puppetlabs/bin 1 /share/admin/bin 1 /usr/local/Modules/bin 1 /usr/sbin 1 /usr/local/sbin 1 /usr/lpp/mmfs/bin 1'
DEBUG [#1:load:pigz/2.6] set-env: PATH=/share/apps/centos7/pigz/gcc/4.8.5/2.6/bin:/data/ITSR-Scheduler/8.6.12b/bin/lx-amd64:/usr/local/Modules/bin:/tmp/3101946.1.all.q:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/puppetlabs/bin:/share/admin/bin:/share/admin/toolsbin:/data/home/user/MyUtils/bin
DEBUG [#1:load:pigz/2.6] interp-sync-env: set var='::env(PATH)', val='/share/apps/centos7/pigz/gcc/4.8.5/2.6/bin:/data/ITSR-Scheduler/8.6.12b/bin/lx-amd64:/usr/local/Modules/bin:/tmp/3101946.1.all.q:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/puppetlabs/bin:/share/admin/bin:/share/admin/toolsbin:/data/home/user/MyUtils/bin' on interp(s) __modfile_load_1_1 __modrc_1
DEBUG [#1:load:pigz/2.6] execute-modulefile: module-info mode
DEBUG [#1:load:pigz/2.6] setModuleVersSpec: Set module 'pigz/2.6' (escglob 'pigz/2.6'),  module name 'pigz' (re ''), module root 'pigz', version cmp 'eq', version(s) '', variant(s) '' and module name version spec 'pigz/2.6' for argument 'pigz/2.6' (raw 'pigz/2.6'), extra specifier(s) '' and matching available module(s) ''
DEBUG [#1:load:pigz/2.6] execute-modulefile: exiting /share/apps/environmentmodules/centos7/general/pigz/2.6
DEBUG setState: debug_msg_prefix set to ''
DEBUG setState: specifiedname set to ''
DEBUG setState: modulename set to ''
DEBUG setState: modulenamevr set to ''
DEBUG setState: modulefile set to '{}'
DEBUG doesModuleConflict: 'pigz/2.6' conflicts with '' (declared as '')
DEBUG setState: force set to '0'
DEBUG add-path: (LOADEDMODULES pigz/2.6) cmd=append-path, mode=load, dflbhv=append
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=append, var=LOADEDMODULES, val=pigz/2.6, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV_LOADEDMODULES (internal=1, val=)
DEBUG set-env: LOADEDMODULES=pigz/2.6
DEBUG interp-sync-env: set var='::env(LOADEDMODULES)', val='pigz/2.6' on interp(s) __modfile_load_1_1 __modrc_1
DEBUG add-path: (--duplicates _LMFILES_ /share/apps/environmentmodules/centos7/general/pigz/2.6) cmd=append-path, mode=load, dflbhv=append
DEBUG parsePathCommandArgs: (delim=:, allow_dup=1, idx_val=0, ign_refcount=0, bhv=append, var=_LMFILES_, val=/share/apps/environmentmodules/centos7/general/pigz/2.6, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV__LMFILES_ (internal=1, val=)
DEBUG set-env: _LMFILES_=/share/apps/environmentmodules/centos7/general/pigz/2.6
DEBUG interp-sync-env: set var='::env(_LMFILES_)', val='/share/apps/environmentmodules/centos7/general/pigz/2.6' on interp(s) __modfile_load_1_1 __modrc_1
DEBUG doesModuleConflict: 'pigz/2.6' conflicts with '' (declared as '')
DEBUG setModuleDependency: set requirements of 'pigz/2.6' to ''
DEBUG setModuleDependency: set NPO requirements of 'pigz/2.6' to ''
DEBUG add-path: (__MODULES_LMCONFLICT pigz/2.6&pigz) cmd=append-path, mode=load, dflbhv=append
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=append, var=__MODULES_LMCONFLICT, val=pigz/2.6&pigz, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV___MODULES_LMCONFLICT (internal=1, val=)
DEBUG set-env: __MODULES_LMCONFLICT=pigz/2.6&pigz
DEBUG interp-sync-env: set var='::env(__MODULES_LMCONFLICT)', val='pigz/2.6&pigz' on interp(s) __modfile_load_1_1 __modrc_1
DEBUG add-path: (__MODULES_LMALTNAME pigz/2.6&pigz/default&pigz&as|pigz/latest) cmd=append-path, mode=load, dflbhv=append
DEBUG parsePathCommandArgs: (delim=:, allow_dup=0, idx_val=0, ign_refcount=0, bhv=append, var=__MODULES_LMALTNAME, val=pigz/2.6&pigz/default&pigz&as|pigz/latest, nbval=1)
DEBUG unset-env: __MODULES_PUSHENV___MODULES_LMALTNAME (internal=1, val=)
DEBUG set-env: __MODULES_LMALTNAME=pigz/2.6&pigz/default&pigz&as|pigz/latest
DEBUG interp-sync-env: set var='::env(__MODULES_LMALTNAME)', val='pigz/2.6&pigz/default&pigz&as|pigz/latest' on interp(s) __modfile_load_1_1 __modrc_1
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG popSettings: previously saved settings flushed (#1)
DEBUG setConf: tag_abbrev set to 'auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS:keep-loaded=kL'
DEBUG setConf: tag_color_name set to ''
DEBUG setConf: term_width set to '0'
DEBUG setState: term_columns set to '213'
DEBUG unsetState: report_sep_next unset

DEBUG setState: report_sep_next set to '1'
Loading pigz/2.6
  Evaluate modulefile: '/share/apps/environmentmodules/centos7/general/pigz/2.6' as 'pigz/2.6'
DEBUG unsetState: report_sep_next unset

DEBUG setState: report_sep_next set to '1'
DEBUG setState: evalid set to 'restore-stash-1695207488863-0'
DEBUG setState: msgrecordid set to ''
DEBUG setState: mode set to ''
DEBUG setState: evalid set to ''
DEBUG setState: commandname set to ''
DEBUG setState: always_read_full_file set to ''
DEBUG setState: autoinit set to '0'
DEBUG setState: shelltype set to 'sh'
DEBUG setState: error_count set to '0'
DEBUG setState: return_false set to '0'

$ module avail pigz --debug
DEBUG setState: cmdline set to '/usr/local/Modules/libexec/modulecmd.tcl bash avail pigz --debug'
DEBUG setConf: tcl_ext_lib set to '/usr/local/Modules/lib/libtclenvmodules.so'
DEBUG Load Tcl extension library (/usr/local/Modules/lib/libtclenvmodules.so)
DEBUG setState: tcl_ext_lib_loaded set to '1'
DEBUG setConf: siteconfig set to '/usr/local/Modules/etc/siteconfig.tcl'
DEBUG sourceSiteConfig: Source site configuration (/usr/local/Modules/etc/siteconfig.tcl)
DEBUG setState: siteconfig_loaded set to '1'
DEBUG setConf: locked_configs set to ''
DEBUG setState: supported_shells set to 'sh bash ksh zsh csh tcsh fish cmd tcl perl python ruby lisp cmake r'
DEBUG setState: shell set to 'bash'
DEBUG setState: subcmd set to 'avail'
DEBUG setState: subcmd_args set to 'pigz'
DEBUG setState: init_error_report set to '1'
DEBUG setConf: verbosity set to 'debug'
DEBUG setState: is_stderr_tty set to '1'
DEBUG setConf: term_background set to 'dark'
DEBUG setConf: colors set to 'hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44:kL=30;48;5;109'
DEBUG setConf: color set to '1'
DEBUG setConf: pager set to '/usr/bin/less -eFKRX'
DEBUG setState: paginate set to '1'
DEBUG setState: report_format set to 'regular'
DEBUG setState: reportfd set to 'file7'
DEBUG setState: timer set to '0'
DEBUG lappendState: modulefile appended with '{}'
DEBUG parseModuleCommandName: (command=avail, cmdvalid=1, cmdempty=0)
DEBUG setConf: avail_indepth set to '1'
DEBUG setConf: search_match set to 'starts_with'
DEBUG parseModuleCommandArgs: (show_oneperline=0, show_mtime=0, show_filter=, search_filter=, search_match=starts_with, dump_state=0, addpath_pos=prepend, not_req=0, tag_list=, otherargs=pigz)
DEBUG setConf: advanced_version_spec set to '1'
DEBUG setConf: variant_shortcut set to ''
DEBUG setModuleVersSpec: Set module 'pigz' (escglob 'pigz'),  module name '.' (re '\.'), module root 'pigz', version cmp 'eq', version(s) '', variant(s) '' and module name version spec 'pigz' for argument 'pigz' (raw 'pigz'), extra specifier(s) '' and matching available module(s) ''
DEBUG lappendState: always_read_full_file appended with '0'
DEBUG lappendState: commandname appended with 'avail'
DEBUG setState: rc_running set to '1'
DEBUG setConf: ignore_user_rc set to '0'
DEBUG unsetState: rc_running unset
DEBUG setConf: avail_output set to 'modulepath:alias:dirwsym:sym:tag:variantifspec:key'
DEBUG getModules: get 'pigz' in  (fetch_mtime=0, search=starts_with wild rc_alias_only, filter=noplaindir)
DEBUG setConf: implicit_default set to '1'
DEBUG setConf: icase set to 'search'
DEBUG setConf: extended_default set to '1'
DEBUG setState: lm_info_cached set to '1'
DEBUG setState: is_win set to '0'
DEBUG setState: path_separator set to ':'
DEBUG setState: sub1_separator set to '&'
DEBUG setState: sub2_separator set to '|'
DEBUG setConf: unload_match_order set to 'returnlast'
DEBUG doesModuleConflict: 'pigz/2.6' conflicts with '' (declared as '')
DEBUG setModuleDependency: set requirements of 'pigz/2.6' to ''
DEBUG setModuleDependency: set NPO requirements of 'pigz/2.6' to ''
DEBUG cacheCurrentModules: 1 loaded
DEBUG setState: inhibit_errreport set to '0'
Get modules: {} matching 'pigz' in ''
DEBUG getModules: create cache entry ':pigz:0:starts_with wild rc_alias_only:noplaindir'
DEBUG setConf: tag_abbrev set to 'auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS:keep-loaded=kL'
DEBUG setConf: tag_color_name set to ''
DEBUG displayElementList: header=global/user modulerc, sgrkey=hi, hstyle=sepline, elt_cnt=0, max_len=0, one_per_line=0, display_idx=0, start_idx=1
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/devtools' resolved to '/share/apps/environmentmodules/centos7/devtools'
DEBUG setState: cwd set to '/data/home/user'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/general' resolved to '/share/apps/environmentmodules/centos7/general'
DEBUG resolvStringWithEnv: '/share/apps/environmentmodules/centos7/libraries' resolved to '/share/apps/environmentmodules/centos7/libraries'
DEBUG resolvStringWithEnv: '/usr/local/Modules/modulefiles' resolved to '/usr/local/Modules/modulefiles'
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/devtools (fetch_mtime=0, search=starts_with wild, filter=noplaindir)
DEBUG findModules: finding 'pigz*' in /share/apps/environmentmodules/centos7/devtools (depthlvl=0, fetch_mtime=0)
DEBUG setConf: ignore_cache set to '0'
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/devtools/.modulecache' cannot be found or read
DEBUG setConf: ignored_dirs set to 'CVS RCS SCCS .svn .git .SYNC .sos'
DEBUG findModules: found 
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/devtools:pigz*:0:0'
Get modules: {} matching 'pigz' in '/share/apps/environmentmodules/centos7/devtools'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/devtools:pigz:0:starts_with wild:noplaindir'
DEBUG displayElementList: header=/share/apps/environmentmodules/centos7/devtools, sgrkey=mp, hstyle=sepline, elt_cnt=0, max_len=0, one_per_line=0, display_idx=0, start_idx=1
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/general (fetch_mtime=0, search=starts_with wild, filter=noplaindir)
DEBUG findModules: finding 'pigz*' in /share/apps/environmentmodules/centos7/general (depthlvl=0, fetch_mtime=0)
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/general/.modulecache' cannot be found or read
DEBUG setConf: mcookie_check set to 'always'
DEBUG findModules: found pigz/2.4 pigz/.version pigz/2.6
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/general:pigz*:0:0'
DEBUG lappendState: modulefile appended with '/share/apps/environmentmodules/centos7/general/pigz/.version'
DEBUG lappendState: modulename appended with 'pigz/.version'
DEBUG lappendState: specifiedname appended with 'pigz/.version'
DEBUG [#1:pigz/.version] lappendState: debug_msg_prefix appended with '{[#1:pigz/.version] }'
Evaluate modulerc: '/share/apps/environmentmodules/centos7/general/pigz/.version'
DEBUG [#1:pigz/.version] execute-modulerc: creating interp __modrc_1
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_version
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_interactive
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var ModuleToolVersion
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var ModuleTool
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var auto_path
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_pkgPath
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_patchLevel
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_platform
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc var tcl_library
DEBUG [#1:pigz/.version] dumpInterpState: saving for __modrc proc list auto_load clock unknown tclLog auto_qualify auto_execok auto_import auto_load_index
DEBUG [#1:pigz/.version] resetInterpState: saving for __modrc_1 command list module-version tell socket subst open eof pwd glob list module-hide pid exec auto_load_index time unknown eval module-forbid lassign lrange module-virtual is-loaded fblocked lsearch auto_import gets case lappend proc break reportTraceExecEnter variable llength module-info auto_execok return linsert is-used error catch clock info split readModuleContent array if fconfigure concat join lreplace source fcopy global switch auto_qualify update close cd for auto_load file append lreverse format unload read package set binary namespace scan versioncmp apply module-tag trace seek while chan flush after vwait dict system continue uplevel foreach lset rename fileevent regexp module-alias lrepeat upvar reportInternalBug formatErrStackTrace encoding expr unset load regsub interp exit setModulesVersion puts incr lindex lsort tclLog uname string
DEBUG [#1:pigz/.version] execute-modulerc: readModuleContent /share/apps/environmentmodules/centos7/general/pigz/.version
DEBUG [#1:pigz/.version] execute-modulerc: setModulesVersion 2.6
DEBUG [#1:pigz/.version] setModuleResolution: pigz/default resolved to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: pigz resolved to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: set symbols 'default' to pigz/2.6
DEBUG [#1:pigz/.version] setModuleResolution: module-version pigz/default = pigz/2.6
DEBUG setState: debug_msg_prefix set to ''
DEBUG setState: specifiedname set to ''
DEBUG setState: modulename set to ''
DEBUG setState: modulefile set to '{}'
DEBUG setState: hiding_threshold set to '0'
DEBUG setModuleResolution: pigz/latest resolved to pigz/2.6
DEBUG setModuleResolution: define auto symbolic version 'pigz/latest' targeting pigz/2.6
Get modules: {pigz/2.4 pigz/default pigz/latest pigz/2.6} matching 'pigz' in '/share/apps/environmentmodules/centos7/general'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/general:pigz:0:starts_with wild:noplaindir'
DEBUG getVersAliasList: 'pigz/2.4' has symbolic version list ''
DEBUG getVersAliasList: 'pigz/latest' has symbolic version list ''
DEBUG getVersAliasList: 'pigz/default' has symbolic version list ''
DEBUG getVersAliasList: 'pigz/2.6' has symbolic version list 'default'
DEBUG getTagList: 'pigz/2.6' has tag list 'loaded'
DEBUG displayElementList: header=/share/apps/environmentmodules/centos7/general, sgrkey=mp, hstyle=sepline, elt_cnt=2, max_len=8, one_per_line=0, display_idx=0, start_idx=1
DEBUG setConf: term_width set to '0'
DEBUG setState: term_columns set to '213'
---------------------------------------------------------------------------------- /share/apps/environmentmodules/centos7/general -----------------------------------------------------------------------------------
DEBUG displayElementList: list={pigz/2.4} {pigz/2.6}
DEBUG displayElementList: rows/cols=1/21, lastcol_item_cnt=0
pigz/2.4  pigz/2.6  
DEBUG unsetState: report_sep_next unset

DEBUG setState: report_sep_next set to '1'
DEBUG getModules: get 'pigz' in /share/apps/environmentmodules/centos7/libraries (fetch_mtime=0, search=starts_with wild, filter=noplaindir)
DEBUG findModules: finding 'pigz*' in /share/apps/environmentmodules/centos7/libraries (depthlvl=0, fetch_mtime=0)
DEBUG findModulesInCacheFile: cache file '/share/apps/environmentmodules/centos7/libraries/.modulecache' cannot be found or read
DEBUG findModules: found 
DEBUG findModules: create cache entry '/share/apps/environmentmodules/centos7/libraries:pigz*:0:0'
Get modules: {} matching 'pigz' in '/share/apps/environmentmodules/centos7/libraries'
DEBUG getModules: create cache entry '/share/apps/environmentmodules/centos7/libraries:pigz:0:starts_with wild:noplaindir'
DEBUG displayElementList: header=/share/apps/environmentmodules/centos7/libraries, sgrkey=mp, hstyle=sepline, elt_cnt=0, max_len=0, one_per_line=0, display_idx=0, start_idx=1
DEBUG getModules: get 'pigz' in /usr/local/Modules/modulefiles (fetch_mtime=0, search=starts_with wild, filter=noplaindir)
DEBUG findModules: finding 'pigz*' in /usr/local/Modules/modulefiles (depthlvl=0, fetch_mtime=0)
DEBUG findModulesInCacheFile: cache file '/usr/local/Modules/modulefiles/.modulecache' cannot be found or read
DEBUG findModules: found 
DEBUG findModules: create cache entry '/usr/local/Modules/modulefiles:pigz*:0:0'
Get modules: {} matching 'pigz' in '/usr/local/Modules/modulefiles'
DEBUG getModules: create cache entry '/usr/local/Modules/modulefiles:pigz:0:starts_with wild:noplaindir'
DEBUG displayElementList: header=/usr/local/Modules/modulefiles, sgrkey=mp, hstyle=sepline, elt_cnt=0, max_len=0, one_per_line=0, display_idx=0, start_idx=1
Key:
DEBUG displayElementList: header=noheader, sgrkey=, hstyle=, elt_cnt=3, max_len=15, one_per_line=0, display_idx=0, start_idx=0
DEBUG displayElementList: list={loaded} {modulepath} {default-version}
DEBUG displayElementList: rows/cols=1/12, lastcol_item_cnt=0
loaded  modulepath  default-version  
DEBUG unsetState: report_sep_next unset

DEBUG setState: report_sep_next set to '1'
DEBUG setState: inhibit_errreport set to '0'
DEBUG setState: commandname set to ''
DEBUG setState: always_read_full_file set to ''
DEBUG setState: autoinit set to '0'
DEBUG setState: error_count set to '0'
DEBUG setState: return_false set to '0'

Additional context

$ qlogin
$ env | grep LMINIT

$ ssh node
$ env | grep LMINIT
__MODULES_LMINIT=module use --append /share/apps/environmentmodules/centos7/devtools:module use --append /share/apps/environmentmodules/centos7/general:module use --append /share/apps/environmentmodules/centos7/libraries:module use --append /usr/local/Modules/modulefiles

xdelaruelle commented 9 months ago

The provided debug output shows a correct execution.

The issue lies in the environment initialization made by qlogin. Either:

your environment is cloned by qlogin but some variables are withdrawn
a new environment initialization is performed but it does not execute the Modules initialization scripts

Provided logs do not help to analyze the issue. To go further, please provide:

the debug output of qlogin command and relative environment initialization
once connected after qlogin, the output of the module config --dump-state command

BBR-U22 commented 9 months ago

Yes, it is something with module initialisation. Oddly that some of the nodes has __MODULES_LMINIT variable correctly set (but configuration of the nodes is completely identical).

Unfortunately I see no debug options for qlogin (according to man pages and we can not set it globally as it is production system)

LMINIT not set:

$ qlogin node_with_bug
$ env | grep -i modules
MANPATH=/usr/local/Modules/share/man
MODULEPATH_modshare=/share/apps/environmentmodules/centos7/libraries:1:/share/apps/environmentmodules/centos7/general:1:/usr/local/Modules/modulefiles:1:/share/apps/environmentmodules/centos7/devtools:1
MODULES_CMD=/usr/local/Modules/libexec/modulecmd.tcl
ENV=/usr/local/Modules/init/profile.sh
PATH=/usr/local/Modules/bin:/tmp/3109174.1.all.q:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/share/admin/bin:/share/admin/toolsbin
MODULEPATH=/share/apps/environmentmodules/centos7/devtools:/share/apps/environmentmodules/centos7/general:/share/apps/environmentmodules/centos7/libraries:/usr/local/Modules/modulefiles
LOADEDMODULES=
__MODULES_SHARE_MANPATH=:1
BASH_ENV=/usr/local/Modules/init/bash
MODULESHOME=/usr/local/Modules
SGE_O_PATH=/usr/local/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/share/admin/bin:/share/admin/toolsbin
 if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
BASH_FUNC__module_raw()=() {  eval "$(/usr/bin/tclsh '/usr/local/Modules/libexec/modulecmd.tcl' bash "$@")";

$ module config --dump-state

Modules Release 5.3.0 (2023-05-14)

- Config. name ---------.- Value (set by if default overridden) ---------------
advanced_version_spec     1
auto_handling             1
avail_indepth             1
avail_output              modulepath:alias:dirwsym:sym:tag:variantifspec:key
avail_terse_output        modulepath:alias:dirwsym:sym:tag:variantifspec
cache_buffer_bytes        32768
cache_expiry_secs         0
collection_pin_tag        0
collection_pin_version    0
collection_target         <undef>
color                     auto
colors                    hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44:kL=30;48;5;109
contact                   root@localhost
csh_limit                 4000
editor                    vi
extended_default          1
extra_siteconfig          <undef>
home                      /usr/local/Modules (env-var)
icase                     search
ignore_cache              0
ignore_user_rc            0
ignored_dirs              CVS RCS SCCS .svn .git .SYNC .sos
implicit_default          1
implicit_requirement      1
list_output               header:idx:variant:sym:tag:key
list_terse_output         header
locked_configs            
mcookie_check             always
mcookie_version_check     1
ml                        1
nearly_forbidden_days     14
pager                     /usr/bin/less -eFKRX
protected_envvars         <undef>
quarantine_support        0
rcfile                    <undef>
redirect_output           1
reset_target_state        __init__
run_quarantine            <undef>
search_match              starts_with
set_shell_startup         0
shells_with_ksh_fpath     
silent_shell_debug        0
siteconfig                /usr/local/Modules/etc/siteconfig.tcl
tag_abbrev                auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS:keep-loaded=kL
tag_color_name            
tcl_ext_lib               /usr/local/Modules/lib/libtclenvmodules.so
tcl_linter                nagelfar.tcl
term_background           dark
term_width                0
unload_match_order        returnlast
variant_shortcut          
verbosity                 normal
wa_277                    1

- State name -----------.- Value ----------------------------------------------
WARNING: couldn't find name for group id "20540": Success
always_read_full_file     1
autoinit                  0
clock_seconds             1695805955
cmdline                   /usr/local/Modules/libexec/modulecmd.tcl bash config --dump-state
commandname               config
domainname                (none)
error_count               0
extra_siteconfig_loaded   0
false_rendered            0
force                     0
hiding_threshold          0
inhibit_errreport         0
inhibit_interp            0
init_error_report         1
is_stderr_tty             1
is_win                    0
kernelversion             #1 SMP Wed Jan 25 16:41:43 UTC 2023
lm_info_cached            0
machine                   x86_64
modulefile                {}
nodename                  node
os                        Linux
osversion                 3.10.0-1160.83.1.el7.x86_64
paginate                  1
path_separator            :
report_format             regular
reportfd                  file7
return_false              0
shell                     bash
shelltype                 sh
siteconfig_loaded         1
sub1_separator            &
sub2_separator            |
subcmd                    config
subcmd_args               --dump-state
supported_shells          sh bash ksh zsh csh tcsh fish cmd tcl perl python ruby lisp cmake r
tcl_ext_lib_loaded        1
tcl_version               8.5.13
term_columns              213
timer                     0
usergroups                <undef>
username                  user

- Env. variable --------.- Value ----------------------------------------------
BASH_FUNC__module_raw()   () {  eval "$(/usr/bin/tclsh '/usr/local/Modules/libexec/modulecmd.tcl' bash "$@")";
 _mlstatus=$?;
 return $_mlstatus
}
BASH_FUNC_module()        () {  local _mlredir=1;
 if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
 _mlredir=0;
 else
 if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
 _mlredir=1;
 fi;
 fi;
 fi;
 case " $@ " in 
 *' --no-redirect '*)
 _mlredir=0
 ;;
 *' --redirect '*)
 _mlredir=1
 ;;
 esac;
 if [ $_mlredir -eq 0 ]; then
 _module_raw "$@";
 else
 _module_raw "$@" 2>&1;
 fi
}
LOADEDMODULES             
MODULEPATH                /share/apps/environmentmodules/centos7/devtools:/share/apps/environmentmodules/centos7/general:/share/apps/environmentmodules/centos7/libraries:/usr/local/Modules/modulefiles
MODULEPATH_modshare       /share/apps/environmentmodules/centos7/libraries:1:/share/apps/environmentmodules/centos7/general:1:/usr/local/Modules/modulefiles:1:/share/apps/environmentmodules/centos7/devtools:1
MODULESHOME               /usr/local/Modules
MODULES_CMD               /usr/local/Modules/libexec/modulecmd.tcl
__MODULES_SHARE_MANPATH   :1

LMINIT is set:

$ qlogin node_without_bug
$ env | grep -i modules
MANPATH=/usr/local/Modules/share/man
__MODULES_LMINIT=module use --append /share/apps/environmentmodules/centos7/devtools:module use --append /share/apps/environmentmodules/centos7/general:module use --append /share/apps/environmentmodules/centos7/libraries:module use --append /usr/local/Modules/modulefiles
MODULES_CMD=/usr/local/Modules/libexec/modulecmd.tcl
PATH=/usr/local/Modules/bin:/tmp/3109182.1.all.q:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/share/admin/bin:/share/admin/toolsbin
MODULEPATH=/share/apps/environmentmodules/centos7/devtools:/share/apps/environmentmodules/centos7/general:/share/apps/environmentmodules/centos7/libraries:/usr/local/Modules/modulefiles
LOADEDMODULES=
__MODULES_SHARE_MANPATH=:1
MODULESHOME=/usr/local/Modules
SGE_O_PATH=/usr/local/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/share/admin/bin:/share/admin/toolsbin
 if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
BASH_FUNC__module_raw()=() {  eval "$(/usr/bin/tclsh '/usr/local/Modules/libexec/modulecmd.tcl' bash "$@")";

$ module config --dump-state

Modules Release 5.3.0 (2023-05-14)

- Config. name ---------.- Value (set by if default overridden) ---------------
advanced_version_spec     1
auto_handling             1
avail_indepth             1
avail_output              modulepath:alias:dirwsym:sym:tag:variantifspec:key
avail_terse_output        modulepath:alias:dirwsym:sym:tag:variantifspec
cache_buffer_bytes        32768
cache_expiry_secs         0
collection_pin_tag        0
collection_pin_version    0
collection_target         <undef>
color                     auto
colors                    hi=1:db=2:tr=2:se=2:er=91:wa=93:me=95:in=94:mp=1;94:di=94:al=96:va=93:sy=95:de=4:cm=92:aL=100:L=90;47:H=2:F=41:nF=43:S=46:sS=44:kL=30;48;5;109
contact                   root@localhost
csh_limit                 4000
editor                    vi
extended_default          1
extra_siteconfig          <undef>
home                      /usr/local/Modules (env-var)
icase                     search
ignore_cache              0
ignore_user_rc            0
ignored_dirs              CVS RCS SCCS .svn .git .SYNC .sos
implicit_default          1
implicit_requirement      1
list_output               header:idx:variant:sym:tag:key
list_terse_output         header
locked_configs            
mcookie_check             always
mcookie_version_check     1
ml                        1
nearly_forbidden_days     14
pager                     /usr/bin/less -eFKRX
protected_envvars         <undef>
quarantine_support        0
rcfile                    <undef>
redirect_output           1
reset_target_state        __init__
run_quarantine            <undef>
search_match              starts_with
set_shell_startup         0
shells_with_ksh_fpath     
silent_shell_debug        0
siteconfig                /usr/local/Modules/etc/siteconfig.tcl
tag_abbrev                auto-loaded=aL:loaded=L:hidden=H:hidden-loaded=H:forbidden=F:nearly-forbidden=nF:sticky=S:super-sticky=sS:keep-loaded=kL
tag_color_name            
tcl_ext_lib               /usr/local/Modules/lib/libtclenvmodules.so
tcl_linter                nagelfar.tcl
term_background           dark
term_width                0
unload_match_order        returnlast
variant_shortcut          
verbosity                 normal
wa_277                    1

- State name -----------.- Value ----------------------------------------------
WARNING: couldn't find name for group id "20497": Success
always_read_full_file     1
autoinit                  0
clock_seconds             1695806887
cmdline                   /usr/local/Modules/libexec/modulecmd.tcl bash config --dump-state
commandname               config
domainname                (none)
error_count               0
extra_siteconfig_loaded   0
false_rendered            0
force                     0
hiding_threshold          0
inhibit_errreport         0
inhibit_interp            0
init_error_report         1
is_stderr_tty             1
is_win                    0
kernelversion             #1 SMP Wed Jan 25 16:41:43 UTC 2023
lm_info_cached            0
machine                   x86_64
modulefile                {}
nodename                  node
os                        Linux
osversion                 3.10.0-1160.83.1.el7.x86_64
paginate                  1
path_separator            :
report_format             regular
reportfd                  file7
return_false              0
shell                     bash
shelltype                 sh
siteconfig_loaded         1
sub1_separator            &
sub2_separator            |
subcmd                    config
subcmd_args               --dump-state
supported_shells          sh bash ksh zsh csh tcsh fish cmd tcl perl python ruby lisp cmake r
tcl_ext_lib_loaded        1
tcl_version               8.5.13
term_columns              213
timer                     0
usergroups                <undef>
username                  user

- Env. variable --------.- Value ----------------------------------------------
BASH_FUNC__module_raw()   () {  eval "$(/usr/bin/tclsh '/usr/local/Modules/libexec/modulecmd.tcl' bash "$@")";
 _mlstatus=$?;
 return $_mlstatus
}
BASH_FUNC_module()        () {  local _mlredir=1;
 if [ -n "${MODULES_REDIRECT_OUTPUT+x}" ]; then
 if [ "$MODULES_REDIRECT_OUTPUT" = '0' ]; then
 _mlredir=0;
 else
 if [ "$MODULES_REDIRECT_OUTPUT" = '1' ]; then
 _mlredir=1;
 fi;
 fi;
 fi;
 case " $@ " in 
 *' --no-redirect '*)
 _mlredir=0
 ;;
 *' --redirect '*)
 _mlredir=1
 ;;
 esac;
 if [ $_mlredir -eq 0 ]; then
 _module_raw "$@";
 else
 _module_raw "$@" 2>&1;
 fi
}
LOADEDMODULES             
MODULEPATH                /share/apps/environmentmodules/centos7/devtools:/share/apps/environmentmodules/centos7/general:/share/apps/environmentmodules/centos7/libraries:/usr/local/Modules/modulefiles
MODULESHOME               /usr/local/Modules
MODULES_CMD               /usr/local/Modules/libexec/modulecmd.tcl
__MODULES_LMINIT          module use --append /share/apps/environmentmodules/centos7/devtools:module use --append /share/apps/environmentmodules/centos7/general:module use --append /share/apps/environmentmodules/centos7/libraries:module use --append /usr/local/Modules/modulefiles
__MODULES_SHARE_MANPATH   :1

xdelaruelle commented 9 months ago

Thanks for these inputs.

node_with_bug is configured differently than node_without_bug. ENV and BASH_ENV variables are defined on node_with_bug. Which means set_shell_startup option is enabled on this node, whereas it is disabled on node_without_bug.

There is another awkward difference that I spot: MODULEPATH_modshare is defined on node_with_bug but such variable is only defined on Modules v4 (not v5).

So I would suggest to check the differences between node_with_bug and node_without_bug filesystems:

compare files in /usr/local/Modules (especially files in init and etc directories)
also check if there is no double module installation on node_with_bug

cea-hpc / modules