search

ACCESS-NRI / dev_coupling

0 stars 0 forks source link

CIME on Gadi using batch submission: mpirun command not found #2

Closed dougiesquire closed 1 year ago

dougiesquire commented 1 year ago

Running CIME cases on Gadi seems to work fine when running on the login node.

However, when CIME is configured to run cases as batch jobs, the jobs fail intermittently with:

/bin/sh: mpirun: command not found

What's very strange is that sometimes mpirun is available and sometimes it isn't.

dougiesquire commented 1 year ago

When running ./case.submit --no-batch (which works), a number of calls are made which load the environment (and provide mpirun):

  1. CIME/case/case_submit.py LINE 166: case.check_case(skip_pnl=skip_pnl, chksum=chksum)
  2. CIME/case/case_submit.py LINE 174: case.load_env()
  3. CIME/case/case_run.py LINE 99: _pre_run_check(case, lid, skip_pnl=skip_pnl, da_cycle=da_cycle) which also calls case.load_env

Instances 1 and 2 above are only called in --no-batch mode and are not crucial to the environment being available at model runtime (I can comment out these lines and still run successfully). However, commenting out 3 results in the mpirun: command not found error even when running with --non-batch.

dougiesquire commented 1 year ago

Inside _pre_run_check, this boils down further to the line

CIME/case/case_run.py LINE 49: case.load_env(reset=True)

which successfully loads the environment in --no-batch but sometimes not in batch mode. Why?

dougiesquire commented 1 year ago

Within case.load_env, the line

CIME/case/case.py LINE 2071: self._loaded_envs = env_module.load_env(self, job=job, verbose=verbose)

loads the environment

dougiesquire commented 1 year ago

CIME loads modules on Gadi using CIME.XML.env_mach_specific._load_module_modules. Within this function, a module command (cmd) is run using subprocess and the output (py_module_code) is executed.

During a successful (e.g. no batch) run, cmd is "/opt/Modules/v4.3.0/bin/modulecmd python load openmpi intel-compiler intel-mkl netcdf pnetcdf python3-as-python": which is run successfully (stat=0, errout is empty), and py_module_code is:

import os
os.environ['CPATH'] = '/apps/pnetcdf/1.11.2/include:/apps/netcdf/4.7.3/include:/apps/intel-ct/2022.1.0/mkl/include:/apps/intel-ct/2021.6.0/compiler/linux/include:/apps/openmpi/4.1.4/include'
os.environ['CMPLR_ROOT'] = '/apps/intel-ct/2021.6.0/compiler'
os.environ['CMAKE_PREFIX_PATH'] = '/apps/intel-ct/2021.6.0/compiler/linux/IntelDPCPP'
os.environ['OPENMPI_ROOT'] = '/apps/openmpi/4.1.4'
os.environ['I_MPI_F90'] = 'ifort'
os.environ['VT_CXX'] = 'icpc'
os.environ['MODULES_LMALTNAME_modshare'] = 'openmpi/4.1.4&openmpi/default&openmpi:1:netcdf/4.7.3&netcdf/default&netcdf:1'
os.environ['NETCDF_BASE'] = '/apps/netcdf/4.7.3'
os.environ['NCI_PYTHONX_AS_PYTHON'] = 'python3'
os.environ['OMPI_MCA_orte_tmpdir_base'] = '/tmp'
os.environ['PNETCDF_VERSION'] = '1.11.2'
os.environ['C_INCLUDE_PATH'] = '/apps/pnetcdf/1.11.2/include:/apps/netcdf/4.7.3/include:/apps/intel-ct/2022.1.0/mkl/include:/apps/intel-ct/2021.6.0/compiler/linux/include:/apps/openmpi/4.1.4/include'
os.environ['HDF5_CC'] = 'icc'
os.environ['CMAKE_PREFIX_PATH_modshare'] = '/apps/intel-ct/2021.6.0/compiler/linux/IntelDPCPP:1'
os.environ['F77'] = 'ifort'
os.environ['INTEL_COMPILER_BASE'] = '/apps/intel-ct/2021.6.0/compiler'
os.environ['I_MPI_CXX'] = 'icpc'
os.environ['OMPI_ROOT'] = '/apps/openmpi/4.1.4'
os.environ['MANPATH'] = '/apps/netcdf/4.7.3/share/man:/apps/intel-ct/2021.6.0/compiler/documentation/en/man/common:/apps/openmpi/4.1.4/share/man:/home/599/ds0092/.linuxbrew/share/man:/home/599/ds0092/.linuxbrew/share/man:/opt/pbs/default/share/man::::'
os.environ['VT_FC'] = 'ifort'
os.environ['OMPI_CC'] = 'icc'
os.environ['PKG_CONFIG_PATH'] = '/apps/netcdf/4.7.3/lib/pkgconfig:/apps/intel-ct/2022.1.0/mkl/lib/pkgconfig:/half-root/usr/lib64/pkgconfig'
os.environ['MANPATH_modshare'] = '/apps/netcdf/4.7.3/share/man:1:/apps/intel-ct/2021.6.0/compiler/documentation/en/man/common:1::1:/home/599/ds0092/.linuxbrew/share/man:1:/apps/openmpi/4.1.4/share/man:1:/opt/pbs/default/share/man:2'
os.environ['C_INCLUDE_PATH_modshare'] = '/apps/intel-ct/2021.6.0/compiler/linux/include:1:/apps/pnetcdf/1.11.2/include:1:/apps/openmpi/4.1.4/include:1:/apps/netcdf/4.7.3/include:1:/apps/intel-ct/2022.1.0/mkl/include:1'
os.environ['PNETCDF_BASE'] = '/apps/pnetcdf/1.11.2'
os.environ['F90'] = 'ifort'
os.environ['INTEL_MKL_ROOT'] = '/apps/intel-ct/2022.1.0/mkl'
os.environ['LD_LIBRARY_PATH_modshare'] = '/apps/pnetcdf/1.11.2/lib:1:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:1:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:1:/apps/openmpi/4.1.4/lib:1:/apps/netcdf/4.7.3/lib:1:/apps/openmpi/4.1.4/lib/profilers:1:/apps/intel-ct/2022.1.0/mkl/lib/intel64:1:/apps/intel-ct/2021.6.0/compiler/linux/lib:1'
os.environ['OPENMPI_BASE'] = '/apps/openmpi/4.1.4'
os.environ['HDF4_CLINKER'] = 'icc'
os.environ['PATH'] = '/apps/pnetcdf/1.11.2/bin:/apps/netcdf/4.7.3/bin:/apps/intel-ct/wrapper:/apps/intel-ct/2021.6.0/compiler/linux/bin/intel64:/apps/intel-ct/2021.6.0/compiler/linux/bin:/apps/openmpi/wrapper/fortran:/apps/openmpi/wrapper:/apps/openmpi/4.1.4/bin:~/.linuxbrew/bin:/g/data/xv83/users/ds0092/software/:/g/data/xv83/users/ds0092/software/oh-my-posh:/g/data/xv83/users/ds0092/software/fre-nctools_regrid:/g/data/xv83/users/ds0092/software/miniconda3/envs/xmovie/bin/:/g/data/v14/common/bin:/home/599/ds0092/.vscode-server/bin/d045a5eda657f4d7b676dedbfa7aab8207f8a075/bin/remote-cli:/home/599/ds0092/.linuxbrew/bin:/home/599/ds0092/.linuxbrew/sbin:/home/599/ds0092/.linuxbrew/bin:/g/data/xv83/users/ds0092/software:/g/data/xv83/users/ds0092/software/oh-my-posh:/g/data/xv83/users/ds0092/software/fre-nctools_regrid:/g/data/xv83/users/ds0092/software/miniconda3/envs/xmovie/bin:/g/data/v14/common/bin:/home/599/ds0092/.local/bin:/home/599/ds0092/bin:/home/599/ds0092/.linuxbrew/bin:/home/599/ds0092/.linuxbrew/sbin:/g/data/xv83/users/ds0092/software/miniconda3/bin:/g/data/xv83/users/ds0092/software/miniconda3/condabin:/home/599/ds0092/.linuxbrew/bin:/g/data/xv83/users/ds0092/software:/g/data/xv83/users/ds0092/software/oh-my-posh:/g/data/xv83/users/ds0092/software/fre-nctools_regrid:/g/data/xv83/users/ds0092/software/miniconda3/envs/xmovie/bin:/g/data/v14/common/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin'
os.environ['HDF5_CLINKER'] = 'icc'
os.environ['CC'] = 'icc'
os.environ['VT_PFORM_LDIR'] = '/scratch/tm70/ds0092/tmp'
os.environ['OMPI_F77'] = 'ifort'
os.environ['CPLUS_INCLUDE_PATH_modshare'] = '/apps/intel-ct/2021.6.0/compiler/linux/include:1:/apps/pnetcdf/1.11.2/include:1:/apps/openmpi/4.1.4/include:1:/apps/netcdf/4.7.3/include:1:/apps/intel-ct/2022.1.0/mkl/include:1'
os.environ['LD_LIBRARY_PATH'] = '/apps/pnetcdf/1.11.2/lib:/apps/netcdf/4.7.3/lib:/apps/intel-ct/2022.1.0/mkl/lib/intel64:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:/apps/intel-ct/2021.6.0/compiler/linux/lib:/apps/openmpi/4.1.4/lib:/apps/openmpi/4.1.4/lib/profilers'
os.environ['LOADEDMODULES'] = 'pbs:openmpi/4.1.4:intel-compiler/2021.6.0:intel-mkl/2022.1.0:netcdf/4.7.3:pnetcdf/1.11.2:python3-as-python'
os.environ['I_MPI_CC'] = 'icc'
os.environ['FPATH'] = '/apps/pnetcdf/1.11.2/include:/apps/netcdf/4.7.3/include:/apps/intel-ct/2022.1.0/mkl/include:/apps/intel-ct/2021.6.0/compiler/linux/include:/apps/openmpi/4.1.4/include:/opt/Modules/v4.3.0/init/ksh-functions'
os.environ['MODULES_LMCONFLICT'] = 'openmpi/4.1.4&mpi&lam&mpich&openmpi&intel-mpi&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:intel-compiler/2021.6.0&intel-compiler-llvm&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:intel-mkl/2022.1.0&intel-mkl&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:netcdf/4.7.3&netcdf&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:pnetcdf/1.11.2&pnetcdf&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:python3-as-python&python2-as-python'
os.environ['CXX'] = 'icpc'
os.environ['PKG_CONFIG_PATH_modshare'] = '/apps/netcdf/4.7.3/lib/pkgconfig:1:/apps/intel-ct/2022.1.0/mkl/lib/pkgconfig:1:/half-root/usr/lib64/pkgconfig:1'
os.environ['OMPI_BASE'] = '/apps/openmpi/4.1.4'
os.environ['MKLROOT'] = '/apps/intel-ct/2022.1.0/mkl'
os.environ['NLSPATH'] = '/apps/intel-ct/2022.1.0/mkl/lib/intel64/locale/%l_%t/%N'
os.environ['LD_RUN_PATH_modshare'] = '/apps/pnetcdf/1.11.2/lib:1:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:1:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:1:/apps/openmpi/4.1.4/lib:1:/apps/netcdf/4.7.3/lib:1:/apps/openmpi/4.1.4/lib/profilers:1:/apps/intel-ct/2022.1.0/mkl/lib/intel64:1:/apps/intel-ct/2021.6.0/compiler/linux/lib:1'
os.environ['OCL_ICD_FILENAMES_modshare'] = '/apps/intel-ct/2021.6.0/compiler/linux/lib/x64/libintelocl.so:1'
os.environ['INTEL_MKL_VERSION'] = '2022.1.0'
os.environ['LOADEDMODULES_modshare'] = 'openmpi/4.1.4:1:pnetcdf/1.11.2:1:python3-as-python:1:pbs:1:netcdf/4.7.3:1:intel-compiler/2021.6.0:1:intel-mkl/2022.1.0:1'
os.environ['VT_F77'] = 'ifort'
os.environ['OMPI_FC'] = 'ifort'
os.environ['OPENMPI_VERSION'] = '4.1.4'
os.environ['NETCDF_ROOT'] = '/apps/netcdf/4.7.3'
os.environ['HCOLL_ENABLE_MCAST'] = '0'
os.environ['VT_MAX_FLUSHES'] = '0'
os.environ['LD_RUN_PATH'] = '/apps/pnetcdf/1.11.2/lib:/apps/netcdf/4.7.3/lib:/apps/intel-ct/2022.1.0/mkl/lib/intel64:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:/apps/intel-ct/2021.6.0/compiler/linux/lib:/apps/openmpi/4.1.4/lib:/apps/openmpi/4.1.4/lib/profilers'
os.environ['OCL_ICD_FILENAMES'] = '/apps/intel-ct/2021.6.0/compiler/linux/lib/x64/libintelocl.so'
os.environ['INFOPATH'] = '/apps/netcdf/4.7.3/share/info:/home/599/ds0092/.linuxbrew/share/info:/home/599/ds0092/.linuxbrew/share/info:'
os.environ['INTEL_MKL_BASE'] = '/apps/intel-ct/2022.1.0/mkl'
os.environ['LIBRARY_PATH'] = '/apps/pnetcdf/1.11.2/lib:/apps/netcdf/4.7.3/lib:/apps/intel-ct/2022.1.0/mkl/lib/intel64:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:/apps/intel-ct/2021.6.0/compiler/linux/lib:/apps/openmpi/4.1.4/lib:/apps/openmpi/4.1.4/lib/profilers'
os.environ['DIAGUTIL_PATH_modshare'] = '/apps/intel-ct/2021.6.0/compiler/sys_check/sys_check.sh:1'
os.environ['OMPI_VERSION'] = '4.1.4'
os.environ['_LMFILES_'] = '/opt/Modules/modulefiles/pbs:/apps/Modules/modulefiles/openmpi/4.1.4:/apps/Modules/modulefiles/intel-compiler/2021.6.0:/apps/Modules/modulefiles/intel-mkl/2022.1.0:/apps/Modules/modulefiles/netcdf/4.7.3:/apps/Modules/modulefiles/pnetcdf/1.11.2:/apps/Modules/modulefiles/python3-as-python'
os.environ['INFOPATH_modshare'] = '/apps/netcdf/4.7.3/share/info:1::1:/home/599/ds0092/.linuxbrew/share/info:1'
os.environ['INTEL_COMPILER_VERSION'] = '2021.6.0'
os.environ['HDF4_CC'] = 'icc'
os.environ['LIBRARY_PATH_modshare'] = '/apps/pnetcdf/1.11.2/lib:1:/apps/intel-ct/2021.6.0/compiler/linux/compiler/lib/intel64_lin:1:/apps/intel-ct/2021.6.0/compiler/linux/lib/x64:1:/apps/openmpi/4.1.4/lib:1:/apps/netcdf/4.7.3/lib:1:/apps/openmpi/4.1.4/lib/profilers:1:/apps/intel-ct/2022.1.0/mkl/lib/intel64:1:/apps/intel-ct/2021.6.0/compiler/linux/lib:1'
os.environ['UCC_LOG_LEVEL'] = 'ERROR'
os.environ['MODULES_LMALTNAME'] = 'openmpi/4.1.4&openmpi/default&openmpi:netcdf/4.7.3&netcdf/default&netcdf'
os.environ['NETCDF'] = '/apps/netcdf/4.7.3'
os.environ['NLSPATH_modshare'] = '/apps/intel-ct/2022.1.0/mkl/lib/intel64/locale/%l_%t/%N:1'
os.environ['FPATH_modshare'] = '/apps/intel-ct/2021.6.0/compiler/linux/include:1:/opt/Modules/v4.3.0/init/ksh-functions:1:/apps/pnetcdf/1.11.2/include:1:/apps/openmpi/4.1.4/include:1:/apps/netcdf/4.7.3/include:1:/apps/intel-ct/2022.1.0/mkl/include:1'
os.environ['MODULES_LMCONFLICT_modshare'] = 'openmpi/4.1.4&mpi&lam&mpich&openmpi&intel-mpi&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1:intel-mkl/2022.1.0&intel-mkl&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1:pnetcdf/1.11.2&pnetcdf&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1:intel-compiler/2021.6.0&intel-compiler-llvm&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1:netcdf/4.7.3&netcdf&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1:python3-as-python&python2-as-python:1'
os.environ['CPLUS_INCLUDE_PATH'] = '/apps/pnetcdf/1.11.2/include:/apps/netcdf/4.7.3/include:/apps/intel-ct/2022.1.0/mkl/include:/apps/intel-ct/2021.6.0/compiler/linux/include:/apps/openmpi/4.1.4/include'
os.environ['VT_CC'] = 'icc'
os.environ['OMPI_CXX'] = 'icpc'
os.environ['VT_F90'] = 'ifort'
os.environ['NETCDF_VERSION'] = '4.7.3'
os.environ['INTEL_COMPILER_ROOT'] = '/apps/intel-ct/2021.6.0/compiler'
os.environ['I_MPI_F77'] = 'ifort'
os.environ['FC'] = 'ifort'
os.environ['CPATH_modshare'] = '/apps/intel-ct/2021.6.0/compiler/linux/include:1:/apps/pnetcdf/1.11.2/include:1:/apps/openmpi/4.1.4/include:1:/apps/netcdf/4.7.3/include:1:/apps/intel-ct/2022.1.0/mkl/include:1'
os.environ['_LMFILES__modshare'] = '/apps/Modules/modulefiles/intel-compiler/2021.6.0:1:/apps/Modules/modulefiles/intel-mkl/2022.1.0:1:/opt/Modules/modulefiles/pbs:1:/apps/Modules/modulefiles/openmpi/4.1.4:1:/apps/Modules/modulefiles/pnetcdf/1.11.2:1:/apps/Modules/modulefiles/python3-as-python:1:/apps/Modules/modulefiles/netcdf/4.7.3:1'
os.environ['UCC_TLS'] = 'nccl,ucp'
os.environ['PNETCDF_ROOT'] = '/apps/pnetcdf/1.11.2'
os.environ['PATH_modshare'] = '/usr/bin:1:/g/data/xv83/users/ds0092/software/miniconda3/envs/xmovie/bin:1:/g/data/v14/common/bin:1:/apps/intel-ct/2021.6.0/compiler/linux/bin:1:/opt/Modules/v4.3.0/bin:1:/opt/bin:1:/apps/intel-ct/wrapper:1:/g/data/xv83/users/ds0092/software:1:/g/data/xv83/users/ds0092/software/miniconda3/condabin:1:/home/599/ds0092/.linuxbrew/bin:1:/apps/intel-ct/2021.6.0/compiler/linux/bin/intel64:1:/home/599/ds0092/.linuxbrew/sbin:1:/home/599/ds0092/.local/bin:2:/home/599/ds0092/bin:2:/g/data/xv83/users/ds0092/software/miniconda3/envs/xmovie/bin/:1:/bin:1:/home/599/ds0092/.vscode-server/bin/d045a5eda657f4d7b676dedbfa7aab8207f8a075/bin/remote-cli:1:/g/data/xv83/users/ds0092/software/fre-nctools_regrid:1:/g/data/xv83/users/ds0092/software/:1:/apps/openmpi/wrapper:1:/opt/pbs/default/bin:2:/g/data/xv83/users/ds0092/software/oh-my-posh:1:~/.linuxbrew/bin:1:/opt/nci/bin:1:/usr/sbin:1:/apps/openmpi/wrapper/fortran:1:/apps/pnetcdf/1.11.2/bin:1:/apps/openmpi/4.1.4/bin:1:/usr/local/sbin:1:/apps/netcdf/4.7.3/bin:1:/g/data/xv83/users/ds0092/software/miniconda3/bin:1'
os.environ['DIAGUTIL_PATH'] = '/apps/intel-ct/2021.6.0/compiler/sys_check/sys_check.sh'
_mlstatus = True

During an unsuccessful batch run, cmd is: "/opt/Modules/v4.3.0/bin/modulecmd python load openmpi intel-compiler intel-mkl netcdf pnetcdf python3-as-python" which is run sucessfully (stat=0, errout is empty), but py_module_code is empty.

dougiesquire commented 1 year ago

Adding a module purge command to ccs_config_cesm/machines/config_machines.xml appears to have fixed this issue - https://github.com/ACCESS-NRI/ccs_config_cesm/commit/9dd011379064789ca0c4053947b60dfb5fda93d4