how to load user modules?

[aekiss]

Is there a way to load custom user modules?

I found this in the code but don't know how to use it https://github.com/payu-org/payu/blob/a83d60b09f8ff93432cde302d36904edc4b79f16/payu/experiment.py#L256-L259

I ask because we need to load some custom modules via

module use /g/data/v45/aek156/spack/share/spack/modules/linux-rocky8-cascadelake
module load netcdf-c-4.9.0-intel-2021.6.0-t5vmcra
module load parallel-netcdf-1.12.3-intel-2021.6.0-xty6sc5
module load xerces-c-3.2.3-intel-2021.6.0-z57iwgl

to get the ACCESS-OM3 executable to run. I tried doing this via a setup userscript but it didn't work - see https://github.com/COSIMA/access-om3/issues/17#issuecomment-1519344108

[aekiss]

I've also tried wrapping the executable in a shell script and using the wrapper in config.yaml but this fails because payu can't execute ldd on the wrapper script (not a dynamic executable). See https://github.com/COSIMA/access-om3/issues/17#issuecomment-1504320703

[angus-g]

I think you can do it just by adding something like the following to your config.yaml (at the root level):

modules:
  - netcdf-c-4.9.0-intel-2021.6.0-t5vmcra
  - parallel-netcdf-1.12.3-intel-2021.6.0-xty6sc5
  - xerces-c-3.2.3-intel-2021.6.0-z57iwgl

The module use is handled by the MODULEPATH environment variable, so if you have run this before payu, it'll get exported into the job, i.e.

$ module use /g/data/v45/aek156/spack/share/spack/modules/linux-rocky8-cascadelake
$ payu run                                                                                                          
Loading input manifest: manifests/input.yaml                                                                        
Loading restart manifest: manifests/restart.yaml                                                                    
Loading exe manifest: manifests/exe.yaml                                                                            
payu: Found modules in /opt/Modules/v4.3.0                                                                          
qsub <snip> -v ...,MODULEPATH=/g/data/v45/aek156/spack/share/spack/modules/linux-rocky8-cascadelake:...:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles <snip> -W umask=027 -l storage=gdata/x77+scratch/x77 -- payu-run

[aekiss]

Awesome, thanks @angus-g, that seems to have done the trick! Now it loads the modules

$ cat MOM6-CICE6-WW3.e81828398
Currently Loaded Modulefiles:
 1) openmpi/4.1.4(default)
 2) pbs
 3) netcdf-c-4.9.0-intel-2021.6.0-t5vmcra
 4) parallel-netcdf-1.12.3-intel-2021.6.0-xty6sc5
 5) xerces-c-3.2.3-intel-2021.6.0-z57iwgl
payu: Model exited with error code 233; aborting.

(it still crashes, but with a different error than before that seems unrelated to library loading)

[aekiss]

Is modules documented anywhere? I feel like it should be here https://payu.readthedocs.io/en/latest/config.html#miscellaneous

[angus-g]

Not that I'm aware of, but I agree it should be documented if it exists. I guess all the automatic module discovery stuff was meant to supersede it, though there are always edge cases!

[aekiss]

ok I've had a go at documenting it - see PR https://github.com/payu-org/payu/pull/338