mom-ocean / MOM5

The Modular Ocean Model
https://mom-ocean.github.io/
GNU Lesser General Public License v3.0
82 stars 95 forks source link

Daily fast run tests failing #372

Closed aidanheerdegen closed 1 year ago

aidanheerdegen commented 1 year ago

The daily fast run tests have started failing

https://accessdev.nci.org.au/jenkins/blue/organizations/jenkins/mom-ocean.org%2FMOM5_run/detail/MOM5_run/1198/pipeline

There have been no code changes, so it is somewhat mysterious why this would be happening now.

Specifically the reported error is:

ifort: error #10106: Fatal error in /apps/intel/compilers_and_libraries_2019.5.281/linux/bin/intel64/fortcom, terminated by kill signal
compilation aborted for /scratch/v45/aph502/scratch/jenkins/workspace/mom-ocean.org/MOM5_run/src/ocean_shared/generic_tracers/generic_TOPAZ.F90 (code 1)
make: *** [Makefile:85: generic_TOPAZ.o] Error 1
make: *** Waiting for unfinished jobs....
Make failed to create lib_ocean.a

So the compilation of src/ocean_shared/generic_tracers/generic_TOPAZ.F90 is throwing an error.

The only thing I can think of is some change in my account settings, as this is run under my account, but there is nothing I can think of in the last few days.

I have not tried to reproduce the compilation error on a login node yet.

aidanheerdegen commented 1 year ago

Cleaned out my home directory: wasn't over quota, but was close to it. Tests now working, so assume that was the root cause

https://accessdev.nci.org.au/jenkins/blue/organizations/jenkins/mom-ocean.org%2FMOM5_run/detail/MOM5_run/1199/pipeline