FUNIT test not working on izumi

[adrifoster]

Brief summary of bug

The FUNIT test is no longer working on izumi. This doesn't seem to be related to a code change, but rather to a change on izumi itself, as previous baselines which passed now also fail.

General bug information

Fails with these errors:

2024-04-04 13:47:39: Compiler is intel
/scratch/cluster/afoster/tests_0404-115752iz/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.GC.0404-115752iz_int/bld/env_mach_specific.xml already exists, delete to replace
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using PFUNIT_PATH: /fs/cgd/csm/tools/pFUnit/pFUnit4.7.0_izumi_Intel20.0.1_noMPI_noOpenMP
Setting NETCDF environment variable: /usr/local/netcdf-c-4.7.4-f-4.5.2-intel-cluster-20.0.1
==================================================
Running make for __command_line_test__/__command_line_test__.
==================================================
ERROR: Command: 'make -j 8' failed with error 'make: *** No targets specified and no makefile found.  Stop.' from dir '/scratch/cluster/afoster/tests_0404-115752iz/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel.GC.0404-115752iz_int/bld/__command_line_test__/__command_line_test__'

 ---------------------------------------------------
2024-04-04 13:47:39: ERROR: RUN FAIL for FUNIT
 ---------------------------------------------------

CTSM version you are using: ctsm5.1.dev175

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: FUNIT test on izumi

[ekluzek]

It looks like this has been randomly passing or failing since about this time. Several tags after this have it PASS, but some FAIL.

ctsm5.1.dev175 passes, and ctsm5.1.dev176 fail. But, then ctsm5.2.0 PASSes as well as ctsm5.2.001-ctsm5.2.005, and ctsm5.2.006 and ctsm5.2.007 failing as we see below.

/fs/cgd/csm/ccsm_baselines/ctsm5.2.001/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:PASS FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=169 /fs/cgd/csm/ccsm_baselines/ctsm5.2.002/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:PASS FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=161 /fs/cgd/csm/ccsm_baselines/ctsm5.2.003/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:PASS FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=165 /fs/cgd/csm/ccsm_baselines/ctsm5.2.004/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:PASS FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=192 /fs/cgd/csm/ccsm_baselines/ctsm5.2.005/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:PASS FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=163 /fs/cgd/csm/ccsm_baselines/ctsm5.2.006/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:FAIL FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=7 /fs/cgd/csm/ccsm_baselines/ctsm5.2.007/FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel/TestStatus:FAIL FUNITCTSM_P1x1.f10_f10_mg37.I2000Clm50Sp.izumi_intel RUN time=9

With Derecho down this becomes important again.

[samsrabin]

This seems to be due to the change in ccs_config/machines/izumi/config_machines.xml from using lang/python/3.7.0 to lang/python/3.11.5. The unit tests work (on ctsm5.2.027, at least) if you revert to the former.

I think the problem happens in the cmake command here. It looks like CMAKE_BINARY_DIR in cime/CIME/non_py/src/CMake/CIME_initial_setup.cmake is set to src (the directory from which the run_tests.py command is called) under 3.11 but src/unit_tests.temp/__command_line_test__/__command_line_test__ (the directory where the tests are supposed to actually happen) under 3.7. This causes include to look for src/Macros.cmake, which doesn't exist. That file is in unit_tests.temp/, unit_tests.temp/cmake_macros/, and (as a softlink) unit_tests.temp/__command_line_test__/__command_line_test__/.