test_forward_SOI_v.abi_g18 fails the reference check

The test_forward_SOI_v.abi_g18 test fails when in the new tagged jedi bundle, but passes if crtm is built separately. Both builds are using ecbuild with CMAKE_BUILD_TYPE=Debug. This is with gfortran 11.3.1. Compilation occurs on an Intel Xeon Platinum 9242 CPU. The host OS is Rocky Linux 9.0.

The test outputs differ subtly, and there are only trivial differences in how ecbuild set flags between the two builds. I haven't been able to spot any OpenMP issues with the Intel Inspector tool, which is expected since I'm not seeing any OpenMP blocks around the SOI modules. I suspect a compiler bug somewhere.

ecbuild outputs:

crtm only: crtm_ecbuild.txt
crtm as part of the jedi bundle: jedi_ecbuild.txt

It may look odd, but f95 is a direct symlink to gfortran on this system. No idea why ecbuild prefers one over the other. Ecbuild gives essentially identical outputs for both builds.

Differences:

crtm:

-- [crtm] (3.0.0) [74cdc94]
-- Feature TESTS enabled

......

-- Fortran -- GNU 11.3.1
--     compiler   : /usr/bin/f95
--     flags      :  -D_REAL8_ -ffree-line-length-none -O0 -g -fcheck=bounds -ffpe-trap=invalid,zero,overflow -fbacktrace
--     link flags : -fopenmp

jedi-bundle:

-- ---------------------------------------------------------
-- Adding bundle project crtm
-- ---------------------------------------------------------
-- [crtm] (3.0.0) [74cdc94]
-- Feature TESTS enabled
-- Found OpenMP: TRUE (found version "4.5") found components: Fortran

......

-- Fortran -- GNU 11.3.1
--     compiler   : /usr/bin/gfortran
--     flags      :  -O0 -g -fcheck=bounds -fbacktrace -finit-real=snan
--     link flags :

Although the link flags differ, when I look further down at the build logs, they are consistent. This may be a display bug in ecbuild.

Build flags:

Both builds use identical flags when compiling test_forward_test_SOI. They use the same backend libraries. Both use -fopenmp. Both set -D_REAL8_.

crtm only build:

[ 86%] Linking Fortran executable test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E remove /mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/test/test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E cmake_link_script CMakeFiles/test_forward_test_SOI.dir/link.txt --verbose=1
/usr/bin/f95 -fopenmp     -Wl,--disable-new-dtags -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -Wl,-rpath -Wl,/usr/lib64 -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -L/usr/lib64  -D_REAL8_ -ffree-line-length-none -O0 -g -fcheck=bounds -ffpe-trap=invalid,zero,overflow -fbacktrace CMakeFiles/test_forward_test_SOI.dir/mains/regression/forward/test_SOI/test_SOI.f90.o -o test_forward_test_SOI  -Wl,-rpath,/mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib ../lib/libcrtm.so /usr/lib/gcc/x86_64-redhat-linux/11/libgomp.so /usr/lib64/libpthread.a /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib/libnetcdff.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib -lnetcdff -lnetcdf -lnetcdf -lm /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib/libnetcdf.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib -lnetcdf /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempif08.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempi_ignore_tkr.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_mpifh.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi.so 
make[2]: Leaving directory '/mnt/ram/builds/skylab-5.0.0/crtm/build/gcc'

jedi bundle build:

[ 80%] Linking Fortran executable test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/build/gcc/crtm/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E remove /mnt/ram/builds/skylab-5.0.0/build/gcc/crtm/test/test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/build/gcc/crtm/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E cmake_link_script CMakeFiles/test_forward_test_SOI.dir/link.txt --verbose=1
/usr/bin/gfortran -fopenmp     -Wl,--disable-new-dtags -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -Wl,-rpath -Wl,/usr/lib64 -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -L/usr/lib64  -D_REAL8_ -ffree-line-length-none -O0 -g -fcheck=bounds -ffpe-trap=invalid,zero,overflow -fbacktrace CMakeFiles/test_forward_test_SOI.dir/mains/regression/forward/test_SOI/test_SOI.f90.o -o test_forward_test_SOI  -Wl,-rpath,/mnt/ram/builds/skylab-5.0.0/build/gcc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib ../../lib/libcrtm.so /usr/lib/gcc/x86_64-redhat-linux/11/libgomp.so /usr/lib64/libpthread.a /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib/libnetcdff.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib -lnetcdff -lnetcdf -lnetcdf -lm /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib/libnetcdf.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib -lnetcdf /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempif08.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempi_ignore_tkr.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_mpifh.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi.so 
make[2]: Leaving directory '/mnt/ram/builds/skylab-5.0.0/build/gcc'
[ 80%] Built target test_forward_test_SOI

Test logs:

crtm only: log_nojedi_crtm.txt
crtm as part of the jedi bundle: log_jedi_crtm_test.txt

JCSDA / CRTMv3