search

E3SM-Project / scorpio

A high-level Parallel I/O Library for structured grid applications
19 stars 16 forks source link

mpas-framework linking errors with updated Fortran interface of SCORPIO #543

Closed dqwu closed 9 months ago

dqwu commented 10 months ago

Steps to reproduce on ANL GCE nodes:

git clone https://github.com/E3SM-Project/E3SM.git
cd E3SM

git submodule update --init --recursive

cd externals/scorpio
git checkout scorpio-v1.5.0

cd ../../cime/scripts
./create_test SMS.T62_oQU120_ais20.MPAS_LISIO_TEST --no-run

Build errors:

/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_text':
mpas_io.f90:(.text+0x349f): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_real1d':
mpas_io.f90:(.text+0x3bb5): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: mpas_io.f90:(.text+0x3bfa): undefined reference to `__spio_inq_att_MOD_pio_inq_attlen_file_varid_attname_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_real0d':
mpas_io.f90:(.text+0x4634): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: mpas_io.f90:(.text+0x469e): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_int1d':
mpas_io.f90:(.text+0x4d1e): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: mpas_io.f90:(.text+0x4d6b): undefined reference to `__spio_inq_att_MOD_pio_inq_attlen_file_varid_attname_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_int0d':
mpas_io.f90:(.text+0x549e): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'

Not reproducible with default scorpio 1.4.1 release. Seems to be related to recent mpas-framework code changes.

jgfouca commented 10 months ago

Is there something missing in the steps like changing PIO_TYPE? I don't even make it to the MODEL_BUILD phase with that version of scorpio:

/home/jgfouca/E3SM-clean/share/util/shr_pio_mod.F90:2:6:

   use pio
      1
Fatal Error: Can't open module file ‘pio.mod’ for reading at (1): No such file or directory
compilation terminated.
dqwu commented 10 months ago

@jgfouca Maybe that is another build issue only reproducible on mappy. For now, could you please try Chrysalis or ANL GCE nodes?

jayeshkrishna commented 10 months ago

Did the order of linking change in CIME for mpas libraries (specifically ../../mpas-framework/src/libcommon.a ) ?

jayeshkrishna commented 10 months ago

Maybe the source of the issue is https://github.com/E3SM-Project/E3SM/pull/5923/files#diff-75365ff294e10b3b929daccf29d4d9640e5a963542d83b7c9710a6ce8feb2408L33 ?

jayeshkrishna commented 10 months ago

So the issue seems to be the linking order for libcommon.a , if its moved (from the end of the link order - the last line in e3sm build log) before libpiof.a the linking issues go away,

python3 /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/Tools/e3sm_compile_wrap.py  /nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/mpich/4.0/gcc-11.1.0/bin/mpif90 CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/cime_comp_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/cime_driver.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/component_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/component_type_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/cplcomp_exchange_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/map_glc2lnd_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/map_lnd2glc_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/map_lnd2rof_irrig_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/mrg_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_aoflux_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_atm_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_glc_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_iac_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_ice_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_lnd_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_ocn_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_rof_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_wav_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_diagBGC_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_diag_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_domain_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_flux_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_frac_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_hist_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_io_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_map_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_map_type_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_nlmap_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_rest_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/t_driver_timers_mod.F90.o -o /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/e3sm.exe  ../atm/libatm.a ../lnd/liblnd.a ../../mpas-framework/src/libice.a ../../mpas-framework/src/libocn.a ../rof/librof.a ../../mpas-framework/src/libglc.a ../wav/libwav.a ../iac/libiac.a ../esp/libesp.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/mct/noesmf/c1a1l1i1o1r1g1w1i1e1/lib/libcsm_share.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libmct.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libmpeu.a ../../mpas-framework/src/libcommon.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libpiof.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libpioc.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libgptl.a -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/netcdf/4.8.0c-4.3.1cxx-4.5.3f-parallel/mpich-4.0/gcc-11.1.0/lib -lnetcdff -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/netcdf/4.8.0c-4.3.1cxx-4.5.3f-parallel/mpich-4.0/gcc-11.1.0/lib -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/hdf5/1.12.1/mpich-4.0/gcc-11.1.0/lib -lnetcdf -lhdf5_hl -lhdf5 -lm -ldl -lz -lcurl -lblas -llapack -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/adios2/2.8.3.patch/mpich-4.0/gcc-11.1.0/lib -ladios2_c_mpi -ladios2_c -ladios2_core_mpi -ladios2_core -ladios2_evpath -ladios2_ffs -ladios2_dill -ladios2_atl -ladios2_enet -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/pnetcdf/1.12.2/mpich-4.0/gcc-11.1.0/lib -lpnetcdf -lstdc++ /usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libblas.so /usr/lib/x86_64-linux-gnu/libf77blas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libblas.so /usr/lib/x86_64-linux-gnu/libf77blas.so /usr/lib/x86_64-linux-gnu/libatlas.so

The original linking error (on anlgce) is in e3sm.bldlog.*,

...
[100%] Linking Fortran executable /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/e3sm.exe
cd /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/cmake-bld/cmake/cpl && /nfs/gce/software/spack/opt/spack/linux-ubuntu20.04-x86_64/gcc-9.3.0/cmake-3.20.5-zyz2eld/bin/cmake -E cmake_link_script CMakeFiles/e3sm.exe.dir/link.txt --verbose=1
python3 /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/Tools/e3sm_compile_wrap.py  /nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/mpich/4.0/gcc-11.1.0/bin/mpif90 CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/cime_comp_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/cime_driver.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/component_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/component_type_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/cplcomp_exchange_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/map_glc2lnd_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/map_lnd2glc_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/map_lnd2rof_irrig_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/mrg_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_aoflux_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_atm_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_glc_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_iac_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_ice_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_lnd_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_ocn_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_rof_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/prep_wav_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_diagBGC_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_diag_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_domain_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_flux_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_frac_mct.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_hist_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_io_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_map_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_map_type_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_nlmap_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/seq_rest_mod.F90.o CMakeFiles/e3sm.exe.dir/scratch/jayesh/E3SM_master/driver-mct/main/t_driver_timers_mod.F90.o -o /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/e3sm.exe  ../atm/libatm.a ../lnd/liblnd.a ../../mpas-framework/src/libice.a ../../mpas-framework/src/libocn.a ../rof/librof.a ../../mpas-framework/src/libglc.a ../wav/libwav.a ../iac/libiac.a ../esp/libesp.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/mct/noesmf/c1a1l1i1o1r1g1w1i1e1/lib/libcsm_share.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libmct.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libmpeu.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libpiof.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libpioc.a /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/gnu/mpich/nodebug/nothreads/mct/lib/libgptl.a -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/netcdf/4.8.0c-4.3.1cxx-4.5.3f-parallel/mpich-4.0/gcc-11.1.0/lib -lnetcdff -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/netcdf/4.8.0c-4.3.1cxx-4.5.3f-parallel/mpich-4.0/gcc-11.1.0/lib -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/hdf5/1.12.1/mpich-4.0/gcc-11.1.0/lib -lnetcdf -lhdf5_hl -lhdf5 -lm -ldl -lz -lcurl -lblas -llapack -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/adios2/2.8.3.patch/mpich-4.0/gcc-11.1.0/lib -ladios2_c_mpi -ladios2_c -ladios2_core_mpi -ladios2_core -ladios2_evpath -ladios2_ffs -ladios2_dill -ladios2_atl -ladios2_enet -L/nfs/gce/projects/climate/software/linux-ubuntu20.04-x86_64/pnetcdf/1.12.2/mpich-4.0/gcc-11.1.0/lib -lpnetcdf -lstdc++ /usr/lib/x86_64-linux-gnu/liblapack.so /usr/lib/x86_64-linux-gnu/libblas.so /usr/lib/x86_64-linux-gnu/libf77blas.so /usr/lib/x86_64-linux-gnu/libatlas.so /usr/lib/x86_64-linux-gnu/libblas.so /usr/lib/x86_64-linux-gnu/libf77blas.so /usr/lib/x86_64-linux-gnu/libatlas.so ../../mpas-framework/src/libcommon.a
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_text':
mpas_io.f90:(.text+0x349f): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_real1d':
mpas_io.f90:(.text+0x3bb5): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: mpas_io.f90:(.text+0x3bfa): undefined reference to `__spio_inq_att_MOD_pio_inq_attlen_file_varid_attname_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_real0d':
mpas_io.f90:(.text+0x4634): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: mpas_io.f90:(.text+0x469e): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_int1d':
mpas_io.f90:(.text+0x4d1e): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
/usr/bin/ld: mpas_io.f90:(.text+0x4d6b): undefined reference to `__spio_inq_att_MOD_pio_inq_attlen_file_varid_attname_offlen'
/usr/bin/ld: ../../mpas-framework/src/libcommon.a(mpas_io.f90.o): in function `__mpas_io_MOD_mpas_io_get_att_int0d':
mpas_io.f90:(.text+0x549e): undefined reference to `__spio_inq_att_MOD_pio_inq_att_file_vid_offlen'
collect2: error: ld returned 1 exit status
Target /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/e3sm.exe built in 4.501805 seconds
make[2]: *** [cmake/cpl/CMakeFiles/e3sm.exe.dir/build.make:557: /scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/e3sm.exe] Error 1
make[2]: Leaving directory '/scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/cmake-bld'
make[1]: *** [CMakeFiles/Makefile2:632: cmake/cpl/CMakeFiles/e3sm.exe.dir/all] Error 2
make[1]: Leaving directory '/scratch/jayesh/e3sm/scratch/SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.anlgce_gnu.20230919_124230_7z24yi/bld/cmake-bld'
make: *** [Makefile:94: all] Error 2
Command exited with non-zero status 2
real 207.77
user 510.50
sys 100.77
jgfouca commented 10 months ago

Thanks, @jayeshkrishna . That should give me the info I need to fix. As a side note, build on chrysalis is incredibly slow. Is there a way to speed it up?

dqwu commented 10 months ago

@jgfouca I also saw very slow build on Chrysalis. Maybe you can try ANL GCE nodes or some supercomputers. PS, for your mentioned build issue (before MODEL_BUILD) on mappy, have you found the reason?

dqwu commented 9 months ago

@jayeshkrishna @jgfouca It seems that this issue has been fixed by E3SM-Project/E3SM#5964

jayeshkrishna commented 9 months ago

Fixed by https://github.com/E3SM-Project/E3SM/pull/5964