search

ESMCI / ccs_config_cesm

CESM CIME Case Control System configuration files
3 stars 44 forks source link

mpi-serial cases on cheyenne having trouble running in ccs_config_cesm0.0.24 #38

Closed ekluzek closed 8 months ago

ekluzek commented 2 years ago

I have several mpi-serial cases that break on cheyenne after updating to ccs_config_cesm0.0.24 and after. I did a git bisect to trace it to that exact change.

An example test that fails is:

SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB_nuopc

Also SMS_Ly1_Mmpi-serial.1x1_brazil.IHistClm50BgcQianRs.cheyenne_intel.clm-output_bgc_highfreq

All I get in the cesm.log is:

Completion(send) value=1 tag=1 Completion(send) value=1 tag=2 Completion(send) value=1 tag=3 Completion(send) value=1 tag=4 Completion(send) value=0 tag=1 Completion(send) value=0 tag=2 Completion(send) value=0 tag=3 Completion(send) value=0 tag=4

The drv.log and med.log show that it's in the middle of initialization. There is a lnd.log and atm.log

There is one mpi-serial test that is in place for CESM alpha testing that looks like it's passing: SMS_D_Ld1_Mmpi-serial.f45_f45_mg37.I2000Clm50SpRs.cheyenne_intel.clm-ptsRLA

There are several changes that come together in this version, updating ESMF, mpt, and pio altogether. I'm going to see if I can separate that out.

jedwards4b commented 2 years ago

@fischer-ncar The mpi-serial version of ESMF was not built with pio.

20220507 064500.961 ERROR PET0 /glade/p/cesmdata/cseg/PROGS/build/24043/esmf-8.3.0b13/src/Infrastructure/Mesh/src/ESMCI_Mesh_FileIO.C:297 ESMCI_mesh_create_from_file() Library needed by ESMF not present - This functionality requires ESMF to be built with the PIO library enabled.

jedwards4b commented 2 years ago

I also found an issue in the ESMF library, we will need a new ESMF build.

jedwards4b commented 2 years ago

Fix in ESMF branch jedwards/FieldReadFix

fischer-ncar commented 2 years ago

I did rebuild the intel mpi-serial version over the weekend us esmf8.3.0b13, and the tests passed.

jedwards4b commented 2 years ago

@fischer-ncar Thanks! Hopefully we'll have a new ESMF tag by the time cheyenne is back online.

ekluzek commented 8 months ago

This obviously was fixed. And with Cheyenne going away doesn't matter.