Exact restart tests are failing

ekluzek commented 1 year ago

The following Exact restart tests are failing in comparing the restart history files to the simulation of a continuous run.

ERS.f09_f09_mg17.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default   COMPARE_base_rest         
ERS.f19_f19_mg17.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default     COMPARE_base_rest         
ERS.f19_f19_mg17.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default   COMPARE_base_rest         
ERS.f19_f19_rHDMAlk_mg17.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default   COMPARE_base_rest         
ERS.nldas2_nldas2_rHDMA_mnldas2.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default      COMPARE_base_rest         
ERS.nldas2_nldas2_rHDMA_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default    COMPARE_base_rest         
ERS.nldas2_nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_gnu.mizuroute-default      COMPARE_base_rest         
ERS.nldas2_nldas2_rUSGS_mnldas2.I2000Clm50SpMizGs.cheyenne_intel.mizuroute-default    COMPARE_base_rest

nmizukami commented 1 year ago

I did some SMS tests for the global domain cases. so far I found the following cases failed (or expected to fail) due to netCDF format. Not sure why this happened (I believe the MERIT-Hydro mesh used to be in cdf5 so rMERIT case used to run, but it because netCDF 4 recently??)

MERIT-Hydro square mesh is in netCDF4 and get error in ESMF reading of this file /glade/p/cesmdata/cseg/inputdata/rof/mizuRoute/meshes/polygon_centroid/MERITmz_global_ctrcrd_sqr_cdf5_ESMFmesh_c20210113.nc

This causes error for

f09_f09_rMERIT (fail confirmed)
f19_f19_rMERIT (expected to fail)
hcru_hcru_rMERIT (expected to fail, also hcru_hcru mesh is netcdf4)

hcur_hcru mesh is in netCDF4 get the error in ESMF reading of this file. /glade/p/cesmdata/cseg/inputdata/share/meshes/360x720_120830_ESMFmesh_c20210507.nc

This causes an error for

hcru_hcru_rHDMA (fail confirmed)
hcru_hcru_rHDMAlk (expected to fail)
hcru_hcru_rMERIT (expected to fail)

I am thinking that we need to make sure all the SMS tests pass before doing ERS tests (so the case for which SMS test fails also fail in ERS test)?

ekluzek commented 1 year ago

Yes, we should make sure a base SMS test runs before trying to fix the ERS test for it. But, SMS tests that do run and ERS tests that fail for just the COMPARE_base_rest part of it are good ones to work on.

ESCOMP / mizuRoute

Exact restart tests are failing #377