ESCOMP / CTSM

Community Terrestrial Systems Model (includes the Community Land Model of CESM)
http://www.cesm.ucar.edu/models/cesm2.0/land/
Other
295 stars 298 forks source link

cnmatrix solver crashes depending on the finidat file, as well as in an IHist simulation #2592

Open slevis-lmwg opened 3 weeks ago

slevis-lmwg commented 3 weeks ago

Summary:

Details: I have encountered three examples this far (see #640):

  1. I2000 test using the original ctsm5.2 f19 finidat file:

    • I generated this finidat from the 5.1 file by running ctsm5.2 with init_interp for one step.
    • I found that the patch with the error had a crop type in the ctsm5.2 fsurdat file that was not present in the same grid cell in the ctsm5.1 fsurdat file. This indicates that init_interp did not obtain satisfactory (or any?) data for that patch.
    • I ran a full new spin-up (https://github.com/NCAR/LMWG_dev/issues/58) and the I2000 test passed, so I have replaced the default finidat for cases such as this test.
  2. NEON-NIWO single-point test with (ironically) the new ctsm5.2 f19 finidat file described in (1):

    • I reran the test with the orig. ctsm5.2 f19 finidat file and it passed.
    • The problem may be similar to (1)
    • In this case we decided to replace the failing test with a NEON-HARV test, which is a better site to support than NIWO anyway according to @wwieder. This one works with the new ctsm5.2 f19 finidat file described in (1).
  3. To spin up the model (see 1), I tried running all the phases (except AD) with cnmatrix; however, cnmatrix crashed in the historical simulation early in year 1901. The problem may be similar to (1).

Recommendations:

General bug information

CTSM version you are using: I'm using the cnmatrix branch (#640). The PR's notes show when the above issues emerged.

Configurations affected: cnmatrix

slevis-lmwg commented 3 weeks ago

If these crashes occur consistently due to newly emerging pfts:

  1. They may indicate a problem that has remained hidden until now.
  2. We may wish to introduce a more graceful error message suggesting a cold start.