search

ESCOMP / CTSM

Community Terrestrial Systems Model (includes the Community Land Model of CESM)
http://www.cesm.ucar.edu/models/cesm2.0/land/
Other
308 stars 312 forks source link

Short clm ERS test hits reasonable wall clock limit on izumi nag #1520

Open glemieux opened 3 years ago

glemieux commented 3 years ago

Brief summary of bug

ERS_D_Ld5.f19_g17.I2000Clm50BgcCru.izumi_nag.clm-default will not complete its run due to hitting the wall clock limit.

General bug information

CTSM version you are using: ctsm5.1.dev057-59-g282920abd

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected:

Details of bug

In the course of updating PR #1275 to provide nag coverage on izumi for all debug tests in the fates category, I included the one non-fates testmod as part of the list via https://github.com/ESCOMP/CTSM/pull/1275/commits/5f3a8228a35caf6b340706caaf1e43804e763294. Although this test is short, it never completes. Only the first set of logs are available and are not compressed. Examining the logs it appears that either the clm or mosart files might be getting hung up during one of the final writes, although all the expected output files do appear to be present when comparing the list to a successful run of the same test on cheyenne.

For the time being this test has been dropped from the fates category on izumi for the nag compiler since we have coverage elsewhere for a different production compiler.

ekluzek commented 3 years ago

It seems to me that the primary reason we had non-FATES tests in the fates testlist was to make sure that non-FATES tests continue to work fine. However, now normally the fates test list is run in conjunction with the aux_clm tests so maybe we don't need non-FATES tests in the fates test list anymore?

glemieux commented 3 years ago

I think it makes sense to still keep a clm-only test in the fates list for when we're testing things that don't require an api update, that way we catch potential issues that a fates-only update might screw up something in ctsm. Our current procedure is to only run the aux_clm tests when we have an api update iirc.

That said, I think having coverage for clm-only in the fates suite is probably only necessary for production compilers (i.e. not nag) since aux_clm on izumi covers that.

billsacks commented 3 years ago

I have only skimmed this, but a couple of notes: