search

ESMCI / cime

Common Infrastructure for Modeling the Earth
http://esmci.github.io/cime
Other
162 stars 209 forks source link

Fix BFAILs, they need to be considered DIFF #4678

Closed jgfouca closed 2 months ago

jgfouca commented 2 months ago

Having BFAILs resolve to overall FAIL prevent bless_test_results and other tools from working properly. For E3SM, a BFAIL has always been considered a DIFF.

Test suite: by-hand Test baseline: Test namelist changes: Test status: bit for bit

Fixes [CIME Github issue #]

User interface changes?:

Update gh-pages html (Y/N)?:

ekluzek commented 2 months ago

Since, this changes behavior in how testing works for us in CESM, I'm highlighting this. I have found it important and helpful to have a way to distinguish between baselines that truly fail and those that the baseline doesn't exist. I'm not saying I can't live with the change, but need to think about it. I'll bring it to the attention of others in CESM.

jgfouca commented 2 months ago

@ekluzek , this is how CIME has always worked until about a month ago. Are you guys (CESM) already dependent on the new behavior? We (E3SM) were dependent on the old behavior.

ekluzek commented 2 months ago

@jgfouca thanks for pointing that out.

Hmmm. OK. I didn't realize this was going back to previous behavior. In that case it's probably fine. I actually wouldn't even have been using versions of cime with the new behavior.

What PR brought in the change in behavior? And you are saying this PR, just reverts part of that PR for labelling test results?

jgfouca commented 2 months ago

@ekluzek , this PR changed the behavior: https://github.com/ESMCI/cime/pull/4653

I did my best to keep the spirit of that PR but restoring the BFAIL behavior.

cacraigucar commented 2 months ago

The CAM folks would like the old implementation restored (this PR). We have not updated our cime external to the one where the change was introduced. I'm not sure if it would break our workflow or not, so we'll just wait for this PR to be brought in before we update cime next.