search

ESMCI / cime

Common Infrastructure for Modeling the Earth
http://esmci.github.io/cime
Other
161 stars 206 forks source link

Fix/memcomp #4599

Open jedwards4b opened 6 months ago

jedwards4b commented 6 months ago

Currently if a test is too short for the memcomp test it fails because the test baseline is missing. It should just skip the test in this case...

This PR causes the test to be skipped and pass, but I would like it to be more informative.

Test suite: Test baseline: Test namelist changes: Test status: bit for bit Fixes: User interface changes?:

Update gh-pages html (Y/N)?:

jedwards4b commented 6 months ago

I'll look into refactoring this as a change in cs.status ...

jedwards4b commented 6 months ago

Actually that won't solve the problem. People are still going to manually look at the TestStatus file and see that the Test indicates fail.

rljacob commented 6 months ago

Do we know about how long the run has to be for a memtest to be valid? Just skip all the mem testing if the test length is to short. Force it on with a command-line argument.

rljacob commented 6 months ago

Or...only run the memtest portion if that file is present. If something about the machine keeps the file from being created, like disk issues, then other things will also go wrong.

ekluzek commented 6 months ago

@jedwards4b as I say above I'm OK with it marked as fail if it says it's because of missing baseline files. Personally I then do some grepping to make sure the FAIL's are only because of that kind of problem.

But, I do like @rljacob idea of not doing these tests for cases that are too short. Marking them as such would also be fine as well.

All -- I know we can change the thresholds for these tests, but is there a way to signal in the testlist*.xml file that you want them turned off? For example for tests that are too short or you just don't want to check it for particular tests?

Still just glad this is being worked on! Thanks @jedwards4b

jedwards4b commented 6 months ago

I thought that this was going to be an easy fix - but it's not, I'm going to set it aside for now.