E3SM-Project / polaris

Testing and analysis for OMEGA, MPAS-Ocean, MALI and MPAS-Seaice
BSD 3-Clause "New" or "Revised" License
6 stars 13 forks source link

Raise error only at task level on baseline failure #128

Closed xylar closed 1 year ago

xylar commented 1 year ago

If baseline comparison fails, raise an error at the task level not at the step level so task execution can complete.

Checklist

xylar commented 1 year ago

Testing

I ran the pr suite, pointing at a baseline that didn't actually exist. I see:

ocean/planar/baroclinic_channel/10km/threads
  * step: init
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:05
  * step: 1thread
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:02
  * step: 2thread
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: validate
          execution:        SUCCESS
          runtime:          0:00:00
  task execution:   SUCCESS
  baseline comp.:   FAIL
  task runtime:     0:00:09
ocean/planar/baroclinic_channel/10km/decomp
  * step: init
          already completed
  * step: 4proc
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: 8proc
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: validate
          execution:        SUCCESS
          runtime:          0:00:00
  task execution:   SUCCESS
  baseline comp.:   FAIL
  task runtime:     0:00:03
ocean/planar/baroclinic_channel/10km/restart
  * step: init
          already completed
  * step: full_run
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: restart_run
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: validate
          execution:        SUCCESS
          runtime:          0:00:00
  task execution:   SUCCESS
  baseline comp.:   FAIL
  task runtime:     0:00:03
ocean/planar/inertial_gravity_wave
  * step: init_200km
          execution:        SUCCESS
          runtime:          0:00:01
  * step: forward_200km
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:02
  * step: init_100km
          execution:        SUCCESS
          runtime:          0:00:01
  * step: forward_100km
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:07
  * step: init_50km
          execution:        SUCCESS
          runtime:          0:00:02
  * step: forward_50km
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:16
  * step: init_25km
          execution:        SUCCESS
          runtime:          0:00:07
  * step: forward_25km
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:23
  * step: analysis
          execution:        SUCCESS
          runtime:          0:00:00
  task execution:   SUCCESS
  baseline comp.:   FAIL
  task runtime:     0:01:00
ocean/single_column/cvmix
  * step: init
          execution:        SUCCESS
          runtime:          0:00:01
  * step: forward
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: viz
          execution:        SUCCESS
          runtime:          0:00:00
  task execution:   SUCCESS
  baseline comp.:   FAIL
  task runtime:     0:00:02
ocean/single_column/ideal_age
  * step: init
          execution:        SUCCESS
          runtime:          0:00:01
  * step: forward
          execution:        SUCCESS
          baseline comp.:   FAIL
          runtime:          0:00:01
  * step: viz
          execution:        SUCCESS
          runtime:          0:00:00
  task execution:   SUCCESS
  baseline comp.:   FAIL
  task runtime:     0:00:02
Task Runtimes:
0:00:09 FAIL ocean/planar/baroclinic_channel/10km/threads
0:00:03 FAIL ocean/planar/baroclinic_channel/10km/decomp
0:00:03 FAIL ocean/planar/baroclinic_channel/10km/restart
0:01:00 FAIL ocean/planar/inertial_gravity_wave
0:00:02 FAIL ocean/single_column/cvmix
0:00:02 FAIL ocean/single_column/ideal_age
Total runtime: 0:01:19
FAIL: 6 tasks failed, see above.

This seems like the expected behavior.

I compared the results with another run of this branch and to all 6 tests passing (including the baseline comparison).

xylar commented 1 year ago

@cbegeman, this is ready to review.

xylar commented 1 year ago

@cbegeman, thanks for having a look and trying it out!