Add timers to steps as they run in a suite or task

[xylar]

This merge adds timers to steps as they run. This is sometimes convenient for long-running steps.

It also adds a display for the duration of the test case, step or suite in hours (important for long tests).

Checklist

• [x] Testing comment in the PR documents testing used to verify the changes

[xylar]

Old output

ocean/baroclinic_channel/10km/threads
  * step: init
  * step: 1thread
  * step: 2thread
  task execution:      SUCCESS
  task validation:     PASS
  baseline comparison: PASS
  task runtime:        00:07
ocean/baroclinic_channel/10km/decomp
  * step: init
  * step: 4proc
  * step: 8proc
  task execution:      SUCCESS
  task validation:     PASS
  baseline comparison: PASS
  task runtime:        00:04
ocean/baroclinic_channel/10km/restart
  * step: init
  * step: full_run
  * step: restart_run
  task execution:      SUCCESS
  task validation:     PASS
  baseline comparison: PASS
  task runtime:        00:04
ocean/inertial_gravity_wave/convergence
  * step: init_200km
  * step: forward_200km
  * step: init_100km
  * step: forward_100km
  * step: init_50km
  * step: forward_50km
  * step: init_25km
  * step: forward_25km
  * step: analysis
  task execution:      SUCCESS
  baseline comparison: PASS
  task runtime:        01:03
ocean/single_column/960km/cvmix
  * step: init
  * step: forward
  * step: viz
  task execution:      SUCCESS
  baseline comparison: PASS
  task runtime:        00:03
ocean/single_column/960km/ideal_age
  * step: init
  * step: forward
  * step: viz
  task execution:      SUCCESS
  baseline comparison: PASS
  task runtime:        00:03
Task Runtimes:
00:07 PASS ocean/baroclinic_channel/10km/threads
00:04 PASS ocean/baroclinic_channel/10km/decomp
00:04 PASS ocean/baroclinic_channel/10km/restart
01:03 PASS ocean/inertial_gravity_wave/convergence
00:03 PASS ocean/single_column/960km/cvmix
00:03 PASS ocean/single_column/960km/ideal_age
Total runtime 01:24
PASS: All passed successfully!

[xylar]

New output

ocean/baroclinic_channel/10km/threads
  * step: init
          runtime:     0:00:03
  * step: 1thread
          runtime:     0:00:02
  * step: 2thread
          runtime:     0:00:01
  test execution:      SUCCESS
  test validation:     PASS
  baseline comparison: PASS
  test runtime:        0:00:06
ocean/baroclinic_channel/10km/decomp
  * step: init
          runtime:     0:00:01
  * step: 4proc
          runtime:     0:00:01
  * step: 8proc
          runtime:     0:00:01
  test execution:      SUCCESS
  test validation:     PASS
  baseline comparison: PASS
  test runtime:        0:00:04
ocean/baroclinic_channel/10km/restart
  * step: init
          runtime:     0:00:01
  * step: full_run
          runtime:     0:00:01
  * step: restart_run
          runtime:     0:00:01
  test execution:      SUCCESS
  test validation:     PASS
  baseline comparison: PASS
  test runtime:        0:00:04
ocean/inertial_gravity_wave/convergence
  * step: init_200km
          runtime:     0:00:00
  * step: forward_200km
          runtime:     0:00:02
  * step: init_100km
          runtime:     0:00:01
  * step: forward_100km
          runtime:     0:00:07
  * step: init_50km
          runtime:     0:00:02
  * step: forward_50km
          runtime:     0:00:16
  * step: init_25km
          runtime:     0:00:09
  * step: forward_25km
          runtime:     0:00:22
  * step: analysis
          runtime:     0:00:00
  test execution:      SUCCESS
  baseline comparison: PASS
  test runtime:        0:01:02
ocean/single_column/960km/cvmix
  * step: init
          runtime:     0:00:01
  * step: forward
          runtime:     0:00:01
  * step: viz
          runtime:     0:00:00
  test execution:      SUCCESS
  baseline comparison: PASS
  test runtime:        0:00:03
ocean/single_column/960km/ideal_age
  * step: init
          runtime:     0:00:01
  * step: forward
          runtime:     0:00:01
  * step: viz
          runtime:     0:00:00
  test execution:      SUCCESS
  baseline comparison: PASS
  test runtime:        0:00:03
Test Runtimes:
0:00:06 PASS ocean/baroclinic_channel/10km/threads
0:00:04 PASS ocean/baroclinic_channel/10km/decomp
0:00:04 PASS ocean/baroclinic_channel/10km/restart
0:01:02 PASS ocean/inertial_gravity_wave/convergence
0:00:03 PASS ocean/single_column/960km/cvmix
0:00:03 PASS ocean/single_column/960km/ideal_age
Total runtime: 0:01:22
PASS: All passed successfully!

[xylar]

Testing

I ran the pr suite with these changes, see above.

[xylar]

@cbegeman, this should be an easier one. I'm just looking for a thumbs up or some quick suggestions of how to format things in a way you find more intuitive.

[xylar]

Thanks so much, @cbegeman!