RussTreadon-NOAA
commented
3 months ago While test_gdasapp_atm_jjob_var_inc runs to completion on Orion, it fails on Hera. Below is the Hera traceback
0: Info : Registry status: 0
2: corrupted size vs. prev_size
5: corrupted size vs. prev_size
0:
0: Run: Finishing gdasapp::fv3inc
0: OOPS_STATS
...
0: OOPS_STATS util::Timers::measured : 16002.43 1 16002.4301
0: OOPS_STATS ------------------------------------ Timing Statistics -------------------------------------
srun: error: h20c45: task 2: Aborted (core dumped)
srun: Terminating StepId=58208170.0
0: slurmstepd: error: *** STEP 58208170.0 ON h20c45 CANCELLED AT 2024-04-13T10:59:40 ***
srun: error: h20c45: tasks 0,3-4: Terminated
srun: error: h20c45: task 5: Aborted (core dumped)
srun: error: h20c45: task 1: TerminatedThe printout
2: corrupted size vs. prev_size
5: corrupted size vs. prev_sizeis not in the Orion job log file.
I reran the job on Hera with --mem=0 to request all memory on the Hera node. test_gdasapp_atm_jjob_var_inc still failed in the same way.
test_gdasapp_fv3jedi_fv3inc also failed on Hera despite feature/update_jjob_tests including GDASApp PR #1039. The Hera traceback for test_gdasapp_fv3jedi_fv3inc contains the following
Test : FV3 Increment:
Test : ----------------------------------------------------------------------------------------------------
Test : Increment print | number of fields = 9 | cube sphere face size: C12
Test : eastward_wind | Min:-4.3422836419308410e+00 Max:+1.2320940067737499e+01 RMS:+3.0957235443709130e-01
Test : northward_wind | Min:-4.1090470888107049e+00 Max:+5.4552721209750796e+00 RMS:+3.1062842460180656e-01
Test : air_temperature | Min:-5.2980343087781989e-01 Max:+5.1811022097894011e-01 RMS:+3.5920751813835923e-02
Test : specific_humidity | Min:-2.8092260972819617e-04 Max:+2.9434075393080551e-04 RMS:+1.6532405760382759e-05
Test : cloud_liquid_ice | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
Test : cloud_liquid_water | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
Test : ozone_mass_mixing_ratio | Min:+0.0000000000000000e+00 Max:+0.0000000000000000e+00 RMS:+0.0000000000000000e+00
Test : air_pressure_thickness | Min:-2.9992886080290191e+00 Max:+1.5291703492039233e+00 RMS:+1.7535872214547940e-01
Test : hydrostatic_layer_thickness | Min:-4.6699236754648155e-01 Max:+7.4693987323735200e-01 RMS:+3.1162055487823255e-02
Test : ----------------------------------------------------------------------------------------------------
double free or corruption (!prev)
srun: error: h32m52: task 5: Aborted (core dumped)
srun: Terminating StepId=58208119.0
slurmstepd: error: *** STEP 58208119.0 ON h32m52 CANCELLED AT 2024-04-13T10:51:05 ***
srun: error: h32m52: tasks 0,2-4: Terminated
srun: error: h32m52: task 1: Terminated
srun: Force Terminated StepId=58208119.0Not sure what's going on.
DavidNew-NOAA
This PR addresses issue #1011, related to the failure of the "gdasatmanlvar" jjob test due to a change in the name of the "gdasatmanlvar" run script, and, with the upcoming Global Workflow PR #2420, the impending failure of the "gdasatmanlfinal" jjob. This fixes the script name in the "gdasatmanlvar" test and adds a new test for "gdasatmanlfv3inc" which will fix the issure with "gdasatmanlfinal".
The "gdasatmanlfv3inc" and "gdasatmanlfinal" won't pass yes in GW develop, but will after #2420 merges GW feature/jediinc2fv3.
This PR is just a verbatim copy of @RussTreadon-NOAA 's work, taken from his comment here. I re-ran the new tests, and they also passed for me in GW feature/jediinc2fv3.