wx20jjung
commented
2 months ago @DavidHuber-NOAA I have been struggling with this. I do not know what the true intent of nread is. Was it developed for 2D (horizontal) observations or 3D data like the radiances? I interpret the code as being for the former. My argument is based on the allocate statement for nloc and icrit a few lines down. Both of these variables use itxmax, which is the total counts of a 2D spatial (horizontal) grid. To be consistent, I feel ncounts1 (and nread) should also be from a 2D spatial grid and not multiplied by the number of channels.
An alternative would be to add another argument to combine_radobs specifically for ncounts and ncounts1, which would all be long integers from and to the various read routines.
DavidHuber-NOAA
RussTreadon-NOAA
Description
When the GSI is built in debug mode, the code failed when the read_iasi routine called combine_radobs. In tracking down this problem several other minor problems were discovered and fixed.
Resolves #789
The first group of commits are some write format changes I missed in the output of the runtime directory. I also found where the maximum channel number was set to 3000. I changed this array to be dynamic and allocated for each instrument.
The second group of commits resolves 2 errors I found while trying to fix the debug issue. The first was an error some of the IASI temperature values were "NAN"s. I traced this back to some of the cscale values (a scaling factor for IASI radiances, in the BUFR file), were missing values. The missing cscale values are associated with the shortwave side of the water vapor region and the shortwave channels. None of these channels are currently used. These were only found in the direct broadcast data. I did not find an instance where the operational data had missing values. The second error was an integer overflow problem with the variable nread being passed into combine radobs. Nread is used in most satellite instrument reads but the main failure was from read_iasing (to be added later). Nread is a counter and is ultimately number_of_profiles number_of_channels. This number exceeded the memory for an integer and was ultimately a negative number. In combineradobs, nread is compared to the total number of elements of the thinning box or itxmax to the number of elements of the task thinning box. In this case, the task thinning box was negative. I added logic to the various read routines to pass the number of profiles kept on each task. This makes nread and itxmax consistent. This change caused the total counts to be different for some instruments. Here is an example from the gsistat file for IASI. previous: o-g 01 rad metop-b iasi 331026696 6148957 690460 0.24889E+06 0.24889E+06 0.36047 0.36047 new: o-g 01 rad metop-b iasi 44632896 6148957 690460 0.24889E+06 0.24889E+06 0.36047 0.36047 The total counts after thinning and total counts used are identical along with the other statistics.
The actual failure of read_iasi in debug mode came down to putting an if() cycle in a different place. Read_iasi is now consistent with read_cris.
Type of change
How Has This Been Tested?
The main testing and cycling experiments were conducted on S4 at C192 resolution. After a single cycle spinup I setup a "control" and "experiment" and run both independently changing the GSIEXEC variable to point to the gsi master branch (control) and the IASI_debug_fix branch (experiment). The control and experiment were run through 4 cycle. Static tests were also conducted on Jet.
The ctests on Hera all passed: [Jim.Jung@hfe10 build]$ ctest -j 6 Test project /scratch1/NCEPDEV/jcsda/Jim.Jung/save/ctests/update/build Start 1: global_4denvar Start 6: global_enkf Start 2: rtma Start 3: rrfs_3denvar_rdasens Start 4: hafs_4denvar_glbens Start 5: hafs_3denvar_hybens 1/6 Test #3: rrfs_3denvar_rdasens ............. Passed 492.94 sec 2/6 Test #2: rtma ............................. Passed 970.72 sec 3/6 Test #5: hafs_3denvar_hybens .............. Passed 1043.42 sec 4/6 Test #6: global_enkf ...................... Passed 1118.55 sec 5/6 Test #4: hafs_4denvar_glbens .............. Passed 1159.69 sec 6/6 Test #1: global_4denvar ................... Passed 1909.17 sec
100% tests passed, 0 tests failed out of 6
Total Test time (real) = 1909.21 sec
The rrfs ctest timed out on jet, all others passed. Test project /lfs5/HFIP/hfv3gfs/Jim.Jung/noscrub/ctests/update/build Start 1: global_4denvar Start 2: rtma Start 3: rrfs_3denvar_rdasens Start 4: hafs_4denvar_glbens Start 5: hafs_3denvar_hybens Start 6: global_enkf 1/6 Test #6: global_enkf ...................... Passed 1779.50 sec 2/6 Test #2: rtma ............................. Passed 1938.22 sec 3/6 Test #1: global_4denvar ................... Passed 1991.97 sec 4/6 Test #5: hafs_3denvar_hybens .............. Passed 2000.38 sec 5/6 Test #4: hafs_4denvar_glbens .............. Passed 2310.42 sec
The rrfs test also failed on hercules [jjung@hercules-login-3 build]$ ctest Test project /work/noaa/nesdis-rdo1/jjung/noscrub/ctests/update/build Start 1: global_4denvar 1/6 Test #1: global_4denvar ................... Passed 1741.09 sec Start 2: rtma 2/6 Test #2: rtma ............................. Passed 965.23 sec Start 3: rrfs_3denvar_rdasens 3/6 Test #3: rrfs_3denvar_rdasens .............***Failed 484.58 sec Start 4: hafs_4denvar_glbens 4/6 Test #4: hafs_4denvar_glbens .............. Passed 1161.19 sec Start 5: hafs_3denvar_hybens 5/6 Test #5: hafs_3denvar_hybens .............. Passed 1094.42 sec Start 6: global_enkf 6/6 Test #6: global_enkf ...................... Passed 847.14 sec
83% tests passed, 1 tests failed out of 6
Total Test time (real) = 6293.65 sec
The following tests FAILED: 3 - rrfs_3denvar_rdasens (Failed) The memory for rrfs_3denvar_rdasens_loproc_updat is 1106724 KBs. This has exceeded maximum allowable memory of 1098631 KBs, resulting in Failure memthresh of the regression test.
Checklist
Please add @ADCollard, @DavidHuber-NOAA and @InnocentSouopgui-NOAA as reviewers.