Release/gfsda.v16

[ADCollard]

We need to be able to switch of active assimilation of saildrone data in operational GSI before obsproc1.3 is implemented. This requires a one line code change.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [X] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

The change has been tested in a single-cycle GSI run. The saildrone observations are given the subtype 02 and are not actively assimilated.

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] New and existing tests pass with my changes
• [x] Any dependent changes have been merged and published

[ADCollard]

@RussTreadon-NOAA If possible can we produce a new GSI tag after this is merged? gfsda.v16.3.20

Suggested reviewers are @ilianagenkova and @dmerkova

[RussTreadon-NOAA]

@ilianagenkova , try the following

mkdir pr795
cd pr795
git clone -b release/gfsda.v16 --recursive https://github.com/ADCollard/GSI.git .

Two files in pr795 differ from a clone of the authoritative release/gfsda.v16. For example

russ.treadon@clogin06:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi> diff -r --exclude=".git" gfsda.v16 pr795
diff -r '--exclude=.git' gfsda.v16/fix/global_convinfo.txt pr795/fix/global_convinfo.txt
28a29
>  ps       180    02  -1     3.0      0      0      0   5.0   3.0   1.0   5.0  0.000300     0    0.     0.      0    0.     0.    8   -9
45a47
>  t        180    02  -1     3.0      0      0      0   7.0   5.6   1.3   7.0  0.004000     0    0.     0.      0    0.     0.    8   -9 
61a64
>  q        180    02  -1     3.0      0      0      0   8.0 100.0  10.0   8.0  0.000500     0    0.     0.      0    0.     0.    6   -9
201a205
>  uv       280    02  -1     3.0      0      0      0   6.0   6.1   1.4   6.0  0.005500     0    0.     0.      0    0.     0.   11   -9
diff -r '--exclude=.git' gfsda.v16/src/gsi/read_prepbufr.f90 pr795/src/gsi/read_prepbufr.f90
725a726,727
>           ! Set saildrone to subtype 02
>           if (nint(hdr(3)) == 560) iobsub = 02
russ.treadon@clogin06:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi> 

[ADCollard]

Changes look good to me. Do we have a prepbufr file containing the saildrone data to confirm that the changes work as intended?

@RussTreadon-NOAA, @ilianagenkova produced some test files here: /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3

[RussTreadon-NOAA]

Thank you @ADCollard . Has anyone run these prepbufr files through the updated code and fix file to confirm the desired behavior?

[RussTreadon-NOAA]

Troubling result

Use same rungsi script with operational GSI fix. Run two times. First run uses operational gsi.x. Second run uses gsi.x built from ADCollard:release/gfsda.v16. The job using the operational gsi.x reproduces operations. The gsi.x built from this PR produces NaN for atms_n21.

I'll recompile and rerun but an independent test would be appreciated.

[dmerkova]

@RussTreadon-NOAA can you give path to successfull run output-cnv file for wind would be ok, thank you

[RussTreadon-NOAA]

@dmerkova ,I don't have a successful run with the PR. The operational gsi.x using operational fix files reproduces operations. The gsi.x built from this PR generates NaN for atms_n21. Hopefully a recompile and rerun will not yield the same result. If it does, we have a bigger problem than to resolve.

[dmerkova]

I understand, I just want to see what happen to data if used without fix, thank you.

[RussTreadon-NOAA]

@dmerkova , I haven't gotten that far. I am not using @ilianagenkova prepbufr file. I just want to ensure that the gsi.x from this PR can reproduce operations when given operational data. Current tests indicate that it can't do this.

[ADCollard]

@RussTreadon-NOAA That is indeed troubling. I do not understand why that would happen.

My output is here: `/lfs/h2/emc/ptmp/andrew.collard/ROTDIRS/v16.3.16_hires_saildrone/gdas.20241002/00/atmos/ gdas.t00z.gsistat' and has no NaNs (apart from the pre-existing GMI issue).

[ADCollard]

... but I was running with Iliana's prepbufr. Which cycle did you run your test for?

[ADCollard]

Also do you have a standalone test script on WCOSS2? I was testing this via the workflow.

[RussTreadon-NOAA]

Recompiling did not help. gsi.x built from this PR still produces NaN for atms_n21.

Confirm that the only difference between src/gsi from this PR and operations is read_prepbufr.f90

russ.treadon@clogin06:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr795> diff -r src/gsi /lfs/h1/ops/prod/packages/gfs.v16.3.19/sorc/gsi.fd/src/gsi
diff -r src/gsi/read_prepbufr.f90 /lfs/h1/ops/prod/packages/gfs.v16.3.19/sorc/gsi.fd/src/gsi/read_prepbufr.f90
726,727d725
<           ! Set saildrone to subtype 02
<           if (nint(hdr(3)) == 560) iobsub = 02
russ.treadon@clogin06:/lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr795> 

Let me examine my run script more closely.

[RussTreadon-NOAA]

The standalone script I submit is rungsi768L127_prod_wcoss2.sh in /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/scripts. This runs 20241004 00Z gdas.

[ADCollard]

Thanks, @RussTreadon-NOAA. I will try with your script. I re-ran the 2024100200 case with the global workflow and I do not see any NaNs.

[RussTreadon-NOAA]

@ADCollard , I suspect the problem is my script. You confirmed that your changes worked from g-w. Don't waste time debugging my script. I'll create merge this PR into the authoritative release/gfsda.v16 and create a tag once we get two peer approvals.

[ADCollard]

@RussTreadon-NOAA For my own sanity I want to be sure this works (even though there is no earthly reason for this!). So I will at least see if I can reproduce the error.

[ADCollard]

@RussTreadon-NOAA OK, I see it when running your script. Let me dig a little. I cannot see how the script will produce this behavior.

[RussTreadon-NOAA]

Your executable, /lfs/h2/emc/da/noscrub/andrew.collard/git/global-workflow_v16.3.16/exec/gsi.x works with the script. Let me see how I am building gsi.x.

[RussTreadon-NOAA]

@ADCollard , looking at /lfs/h2/emc/da/noscrub/andrew.collard/git/global-workflow_v16.3.16/sorc/logs/build_gsi.log I noticed that crtm/2.4.0.1 was loaded. In contrast, release/gfsda.v16:modulefiles/gsi_common.lua loads crtm/2.4.0. I changed this to 2.4.0.1 and recompiled. The resulting gsi.x results from your executable and the operational gsi.x

Does crtm/2.4.0.1 include a change to the crtm library with respect to crtm/2.4.0?

[ADCollard]

@RussTreadon-NOAA Yes! There was a bug in the ATMS_N21 treatment in CRTM2.4.0. 2.4.0.1 was introduced to correct this. This now makes sense to me! Thanks.

[ADCollard]

@RussTreadon-NOAA So we should change the values in the module files to reflect this? I can do that now. I guess this was never tested in standalone mode and so the global-workflow configuration overrode the GSI configuration.

Thanks again!

[RussTreadon-NOAA]

Glad this makes sense. We need to update modulefiles/gsi_common.lua.

@@ -15,7 +15,7 @@ local sfcio_ver=os.getenv("sfcio_ver") or "1.4.1"
 local nemsio_ver=os.getenv("nemsio_ver") or "2.5.2"
 local wrf_io_ver=os.getenv("wrf_io_ver") or "1.2.0"
 local ncio_ver=os.getenv("ncio_ver") or "1.0.0"
-local crtm_ver=os.getenv("crtm_ver") or "2.4.0"
+local crtm_ver=os.getenv("crtm_ver") or "2.4.0.1"

 load(pathJoin("netcdf", netcdf_ver))

I also see that our modulefiles/gsi_wcoss2.lua is a bit behind. Looking at what NCO has loaded in gdas_atmos_analysis_ job log files we should update to

@@ -3,11 +3,11 @@ help([[

 local PrgEnv_intel_ver=os.getenv("PrgEnv_intel_ver") or "8.1.0"
 local intel_ver=os.getenv("intel_ver") or "19.1.3.304"
-local craype_ver=os.getenv("craype_ver") or "2.7.8"
-local cray_mpich_ver=os.getenv("cray_mpich_ver") or "8.1.7"
+local craype_ver=os.getenv("craype_ver") or "2.7.10"
+local cray_mpich_ver=os.getenv("cray_mpich_ver") or "8.1.9"
 local cmake_ver= os.getenv("cmake_ver") or "3.20.2"
 local python_ver=os.getenv("python_ver") or "3.8.6"
-local prod_util_ver=os.getenv("prod_util_ver") or "2.0.10"
+local prod_util_ver=os.getenv("prod_util_ver") or "2.0.14"

 load(pathJoin("PrgEnv-intel", PrgEnv_intel_ver))
 load(pathJoin("intel", intel_ver))

[ADCollard]

OK I will update

[ilianagenkova]

Sorry for the slow reply - too many meetings, Changes look good, i.e.I didn't test run anything.

@ADCollard, do you mind adding a line at the top of read_prepbufr.f90, where the changes log is, to document that the saildrone subtype is introduced in GSI (i.e. not in prepobs) kx=180,280 subtype=02 . Thanks!

[RussTreadon-NOAA]

WCOSS2 tests

Build ADCollard:release/gfsda.v16 at 70c85bb7 on Cactus. Run gsi.x in stand-alone gsi scripts using 20241004 00Z gdas input. For this test use operational gfs.v16.3.19 GSI fix files. PR gsi.x reproduces operations. Repeat test using PR fix. Again, reproduce operations. These are expected results. Neither the source code or convinfo file change alter results in gfs.v16.3.19 operations.

Run on final test in which use prepbufr file from @ilianagenkova. Use GSI fix from this PR in the test. The PR convinfo places saildrone obs in monitor mode. Unfortunately, the 20241004 00Z prepbufr does not appear to contain saildrone data. The gsistat file from the test does not show an type 180 / 280 subtype 0002 data.

@ilianagenkova , do we have a recent cycle with saildrone data?

[ilianagenkova]

@RussTreadon-NOAA [clogin05 /lfs/h2/emc/stmp/iliana.genkova/CRON/R13/com/obsproc/v1.3]$ ll gfs///prepbufr -rw-r----- 1 iliana.genkova rstprod 100100296 Oct 1 21:07 gfs.20241001/18/atmos/gfs.t18z.prepbufr -rw-r----- 1 iliana.genkova rstprod 109288536 Oct 2 03:05 gfs.20241002/00/atmos/gfs.t00z.prepbufr -rw-r----- 1 iliana.genkova rstprod 82710896 Oct 2 14:13 gfs.20241002/06/atmos/gfs.t06z.prepbufr -rw-r----- 1 iliana.genkova rstprod 90002176 Oct 2 14:59 gfs.20241002/12/atmos/gfs.t12z.prepbufr -rw-r----- 1 iliana.genkova rstprod 104919288 Oct 3 03:04 gfs.20241003/00/atmos/gfs.t00z.prepbufr -rw-r----- 1 iliana.genkova rstprod 78576040 Oct 3 08:59 gfs.20241003/06/atmos/gfs.t06z.prepbufr -rw-r----- 1 iliana.genkova rstprod 91610736 Oct 3 15:00 gfs.20241003/12/atmos/gfs.t12z.prepbufr -rw-r----- 1 iliana.genkova rstprod 95284888 Oct 3 21:05 gfs.20241003/18/atmos/gfs.t18z.prepbufr -rw-r----- 1 iliana.genkova rstprod 106210056 Oct 4 03:02 gfs.20241004/00/atmos/gfs.t00z.prepbufr -rw-r----- 1 iliana.genkova rstprod 86805984 Oct 4 08:59 gfs.20241004/06/atmos/gfs.t06z.prepbufr -rw-r----- 1 iliana.genkova rstprod 88825544 Oct 4 15:00 gfs.20241004/12/atmos/gfs.t12z.prepbufr

[ADCollard]

It seems to work for me for 2024100200 gdas.

[RussTreadon-NOAA]

Thank you @ilianagenkova . I ran 20241004 00Z gfs prepbufr through this PR. No type 180 / 280 subtype 0002 data is listed in gsistat. This makes sense. The 20241004 00Z gdas prepbufr file did not contain any type 180 / 280 subtype 0002 data.

Seems we don't receive saildrone data every cycle. I don't know how many saildrones there are or their reporting frequency. It doesn't surprise me that there may be cycles without saildrone data.

I ran 2024100202 00Z (@ADCollard case) through this PR. As expected, I see type 180 / 280 subtype 0002 observation in monitor mode

 o-g 01      ps mon 180 0002      1946  0.145E+00  0.339E+00  0.436E-01  0.436E-01
 o-g 01      uv mon 280 0002 count       1946          0          0          0          0          0          0          0          0          0          0       1946
 o-g 01       t mon 180 0002 count       1946          0          0          0          0          0          0          0          0          0          0       1946
 o-g 01       q mon 180 0002 count       1586          0          0          0          0          0          0          0          0          0          0       1586

[RussTreadon-NOAA]

@ADCollard : Do we want to add the requested comment to read_prepbufr.f90?

[RussTreadon-NOAA]

Suggested addition on line 151

!   2024-10-04  collard - saildrone subtype 02 added for kx 180 / 280 (not from prepbufr)

[ilianagenkova]

@RussTreadon-NOAA , you are correct, there is no saildrone in 20241004 00z prepbufr (b/c I was testing something)

I can regenerate the 20241004 00 prepbufr (with saildrone) in just 15 minutes.

Sorry about that!

[RussTreadon-NOAA]

@ilianagenkova , no need to regenerate any files. The test with @ADCollard 's file is sufficient.

[ADCollard]

@dmerkova @RussTreadon-NOAA I have added the comment as requested. Sorry for the delay.

[RussTreadon-NOAA]

@ADCollard tag gfsda.v16.3.20 has been created.

[ADCollard]

@RussTreadon-NOAA Thankyou!