[End-to-End Test Code Sprint] Add JSON, YAML, Python code for GDASApp end-to-end testing of AHI Himawari-8 satwinds

[BrettHoover-NOAA]

Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8 to GDASApp end-to-end testing

new files include: parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI Himawari-8 satwinds parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data format, sensor, and satellite information for AHI Himawari-8 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for extracting AHI Himawari-8 satwinds from BUFR

modified files include: ush/ioda/bufr2ioda/run_bufr2ioda.py: added "satwind_amv_ahi" to list BUFR_py

End-to-End Test Results

Himawari-8 satwinds consist of 3 observation-types: (LW)IR (252), VIS (242), and both clear-sky and cloud-top WV (250), which are given the same ob-type but can be differentiated by their windComputationMethod values of 3 and 5 respectively. Clear-sky WV winds from AHI/Himawari-8 are not assimilated in GSI.

No thinning results

With horizontal and vertical thinning turned off in GSI, all AHI/Himawari-8 satwinds are assimilated:

(LW)IR satwinds

There are 46877 IR satwinds in the JEDI dataset and 45092 in the GSI dataset - the discrepancy is almost certainly due to the exclusion of IR satwinds over land north of 20 degrees latitude in the GSI, where these observations are removed from the dataset entirely in read_satwnd.f90 and do not carry through to the diag file. When filtering to only compare assimilated IR satwinds, JEDI assimilates 27530 observations and GSI assimilates 27454, a difference of roughly 0.3%. Some of the differences appear along coastlines and could be due to differences in whether the underlying surface-type is defined as land or water:

Comparison of the observations, the HofX, and the difference (ob minus HofX) look good:

VIS satwinds

There are 29814 VIS satwinds for both JEDI and GSI, and both systems assimilate 8992 of them (so I will skip that plot). The rest of the comparisons look good:

(cloud-top) WV satwinds

All clear-sky WV satwinds are rejected in JEDI for compliance with GSI. There are 49867 satwinds in the combined cloud-top and clear-sky observation type for both systems, JEDI assimilates 35190 and GSI assimilates 35080, a differences of less than 0.5%:

Other tests look good:

Thinning Tests

TBD

[BrettHoover-NOAA]

Thinning Tests

Horizontal and vertical thinning (NOTE: no JEDI time-thinning applied in this test, but GSI contains time-thinning) was applied in JEDI with the use of the following filter option in obs pre filters:

- filter: Gaussian Thinning
  horizontal_mesh: 200
  vertical_mesh: 10000
  use_reduced_horizontal_grid: true
  round_horizontal_bin_count_to_nearest: true
  partition_longitude_bins_using_mesh: true

This applies horizontal thinning on a 200 km mesh and vertical thinning on a 10000 Pa mesh, matching the horiz/vert thinning mesh size options chosen in GSI global_convinfo.txt. The remaining options used here were carried over from existing testing of JEDI thinning of scatterometer wind data.

Even when applying only horizontal and vertical thinning in JEDI and comparing to GSI's horizontal/vertical/time thinning, JEDI assimilates roughly 25-40% of the AHI/H-8 satwinds that are assimilated in GSI. I will show only the differences in QC'd JEDI (blue) vs GSI (orange) observations, since the ob, HofX, and OmB values were already tested without thinning applied.

(LW)IR satwinds

There are 46877 IR satwinds in JEDI, versus 16386 satwinds in GSI, prior to QC - JEDI has more than GSI. But after QC, JEDI assimilates 1684 satwinds and GSI assimilates 7352, so JEDI assimilated ob-counts are much lower. The JEDI observations seem to be more spread-out than GSI, implying the difference is due to differences in how thinning is being applied within the mesh grids:

VIS satwinds

There are 29814 VIS satwinds in the JEDI dataset and 12318 in the GSI dataset, prior to QC - JEDI has more than GSI. But after QC, JEDI assimilates 1083 satwinds and GSI assimilates 2885, so JEDI assimilated ob-counts are much lower. The JEDI observations seem to be more spread-out than GSI, implying the difference is due to differences in how thinning is being applied within the mesh grids:

WV (cloud-top only) satwinds

There are 48867 cloud-top WV satwinds in JEDI and 15446 cloud-top WV satwinds in GSI - JEDI has more than GSI. But after QC, JEDI assimilates 2482 satwinds and GSI assimilates 6450, so JEDI assimilated ob-counts are much lower. The JEDI observations seem to be more spread-out than GSI, implying the difference is due to differences in how thinning is being applied within the mesh grids:

There needs to be ongoing work to identify differences in the JEDI vs GSI thinning techniques to explain ob-count differences before and after QC. The thinning filter applied here will be included in the YAML file but commented-out with a note summarizing these results.