[End-to-End Validation] Add JSON, YAML, Python code for GDASApp end-to-end validation of AVHRR NOAA-15/18/19 satwinds

[BrettHoover-NOAA]

Adding satwinds from the Advanced Very High Resolution Radiometer (AVHRR) from NOAA-15/18/19 to GDASApp end-to-end testing

new files include: parm/atm/obs/config/satwind_avhrr_n15.yaml.j2: QC filter YAML for AVHRR NOAA-15 satwinds (jinja2 standard) parm/atm/obs/config/satwind_avhrr_n18.yaml.j2: QC filter YAML for AVHRR NOAA-18 satwinds (jinja2 standard) parm/atm/obs/config/satwind_avhrr_n19.yaml.j2: QC filter YAML for AVHRR NOAA-19 satwinds (jinja2 standard) parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_avhrr.json: JSON containing data format, sensor, and satellite information for AVHRR NOAA-15/18/19 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_avhrr.py: bufr2ioda code for extracting AVHRR NOAA-15/18/19 satwinds from BUFR

End-to-End Test Results

AVHRR satwinds consist of (LW)IR (type=244) from multiple satellites. The effort for this issue is to enroll NOAA-15 (subtype=206), NOAA-18 (subtype=209), and NOAA-19 (subtype=223) satwinds that are tanked and dumped into BUFR subset NC005080. AVHRR satwinds from Metop-A/B/C (subtype=3-5) tanked and dumped into BUFR subset NC005081 will need to be included in the future, but there is currently no test data available for these winds.

Thinning is turned off in both GSI and JEDI for these tests, as JEDI's thinning procedure is structurally different from GSI's and introduces large numbers of asymmetries in assimilation counts.

NOAA-15 satwinds (type=244, subtype=206)

There are 188 NOAA-15 AVHRR satwinds in the test dataset, 144 are assimilated in JEDI and 143 are assimilated in GSI. The single difference in acceptance is at 13700 Pa and the difference in rejection is speculated to be related to differences in definition of the pressure-level of the tropopause, whereby satwinds at a pressure more than 5000 Pa above the tropopause. There are no other QC filters that are a good prospect for this one acceptance difference and it is known that GSI and JEDI differ to some degree on the tropopause pressure.

Accepted observations are distributed similarly between GSI and JEDI:

The windEastward and windNorthward values, their HofX values, and the OmB look good comparing GSI and JEDI:

NOAA-18 satwinds (type=244, subtype=209)

There are 674 NOAA-18 AVHRR satwinds in the test dataset, 605 are assimilated in JEDI and 604 are assimilated in GSI. The single difference in acceptance is at 28700 Pa and the difference in rejection is again speculated to be related to differences in definition of the pressure-level of the tropopause.

Accepted observations are distributed similarly between GSI and JEDI:

The windEastward and windNorthward values, their HofX values, and the OmB look good comparing GSI and JEDI:

NOAA-19 satwinds (type=244, subtype=223)

There are 1392 NOAA-19 AVHRR satwinds in the test dataset, 1091 are assimilated in both JEDI and GSI.

Accepted observations are distributed similarly between GSI and JEDI:

The windEastward and windNorthward values, their HofX values, and the OmB look good comparing GSI and JEDI:

[ADCollard]

@BrettHoover-NOAA These results look good. Can you also comment on the final observation errors (JEDI vs GSI).

[BrettHoover-NOAA]

@ADCollard Sure, final error analysis follows:

Overall error comparisons between JEDI and GSI look good for all three subtypes - there are a few outstanding differences where GSI assigns a higher error to an observation than JEDI, but these are infrequent and are likely due to differences in duplicate error inflation since GSI is processing all AMVs on a single processor and can identify duplicates across types while JEDI processes types on separate processors and the cross-type duplicates are invisible.

NOAA-15 AVHRR satwind errors (type=244, subtype=206) plan map distribution vertical profile

NOAA-18 AVHRR satwind errors (type=244, subtype=209) plan map distribution vertical profile

NOAA-19 AVHRR satwind errors (type=244, subtype=223) plan map distribution vertical profile

[ADCollard]

@BrettHoover-NOAA I agree the differences are most likely the duplicate issue.