[End-to-End Validation] Add Sfc YAML for end-to-end validation

[kevindougherty-noaa]

sfc.yaml using filters used from SWELL.

specificHumidity, airTemperature, virtualTemperature, windEastward, and windNorthward are all rejceted for types 181, 187, 281 and 287 (https://www.emc.ncep.noaa.gov/mmb/data_processing/prepbufr.doc/table_2.htm)

stationPressure is the only variable that should have filters/error assignments.

H(x) Comparison

JEDI h(x) vs. GSI h(x) for station pressure:

Observations vs. JEDI h(x) for station pressure:

JEDI-GSI h(x) difference histogram for station pressure:

QC Flag Comparison

Observation Error

[kevindougherty-noaa]

After testing obs filters to match Emily's conv_ps.yaml closer because we are only assimilating stationPressure, we ran into some issues where all error values were being NaN'ed using the following filter:

# Calculate obs error inflation factors for large discrepancies between model and observations 
- filter: Variable Assignment
  assignments:
  - name: ObsErrorFactorSfcPressure/stationPressure
    type: float
    function:
      name: ObsFunction/ObsErrorFactorSfcPressure
      options:
        geovar_sfc_geomz: surface_geometric_height
        geovar_geomz: geopotential_height
        station_altitude: height 

# Inflate surface pressure observation based on discrepancies between
# model and observations due to terrian
- filter: Perform Action
  filter variables:
  - name: stationPressure
  action:
    name: inflate error
    inflation variable:
      name: ObsErrorFactorSfcPressure/stationPressure

After discussion with @ADCollard and @emilyhcliu, we discovered the sfc_geoval .nc4 file was flipped. So we added a condition to the geovals input section of the yaml as such:

geovals:
  filename: !ENV sfc_geoval_${CDATE}.nc4
  levels_are_top_down: false

This ran appropriately and produced the following output:

These results actually increased the error differences, so we will need to investigate further.

[emilyhcliu]

There are two major error inflation processes for surface pressure. One is from the duplicate check; the other is from the surface pressure check between the model and obs (the filter we fixed this afternoon). So, you can plot the error inflation factor from these two processes. If you can get these two error factors done correctly (compared to GSI), your final obs error should be closer to GSI).

Here are the two error comparison plots (JEDI vs. GSI): ObsErrorFactorDuplicateCheck ObsErrorFactorSfcPressure