Scatterometer Winds (ScatWinds) Validation

[BrettHoover-NOAA]

Need to validate ScatWind observations from all platforms

This issue will track progress on validation of ScatWinds in UFO for GDAS in four stages:

• [x] HofX computation using GSI produced GeoVaLs
• [x] QC count checks (and YAML changes) using GSI produced GeoVaLs
• [x] Obs errors using GSI produced GeoVaLs
• [x] HofX computation in FV3-JEDI

Note this will need to include any near-surface correction applied to the observation/hofx value.

[BrettHoover-NOAA]

There are very few explicit QC features for scatterometer winds (as opposed to, say, atmospheric motion vectors). There is a background check with a 5 m/s cutoff, and the "ambiguity check". A scatterometer wind measurement uses a function that provides a non-unique solution, up to 4 solutions, for the wind speed and direction. Often these 4 potential observations, or "ambiguities", are at right-angles or are 180-degrees opposed to one another. Typically in pre-processing a single ambiguity is selected as the best-fit and presented as the sole wind speed and direction for a given scatwind.

HOWEVER, sometimes the wrong ambiguity is selected, and the ambiguity check is designed to account for this. The check is relatively simple: the fit of the observed wind to the model background (computed as the length of the vector difference) is compared to the fit of a hypothetical wind observation rotated 180-degrees. If the fit of the hypothetical wind is better than for the observed wind, the observation is rejected. You can find the ambiguity check in setupw.f90 here.

A first pass of the scatwind QC in UFO shows that there is high acceptance with just the 5 m/s background check:

Only 2.33% of the test data is in disagreement, and all of it is GSI-fail, JEDI-pass, indicating that JEDI is missing a rejection condition. I believe that this ambiguity check is probably the source of many of these disagreements. In a simple test simulating the ambiguity check, all of the 624 disagreeing observations fail the ambiguity check, meaning they should align with UFO acceptance once a comparable check is in-place:

Note that the ambiguity test is a GSI convention where BOTH the u- and v- components are rejected if EITHER the u- or v- component exceeds a 5 m/s background check, while JEDI treats u- and v- components separately and can reject one while accepting the other. We need to make a science decision about whether this difference is a feature (and should be documented in the exceptions list in NOAA-EMC/JEDI-T2O Issue #47) or a bug and we should strive to reject both components.

I have submitted test-code to draft PR to JCSDA-internal/ufo for adding this test as an obsfunction titled ScatWindsAmbiguityCheck, #2542, from the branch feature/scatwinds_ambiguity_obsfunction. I am currently trying to rebuild GDASApp with this branch of ufo for more formal testing.

[BrettHoover-NOAA]

Digging a little deeper on the issue of tandem u/v rejections (GSI-style) vs independent u/v rejections (JEDI-style): I wrote a small reconciliation function to find any UFO-rejected winds of one component that did not reject the other component from the same observation, and in the test data I find 222 v-winds whose corresponding u-wind was rejected, and 331 u-winds whose corresponding v-wind was rejected, but were not rejected themselves. In reconciliation, I reject the corresponding wind component and adopt the QC flag value of the component that was originally rejected. This action effectively simulates a GSI-style tandem u/v rejection in UFO:

Now when I run the ambiguity check function on either the u-component or v-component of the wind, there are 293 observations in disagreement between GSI and UFO and all 293 are caught by the ambiguity check:

These 293 observations are also found as the only observations out of GSI/UFO acceptance compliance with either the u-component or v-component:

This is very good confirmation that the ambiguity check is the remaining difference between GSI and UFO, excepting the difference in GSI's tandem u/v rejection and UFO's independent u/v rejection. I will build a GDASApp with the ScatWindsAmbiguityCheck obsfunction to try and eliminate these remaining differences.

[BrettHoover-NOAA]

Building a simulator for GSI rejections (so I can assign GSI rejections to a particular criteria), I reach almost, but not quite, total parity with actual GSI rejection:

The differences come down to exactly 2 observations, both of which are erroneously rejected in the GSI simulator. Both of these observations have surface types of 3, which indicates that the surface is a snow type, and should be rejected based on that criteria alone – in fact, they shouldn't appear in the diag file at all, since scatterometer winds in the GSI are tested for surface type and any non-open-ocean (surface type 1) observations are hard-rejected by cycling immediately to the next observation in processing without being retained. It is noteworthy that these two observations are also very close to the coast:

Likely, these 2 observations have an appropriate surface type in GSI but in JEDI, due to some interpolative difference, the surface type is reported as a snow instead of open-ocean. So we can probably safely discard these 2 observations from consideration as outliers with a known (or highly suspected) cause.

[BrettHoover-NOAA]

I was able to test the ScatWindsAmbiguityCheck ObsFunction in GDASApp, and provided that the settings for the filter are correct, all scatterometer winds UFO acceptance is compliant with GSI, with the exception of the 2 observations described above.

What I've discovered is that the Bounds Check filter that is used to perform the ambiguity-check on ASCAT winds requires setting the optional parameter minimum_uv to its lowest acceptable value of 0.0001:

  # Reject ASCAT (Type 290) when ambiguity check fails (returned value is negative)
  - filter: Bounds Check
    filter variables:
    - name: windEastward
    - name: windNorthward
    where:
    - variable: ObsType/windEastward
      is_in: 290
    test variables:
    - name: ObsFunction/ScatWindsAmbiguityCheck
      options:
        test_hofx: GsiHofX
        minimum_uv: 0.0001 # hard-coding a minimum-uv for transparency, want this to basically be zero
    maxvalue: 0.  
    action:
      name: reject

The use of minimum_uv: 0.0001 will force the filter to be applied to any observation in its scope (in this case, observations with a ObsType/windEastward value of 290, or ASCAT winds) with a u- or v-component of at least 0.0001 m/s. This is the explicit minimum acceptable value for minimum_uv as defined in ScatWindsAmbiguityCheck.cc:

  // Retrieve minimum_uv value and assure it is sensible.
  const float min_uv = std::max(0.0001f, options_.minimum_uv.value());

What I've discovered is that if this minimum_uv value is not set in the YAML, I have 65 winds in disagreement, all of which are rejected in GSI for the ambiguity check but pass UFO QC. All 65 winds have u- and v- components with an absolute value of less than 0.5 m/s, which means they would be skipped in ScatWindsAmbiguityCheck if minimum_uv were set to a value of 0.5.

This makes me think that absent of setting minimum_uv in the YAML, the value of minmum_uv is defaulting to a value of 0.5 m/s rather than the hard-coded minimum of 0.0001 m/s. Which is also making me rethink my posture on any filter for any wind type where a minimum_uv can be set. Do I need to explicitly set minmum_uv: 0.0001 everywhere where a minimum_uv option is available but has not been explicitly defined in the YAML?

[BrettHoover-NOAA]

Significant updates to the UFO code-base since February have closed all remaining issues for this task. The enhancements to UFO's code-base are:

• SatWindsSPDBCheck: The name being a slight misnomer, this is actually the gross-error check for all winds but it contains specific modifications for satwinds, particularly those with a speed bias (SPDB). Duplicates the gross-error check in setupw.f90.
• ObsErrorFactorPressureCheck: This check will inflate errors for observations that are very close to either the top or bottom vertical level of the model, which is critical for scatterometer winds since they are measured at the surface (technically assigned to 10m above surface).
• ScatWindsAmbiguityCheck: This function duplicates the QC for scatterometer winds in setupw.f90

Changes to the scatwind geovals file were also made to replicate checks in read_prepbufr.f90 to reject observations with a non-ocean surface type or with surface temperatures at or below 273 K.

The test-data has 25,888 observations (u and v pairs) passing in GSI, while 2 of these observations are rejected in UFO resulting in 25,886 observations passing. The 2 outstanding observations are the ones with differing acceptance criteria are correctly rejected in UFO for having a non-ocean surface type (a value of 3, a snow-type, for both observations), while in GSI the assigned ob-type is ocean surface (a value of 1). This difference is likely traceable to an underlying difference in how the surface category is calculated for the interpolated point in both the GSI and UFO, and as shown in the figure above both of these observations appear to be near a land/ocean boundary in the arctic. These 2 observations are considered an unreconcilable exception and will be referenced in #47.

Also as described above, UFO rejects u and v independently of each other in the ScatWindsAmbiguityCheck while GSI will reject the entire u,v pair if either fails the check. This generates differences in the acceptance between UFO and GSI, but when UFO rejections are reconciled by rejecting the accompanying ob-variable when one ob-variable is rejected, the acceptance between UFO and GSI is identical with the exception of the 2 observations described above.

Observation errors are identical between UFO and GSI to within 1.0E-05 m/s, which seems acceptable.