Release/gfs.v16.3.16 wdqmsfix

[emilyhcliu]

Description

The ObsProc upgraded from version 1.1 to version 1.2, in which the high-resolution radiosonde observations are included. The radiosonde data from ObsProcv1.2 stored in GSI diagnostic files, in some cases, have multiple observations in the same location/time, and these observations often contain bad data (e.g., temperature > 500K or temperature < 50K). These duplicate observations and bad data caused WDQMS processing to fail. This PR improves the handling of these obscure cases.

Example of duplicate data with bad values for temperature:

Example - ObsProc v1.2 station ID 76225
         Latitude   Longitude    Pressure     Time  Observation  Obs_Minus_Forecast_adjusted
 85241  28.629459 -105.604614  161.800003 -0.39472   214.850006                    -1.637150
185242  28.630369 -105.598633  159.899994 -0.39056   215.149994                    -1.470619
185243  28.631210 -105.593018  158.300003 -0.38667   215.750000                    -0.988078
185244  28.631210 -105.593018   19.500000 -0.38667   190.850006                   -34.529102
185246  28.631210 -105.593018   19.500000 -0.38667   518.549988                   293.170898
185247  28.631210 -105.593018   19.500000 -0.38667   518.549988                   293.170898
185248  28.631210 -105.593018    0.600000 -0.38667   364.350006                    97.087112
185249  28.631210 -105.593018    0.600000 -0.38667   382.250000                   114.987114
185250  28.631210 -105.593018    0.600000 -0.38667   489.850006                   222.587112
185251  28.631210 -105.593018    0.600000 -0.38667   308.049988                    40.787113
185252  28.631210 -105.593018    0.600000 -0.38667    31.549999                  -235.712891
185253  28.631210 -105.593018    0.600000 -0.38667   174.949997                   -92.312889
185254  28.631210 -105.593018    0.600000 -0.38667   584.549988                   317.287109
185255  28.631210 -105.593018    0.600000 -0.38667    67.449997                  -199.812881
185256  28.631210 -105.593018    0.600000 -0.38667   487.250000                   219.987106
185257  28.631210 -105.593018    0.600000 -0.38667   154.449997                  -112.812889
185258  28.631210 -105.593018    0.600000 -0.38667   297.850006                    30.587112
185259  28.631210 -105.593018    0.600000 -0.38667   581.950012                   314.687103
185261  28.631210 -105.593018    0.600000 -0.38667   133.949997                  -133.312881
185262  28.631210 -105.593018    0.600000 -0.38667   236.350006                   -30.912888
185263  28.631210 -105.593018    0.600000 -0.38667   341.350006                    74.087112
185264  28.631210 -105.593018    0.600000 -0.38667   443.750000                   176.487106
185265  28.631210 -105.593018    0.600000 -0.38667   546.150024                   278.887115
185266  28.631210 -105.593018    0.600000 -0.38667    98.150002                  -169.112885
185267  28.631210 -105.593018    0.600000 -0.38667    44.349998                  -222.912888
185268  28.631210 -105.593018    0.600000 -0.38667   149.350006                  -117.912888
185269  28.631210 -105.593018    0.600000 -0.38667   251.750000                   -15.512887
185270  28.631210 -105.593018    0.600000 -0.38667   351.549988                    84.287109
185271  28.631210 -105.593018    0.600000 -0.38667   369.450012                   102.187111
185272  28.631210 -105.593018    0.600000 -0.38667   477.049988                   209.787109
185273  28.631210 -105.593018    0.600000 -0.38667   594.750000                   327.487122
185274  28.631210 -105.593018    0.600000 -0.38667   103.250000                  -164.012894
185275  28.631210 -105.593018    0.600000 -0.38667   205.649994                   -61.612888
185276  28.631210 -105.593018    0.600000 -0.38667   313.149994                    45.887112
185277  28.631210 -105.593018    0.600000 -0.38667   418.149994                   150.887115
185278  28.631210 -105.593018    0.600000 -0.38667   520.549988                   253.287109
185279  28.631210 -105.593018    0.600000 -0.38667   466.750000                   199.487106
185280  28.631210 -105.593018    0.600000 -0.38667   569.150024                   301.887115
185281  28.631210 -105.593018    0.600000 -0.38667    21.350000                  -245.912888
185282  28.631210 -105.593018    0.600000 -0.38667   108.349998                  -158.912888
185283  28.631210 -105.593018    0.600000 -0.38667   215.850006                   -51.412888
185284  28.631210 -105.593018    0.600000 -0.38667   318.250000                    50.987114
185285  28.631210 -105.593018    0.600000 -0.38667   441.149994                   173.887115
185286  28.631210 -105.593018    0.600000 -0.38667   523.049988                   255.787109
185287  28.631210 -105.593018    0.600000 -0.38667    77.650002                  -189.612885
185288  28.631210 -105.593018    0.600000 -0.38667   172.350006                   -94.912888
185289  28.631210 -105.593018    0.600000 -0.38667   369.450012                   102.187111
185290  28.631210 -105.593018    0.600000 -0.38667   384.850006                   117.587112
185291  28.631210 -105.593018    0.600000 -0.38667   515.450012                   248.187119
185292  28.631210 -105.593018    0.600000 -0.38667   282.450012                    15.187113
185293  28.631210 -105.593018    0.600000 -0.38667   410.450012                   143.187119
185294  28.631210 -105.593018    0.600000 -0.38667   180.050003                   -87.212891
185295  28.631210 -105.593018    0.600000 -0.38667   308.049988                    40.787113
185297  28.631210 -105.593018    0.600000 -0.38667   203.050003                   -64.212891
185298  28.631210 -105.593018    0.600000 -0.38667   259.450012                    -7.812887
185299  28.631210 -105.593018    0.600000 -0.38667   456.549988                   189.287109
185300  28.631210 -105.593018    0.600000 -0.38667   558.950012                   291.687103
185301  28.631210 -105.593018    0.600000 -0.38667     3.450000                  -263.812897

Resolves issues described in this PR Notes: No GitHub issue opened because I do not have permission to open an issue in this repository

Type of change

• Bug fix (fixes something broken)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO; Changes are documented in the code.

How has this been tested?

Using stand-alone WDQMS processing with input from current operational diagnostic files (from ObsProcv1.2) Cycled tested: 2022050800Z - 2024052206Z using diagnostics files from NCO parallele run for ObsProc1.2 upgrade 2024052212Z -2024052818Z using diagnostic files from Operational GFS (with ObsProc1.2)

All three required output files (TEMP, MARINE, and SYNOP) were generated for each cycle with valid header and content.

Checklist

• [x] Any dependent changes have been merged and published
• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings
• [ ] New and existing tests pass with my changes
• [x] I have made corresponding changes to the documentation if necessary

[emilyhcliu]

@KateFriedman-NOAA and @aerorahul Could you please add @kevindougherty-noaa as a reviewer? Thanks.