search

JCSDA-internal / ioda-converters

Various converters for getting obs data in and out of IODA
8 stars 2 forks source link

add sample_flags for osw #1509

Closed BenjaminRuston closed 1 month ago

BenjaminRuston commented 2 months ago

Description

The NOAA CYGNSS ocean surface winds provide a sample_flags that should be used to flag poor_overall_quality_flag

A change in the list generation is also added this speeds up the converter by 10x or so

Issue(s) addressed

Resolves #1508

Impact

A preQC value is passed from data to IODA file

Checklist

BenjaminRuston commented 2 months ago

adding a --qualitycontrol that then drops data that have preQC flag set

BenjaminRuston commented 2 months ago

@smaticka think this is all good,

they note in the headers that these flags also exist (below) do you think we should add those into the check as well?

the bit1 flag includes those later flags, the comment on the flag notes this in f['sample_flags'].attrs['comment']:

1 = poor_overall_quality: logical OR of poor_quality_wind_nst, poor_quality_wind_sample,

And yes, I agree, the snr could be looked at as a possible additional PreQC.

I've consulted with the CYGNSS team at NOAA and they recommend just using the bit1 from the sample_flags

BenjaminRuston commented 2 months ago

thanks @smaticka this drop is now working https://github.com/JCSDA-internal/ioda-converters/blob/9085b0b9a78d3af9d9450e1aed374fd9e491be8a/src/hdf5/osw_2ioda.py#L309

the is now used with the --qualitycontrol option strict qc is enforced and the resulting file removes the points that do not pass PreQC:

PatNichols commented 1 month ago

@BenjaminRuston Is there a particular epic for this one?