Observer End-to-End Test for OMI

emilyhcliu commented 1 year ago

End-to-End Tests for OMI Processing

[x] Create BUFR converter - python script
[x] Test BUFR converted IODA obs input to H(x)
[x] Test BUFR converted IODA obs input to H(x) + initial error assignment + read routine QC
[x] Test BUFR converted IODA obs input to H(x) + initial error assignment + read routine QC + setup routine QC

Issues found in the existing data filtering yaml (omi_aura.yaml) are documented here:
No observation error assignment (all filled with missing values)
Gross check was not implemented correctly
Quality control variable types do not comply with IODA convention --- they need to be integers
Quality control variable names do not comply with IODA convention --- should be camel case

Updates made and tested

Added observation error assignment and fixed gross check
The quality control flags (toqc, toqf, and afbo) are integers. Modified the filters accordingly in the YAML
The quality control flags names do not comply IODA convention (should be camel case). TOQF - quality indicator; extract bit-10 to check row anomaly ---> totalOzoneQualityFlag TOQC - total ozone error flag (0, 1 are good) ---> totalOzoneQualityCode AFBO - algorithm flag ( exclude 3, 13 indicating data retrieved with c-pair algorithm using 331 and 360 nm) ---> bestOzoneAlgorithmFlag

QC Procedures: (Data thinning not applied)

read routine QC

Observation value sanity check
- Total Ozone quality check - remove data affected by row anomaly (bit 10 of TOQF)
- Scan position check - remove scan angle >=25
- Total Ozone Quality Code Check (TOQC) --- accept TOQC = 0, 1 only
- Algorithm Check --- remove data retrieved using the c-pair algorithm (3, 13)

setup routine QC