Use Python API to convert ADPUPA prepbufr data to IODA and check if this can be used with UFO seamlessly

[PraveenKumar-NOAA]

Configuration

• Test Date is 2021080100
• Model Background: FV-3 6-hour forecast on the cubic-sphere grid
• BUFR Converter :
  • convert timeOffset/timeCycleProfileLevel to dateTime (in units of seconds since 1970-01-01T00:00:00Z)
  • convert longitude from [0,360] to [-180,180]
  • Add conversions of: latitude, longitude, prepbufrDataLeleCategoty, stationIdentification, stationElevation, windEastward, windNorthward, airTemperature, virtualTemperature, specificHumidity

Test using fv3jedi_hofx_nomodel.x with the IODA files converted from BUFR converter

• Convert BUFR to IODA format using Python API
• Run UFO HofX

Check List

Investigate diag output from UFO

• [x] ADPUPA prepbufr version of Python API
• [ ] Compare Python version of IODA with bufr2ioda IODA
• [x] ADPUPA HofX

[PraveenKumar-NOAA]

The bufr -> ioda -> hofx plumbing worked well for the following ADPUPA observations using Python API:

• windEastward
• windNorthward
• airTemperature
• virtualTemperature
• specificHumidity

The validation plots are shown below. The results from running GDASApp for ~/parm/atm/obs/testing/sondes.yaml (no QC) are also shown with these plots, which are very similar and are in agreement:

- windEastward

- windNorthward

- airTemperature

- virtualTemperature

[PraveenKumar-NOAA]

- specificHumidity

[PraveenKumar-NOAA]

ObsValue vs hofx plots with quality marks for the following observations:

windEastward windNorthward airTemperature virtualTemperature specificHumidity

Mostly observations belong to the QM 0-3.

[PraveenKumar-NOAA]

@emilyhcliu bufr -> ioda -> hofx plumbing work is completed for ADPUPA prepbufr and the hofx/GSIhofx/ObsVlaue plots (windEastward, windNorthward, airTemperature, virtualTemperature, specificHumidity) turned out very well. Let me know what you think?

[PraveenKumar-NOAA]

@emilyhcliu verified results again. Looks everything good.

[emilyhcliu]

@PraveenKumar-NOAA Great! Thanks. Based on our discussion this morning, let's do the following: (1) make stationPressure the same dimension as the rest of the variables; this can be done by using where function for numpy array (2) extract QM mark for stationPressure.

[emilyhcliu]

I modified your python script slightly. Here are my modifications: (1) Add one more BUFR Mnemonic called TPC (Temperature Event Program Code)

• TPC=1 ---> TOB is the sensible temperature
• TPC<= 8 and TPC > 1 ---> TOB is the virtual temperature

(2) Create airTemperature and virtualTemperature from TOB based on TPC (3) Create stationPressure based on POB (with the same dimension: Location)

You add TVO as virtual temperature, but TVO is not the virtual temperature used by GSI.
Both virtualTemperature and airTemperature are from TOB TPC is set to tell if TOB is virtual or sensible temperature.

Here is the code table for TPC

  0-12-247 - TPC
PREPBUFR: Temperature (TOB) event program code

0     | Reserved
1     | Initial PREPBUFR processing step "PREPRO" (performed in PREPOBS_PREPDATA program, prior to "PREVENT" and "VIRTMP" steps)
2     | Synthetic tropical cyclone bogus processing step "SYNDATA" (performed in SYNDAT_SYNDATA program, prior to "PREVENT" and "VIRTMP" steps)
3     | Reserved
4     | Pre-quality control step "PREVENT" which adds GFS forecast background and observation error (if present) and performs check of surface pressure
5     | Rawinsonde height/temperature complex quality control step "CQCHT" (performed in PREPOBS_CQCBUFR program, prior to "RADCOR" and "VIRTMP" steps)
6     | Rawinsonde height/temperature intersonde bias (radiation) correction step "RADCOR" (performed in PREPOBS_CQCBUFR program, after "CQCHT" step but prior to "VIRTMP" step)
7     | AIREP, PIREP and AMDAR aircraft quality control step "PREPACQC" (performed in PREOPOBS_PREPACQC program)
8     | Virtual temperature/specific humidity processing step "VIRTMP"
9     | Wind profiler quality control step "CQCPROF" (performed in PREPOBS_PROFCQC program)
10    | Multi-platform OI-quality control step "OIQC" (performed in PREPOBS_OIQCBUFR program)
11    | Global (GFS/GDAS/CDAS) SSI analysis step "SSI" (performed in GLOBAL_SSI and CDAS_SSI programs)
12    | VAD wind quality control step "CQCVAD" (performed in PREPOBS_CQCVAD program)
13    | Regional (ETA/EDAS) 3DVAR analysis step "R3DVAR" (performed in ETA_R3DVAR program)
14    | ACARS aircraft quality control step "ACARSQC" (performed in PREPOBS_ACARSQC program)
15-30 | Reserved
31    | Missing value

I ran the hofx test with the IODA obs file with my modifications. The results are very much like yours. Here are the plots for your reference:

UFO airTemperature	GSI airTemperature

UFO virtualTemperature	GSI virtualTemperature

UFO stationPressure	GSI stationPressure

UFO windEastward	GSI windEastward

UFO windNorthward	GSI windNorthward

UFO specificHumidity	GSI specificHumidity

[emilyhcliu]

Here are a few things to do:

• [ ] Correct sensible temperature and virtual temperature (they should be from TOB based on TPC)
• [ ] Write stationPressure out with the same dimension (Location) as the rest of the variables.
• [ ] write out TOBQM for airtemperature and virtualTemperature
• [ ] write out POBQM for stationPressure

[PraveenKumar-NOAA]

@emilyhcliu thank you, I will update my Python API for this change.

[PraveenKumar-NOAA]

• [x] @emilyhcliu why your GSI plot for stationPressure is different from UFO? Please check my GSI plot here: https://github.com/NOAA-EMC/JEDI-T2O/issues/71

• [x] do we need two pressure QM variables (pressureQM and stationPressureQM) written using the same PQM variable? I think one, pressureQM is sufficient. What you think?

• [x] Same for temperature: do we need two temperature QM variables (airTemperatureQM and virtualTemperatureQM) written using the same TQM variable? I think one, airTemperatureQM is sufficient. What you think?

• [x] what is the difference between airTemperature and sensibleTemperature?

• [x] in feature/amv_goes you mentioned three ways to encode BUFR data to IODA, which one you wanted to be in my Python API. I am currently using one that Ron mentioned in his API example.

[PraveenKumar-NOAA]

Also, if the sensible temperature and virtual temperature are coming from TOB for different TPC, then the virtual temperature is not used correctly in ADPUPA prepbufr YAML.

[emilyhcliu]

@PraveenKumar-NOAA The observation operator for surface pressure is different in GSI and UFO. The UFO (PCorrecter) performs better (closer to observation), because the operator corrects the calculated station pressure with observation.

Yes, I think we need to extract the QM for stationPressure since it is under ObsValue and we can extract it from the pressure QM.

Some for temperature. Since virtualTemperature and airTempreature come from TOB, and we have TOB QM. So, we should be able to get QM for virtualTemprature and airTemperature separately.

They are the same. the official name (IODA convention) is airTemperature

The third way, see example in bufr_ncep_mtiasi3.py

[emilyhcliu]

• [x] @emilyhcliu why your GSI plot for stationPressure is different from UFO? Please check my GSI plot here: Use Python API to convert ADPUPA prepbufr pressure data to IODA and check if this can be used with UFO seamlessly #71
• [x] do we need two pressure QM variables (pressureQM and stationPressureQM) written using the same PQM variable? I think one, pressureQM is sufficient. What you think?
• [x] Same for temperature: do we need two temperature QM variables (airTemperatureQM and virtualTemperatureQM) written using the same TQM variable? I think one, airTemperatureQM is sufficient. What you think?
• [x] what is the difference between airTemperature and sensibleTemperature?
• [x] in feature/amv_goes you mentioned three ways to encode BUFR data to IODA, which one you wanted to be in my Python API. I am currently using one that Ron mentioned in his API example.

Please see my answers above.

[PraveenKumar-NOAA]

@emilyhcliu thank you!

[emilyhcliu]

Also, if the sensible temperature and virtual temperature are coming from TOB for different TPC, then the virtual temperature is not used correctly in ADPUPA prepbufr YAML.

@PraveenKumar-NOAA You are correct. TVO is not the virtual temperature used by GSI.
To do it correctly with YAML, we need to modify C++ code to do that. But, since we have python version, we will use the python one.

For the YAML, You can just remove the TVO and the related virtual temperature entries. Then, add the TPC in YAML. So, in the YAML, there will be only airTemperature and TVO (virualTemperatureEventProgramCode) in the YAML. Users can still use the YAML to generate IODA from prepbufr, but they will need to generate virtual temperature from TPC in UFO themselves.

[PraveenKumar-NOAA]

@emilyhcliu thank you for the clarification.

[PraveenKumar-NOAA]

@emilyhcliu for temperatures, extracted using the TPC:

1. TPC=1 ---> TOB is the sensible temperature
2. TPC<= 8 and TPC > 1 ---> TOB is the virtual temperature

the second condition above also extracts sensible temperature, is this a valid conditional statement for virtual temperature?

[emilyhcliu]

For the second condition, TPC = 8 is virtual temperature and TPC=1 is sensible temperature.
There are 13 points between 2 and 7, mostly 5, and 6. I am not sure they should be sensible or virtual.
We can check together tomorrow.

[emilyhcliu]

@PraveenKumar-NOAA I udpated plot for the ADPUPA stationPressure from GSI (GsiHofXBc) Now the UFO and GSI results for stationPressure look comparable.

[PraveenKumar-NOAA]

@emilyhcliu thanks.

[PraveenKumar-NOAA]

@emilyhcliu do I need to compare Python version of IODA with bufr2ioda IODA? If yes, what I need to look for. I have created an ioda-converters PR for this work.

[PraveenKumar-NOAA]

@emilyhcliu I run end-to-end HofX tests using my prepbufr API (json version) and created the validation plots for the following observations:

- windEastward - windNorthward - airTemperature - virtualTemperature - specificHumidity

These plots look great and are in perfect agreement with the plots that were created using the old version of API.

[PraveenKumar-NOAA]

@emilyhcliu and finally the stationPressure plot: