Open PraveenKumar-NOAA opened 1 year ago
The bufr -> ioda -> hofx plumbing worked well for the following ADPUPA observations using Python API:
The validation plots are shown below. The results from running GDASApp for ~/parm/atm/obs/testing/sondes.yaml (no QC) are also shown with these plots, which are very similar and are in agreement:
- windEastward
- windNorthward
- airTemperature
- virtualTemperature
- specificHumidity
ObsValue vs hofx plots with quality marks for the following observations:
windEastward windNorthward airTemperature virtualTemperature specificHumidity
Mostly observations belong to the QM 0-3.
@emilyhcliu bufr -> ioda -> hofx plumbing work is completed for ADPUPA prepbufr and the hofx/GSIhofx/ObsVlaue plots (windEastward, windNorthward, airTemperature, virtualTemperature, specificHumidity) turned out very well. Let me know what you think?
@emilyhcliu verified results again. Looks everything good.
@PraveenKumar-NOAA Great! Thanks. Based on our discussion this morning, let's do the following: (1) make stationPressure the same dimension as the rest of the variables; this can be done by using where function for numpy array (2) extract QM mark for stationPressure.
I modified your python script slightly. Here are my modifications: (1) Add one more BUFR Mnemonic called TPC (Temperature Event Program Code)
(2) Create airTemperature and virtualTemperature from TOB based on TPC (3) Create stationPressure based on POB (with the same dimension: Location)
You add TVO as virtual temperature, but TVO is not the virtual temperature used by GSI.
Both virtualTemperature and airTemperature are from TOB
TPC is set to tell if TOB is virtual or sensible temperature.
Here is the code table for TPC
0-12-247 - TPC
PREPBUFR: Temperature (TOB) event program code
0 | Reserved
1 | Initial PREPBUFR processing step "PREPRO" (performed in PREPOBS_PREPDATA program, prior to "PREVENT" and "VIRTMP" steps)
2 | Synthetic tropical cyclone bogus processing step "SYNDATA" (performed in SYNDAT_SYNDATA program, prior to "PREVENT" and "VIRTMP" steps)
3 | Reserved
4 | Pre-quality control step "PREVENT" which adds GFS forecast background and observation error (if present) and performs check of surface pressure
5 | Rawinsonde height/temperature complex quality control step "CQCHT" (performed in PREPOBS_CQCBUFR program, prior to "RADCOR" and "VIRTMP" steps)
6 | Rawinsonde height/temperature intersonde bias (radiation) correction step "RADCOR" (performed in PREPOBS_CQCBUFR program, after "CQCHT" step but prior to "VIRTMP" step)
7 | AIREP, PIREP and AMDAR aircraft quality control step "PREPACQC" (performed in PREOPOBS_PREPACQC program)
8 | Virtual temperature/specific humidity processing step "VIRTMP"
9 | Wind profiler quality control step "CQCPROF" (performed in PREPOBS_PROFCQC program)
10 | Multi-platform OI-quality control step "OIQC" (performed in PREPOBS_OIQCBUFR program)
11 | Global (GFS/GDAS/CDAS) SSI analysis step "SSI" (performed in GLOBAL_SSI and CDAS_SSI programs)
12 | VAD wind quality control step "CQCVAD" (performed in PREPOBS_CQCVAD program)
13 | Regional (ETA/EDAS) 3DVAR analysis step "R3DVAR" (performed in ETA_R3DVAR program)
14 | ACARS aircraft quality control step "ACARSQC" (performed in PREPOBS_ACARSQC program)
15-30 | Reserved
31 | Missing value
I ran the hofx test with the IODA obs file with my modifications. The results are very much like yours. Here are the plots for your reference:
UFO airTemperature | GSI airTemperature |
---|---|
UFO virtualTemperature | GSI virtualTemperature |
---|---|
UFO stationPressure | GSI stationPressure |
---|---|
UFO windEastward | GSI windEastward |
---|---|
UFO windNorthward | GSI windNorthward |
---|---|
UFO specificHumidity | GSI specificHumidity |
---|---|
Here are a few things to do:
@emilyhcliu thank you, I will update my Python API for this change.
[x] @emilyhcliu why your GSI plot for stationPressure is different from UFO? Please check my GSI plot here: https://github.com/NOAA-EMC/JEDI-T2O/issues/71
[x] do we need two pressure QM variables (pressureQM and stationPressureQM) written using the same PQM variable? I think one, pressureQM is sufficient. What you think?
[x] Same for temperature: do we need two temperature QM variables (airTemperatureQM and virtualTemperatureQM) written using the same TQM variable? I think one, airTemperatureQM is sufficient. What you think?
[x] what is the difference between airTemperature and sensibleTemperature?
[x] in feature/amv_goes you mentioned three ways to encode BUFR data to IODA, which one you wanted to be in my Python API. I am currently using one that Ron mentioned in his API example.
Also, if the sensible temperature and virtual temperature are coming from TOB for different TPC, then the virtual temperature is not used correctly in ADPUPA prepbufr YAML.
@PraveenKumar-NOAA The observation operator for surface pressure is different in GSI and UFO. The UFO (PCorrecter) performs better (closer to observation), because the operator corrects the calculated station pressure with observation.
Yes, I think we need to extract the QM for stationPressure since it is under ObsValue and we can extract it from the pressure QM.
Some for temperature. Since virtualTemperature and airTempreature come from TOB, and we have TOB QM. So, we should be able to get QM for virtualTemprature and airTemperature separately.
They are the same. the official name (IODA convention) is airTemperature
The third way, see example in bufr_ncep_mtiasi3.py
- [x] @emilyhcliu why your GSI plot for stationPressure is different from UFO? Please check my GSI plot here: Use Python API to convert ADPUPA prepbufr pressure data to IODA and check if this can be used with UFO seamlessly #71
- [x] do we need two pressure QM variables (pressureQM and stationPressureQM) written using the same PQM variable? I think one, pressureQM is sufficient. What you think?
- [x] Same for temperature: do we need two temperature QM variables (airTemperatureQM and virtualTemperatureQM) written using the same TQM variable? I think one, airTemperatureQM is sufficient. What you think?
- [x] what is the difference between airTemperature and sensibleTemperature?
- [x] in feature/amv_goes you mentioned three ways to encode BUFR data to IODA, which one you wanted to be in my Python API. I am currently using one that Ron mentioned in his API example.
Please see my answers above.
@emilyhcliu thank you!
Also, if the sensible temperature and virtual temperature are coming from TOB for different TPC, then the virtual temperature is not used correctly in ADPUPA prepbufr YAML.
@PraveenKumar-NOAA You are correct. TVO
is not the virtual temperature used by GSI.
To do it correctly with YAML, we need to modify C++ code to do that.
But, since we have python version, we will use the python one.
For the YAML, You can just remove the TVO and the related virtual temperature entries. Then, add the TPC in YAML. So, in the YAML, there will be only airTemperature and TVO (virualTemperatureEventProgramCode) in the YAML. Users can still use the YAML to generate IODA from prepbufr, but they will need to generate virtual temperature from TPC in UFO themselves.
@emilyhcliu thank you for the clarification.
@emilyhcliu for temperatures, extracted using the TPC:
the second condition above also extracts sensible temperature, is this a valid conditional statement for virtual temperature?
For the second condition, TPC = 8 is virtual temperature and TPC=1 is sensible temperature.
There are 13 points between 2 and 7, mostly 5, and 6.
I am not sure they should be sensible or virtual.
We can check together tomorrow.
@PraveenKumar-NOAA I udpated plot for the ADPUPA stationPressure from GSI (GsiHofXBc) Now the UFO and GSI results for stationPressure look comparable.
@emilyhcliu thanks.
@emilyhcliu do I need to compare Python version of IODA with bufr2ioda IODA? If yes, what I need to look for. I have created an ioda-converters PR for this work.
@emilyhcliu I run end-to-end HofX tests using my prepbufr API (json version) and created the validation plots for the following observations:
- windEastward - windNorthward - airTemperature - virtualTemperature - specificHumidity
These plots look great and are in perfect agreement with the plots that were created using the old version of API.
@emilyhcliu and finally the stationPressure plot:
Configuration
Test using fv3jedi_hofx_nomodel.x with the IODA files converted from BUFR converter
Check List
Investigate diag output from UFO