Add use of high resolution Raob data

[jderber-NOAA]

The WMO has created a new standard for the reporting of raob data. This allows for the more precise encoding of the data and the reporting of many more levels. Initially we are bypassing the prepbufr file to use this data and using file uprair.

The contents of the uprair file were given to me by Chris Hill and repeated below.

SINGULAR FIELDS

BUFR FIELD UNITS DESCRIPTION 001001 WMOB NUMERIC WMO BLOCK NUMBER
001002 WMOS NUMERIC WMO STATION NUMBER
001006 ACID ( 8)CCITT IA5 AIRCRAFT FLIGHT NUMBER
001011 SMID ( 9)CCITT IA5 SHIP OR MOBILE LAND STATION IDENTIFIER
001081 RSERL (20)CCITT IA5 RADIOSONDE SERIAL NUMBER
001082 RASCN NUMERIC RADIOSONDE ASCENSION NUMBER
001083 RRLSE NUMERIC Radiosonde release number
001125 WGOSIDS NUMERIC WIGOS IDENTIFIER SERIES
001126 WGOSISID NUMERIC WIGOS ISSUER OF IDENTIFIER
001127 WGOSISNM NUMERIC WIGOS ISSUE NUMBER
001128 WGOSLID (16)CCITT IA5 WIGOS LOCAL IDENTIFIER (CHARACTER)
001198 RPID ( 8)CCITT IA5 REPORT IDENTIFIER
002003 A4ME CODE TABLE TYPE OF MEASURING EQUIPMENT USED
002011 RATP CODE TABLE RADIOSONDE TYPE
002013 SIRC CODE TABLE SOLAR AND INFRARED RADIATION CORRECTION
002014 TTSS CODE TABLE TRACKING TECHNIQUE/STATUS OF SYSTEM USED
002015 RACP CODE TABLE Radiosonde completeness
002016 RCONF FLAG TABLE Radiosonde configuration
002017 CAHM CODE TABLE Correction algorithms for humidity measurements 002066 RGRSY CODE TABLE Radiosonde ground receiving system
002067 RFREQ HZ RADIOSONDE OPERATING FREQUENCY
002080 BMFGR CODE TABLE Balloon manufacturer
002081 BTYPE CODE TABLE Type of balloon
002082 BWGHT KG Weight of balloon
002083 BSHEL CODE TABLE Type of balloon shelter
002084 BGTYP CODE TABLE Type of gas used in balloon
002085 BGAMT KG Amount of gas used in balloon
002086 BFTLN M Balloon flight train length
002095 PSENS CODE TABLE TYPE OF PRESSURE SENSOR
002096 TSENS CODE TABLE TYPE OF TEMPERATURE SENSOR
002097 RHSENS CODE TABLE TYPE OF HUMIDITY SENSOR
002191 GHTC CODE TABLE Geopotential height calculation
004001 YEAR YEAR YEAR
004002 MNTH MONTH MONTH
004003 DAYS DAY DAY
004004 HOUR HOUR HOUR
004005 MINU MINUTE MINUTE
004006 SECO SECOND SECOND
004200 RCYR YEAR YEAR - TIME OF RECEIPT 004201 RCMO MONTH MONTH - TIME OF RECEIPT 004202 RCDY DAY DAY - TIME OF RECEIPT 004203 RCHR HOUR HOUR - TIME OF RECEIPT 004204 RCMI MINUTE MINUTE - TIME OF RECEIPT 005001 CLATH DEGREE(N+,S-) LATITUDE (HIGH ACCURACY) 006001 CLONH DEGREE(E+,W-) LONGITUDE (HIGH ACCURACY) 007007 HEIT METERS HEIGHT 007030 HSMSL M HEIGHT OF STATION GROUND ABOVE MEAN SEA LEVEL 007031 HBMSL M HEIGHT OF BAROMETER ABOVE MEAN SEA LEVEL 008002 VSSO CODE TABLE VERTICAL SIGNIFICANCE (SURFACE OBSERVATIONS) 008021 TSIG CODE TABLE TIME SIGNIFICANCE 008202 RCTS CODE TABLE RECEIPT TIME SIGNIFICANCE 011061 AWSB METERS/SECOND ABSOLUTE WIND SHEAR IN 1 KM LAYER BELOW 011062 AWSA METERS/SECOND ABSOLUTE WIND SHEAR IN 1 KM LAYER ABOVE 020011 CLAM CODE TABLE CLOUD AMOUNT 020012 CLTP CODE TABLE CLOUD TYPE 020013 HOCB M HEIGHT OF BASE OF CLOUD 022043 SST1 DEGREES KELVIN SEA/WATER TEMPERATURE 025061 SOFTV (12)CCITT IA5 Software identification and version number 033215 CORN CODE TABLE CORRECTED REPORT INDICATOR 035021 BUHD ( 6)CCITT IA5 BULLETIN BEING MONITORED (TTAAii) 035022 BULTIM ( 6)CCITT IA5 BULLETIN BEING MONITORED (YYGGgg) 035023 BORG ( 4)CCITT IA5 BULLETIN BEING MONITORED (CCCC) 035035 RTERM CODE TABLE Reason for termination 035194 BBB ( 6)CCITT IA5 BULLETIN BEING MONITORED (BBB) 035195 SEQNUM ( 4)CCITT IA5 CHANNEL SEQUENCE NUMBER

REPLICATED FIELDS

BUFR FIELD UNITS DESCRIPTION 004086 LTDS SECOND LONG TIME PERIOD OR DISPLACEMENT
005015 LATDH DEGREE LATITUDE DISPLACEMENT (HIGH ACCURACY)
006015 LONDH DEGREE LONGITUDE DISPLACEMENT (HIGH ACCURACY)
007004 PRLC PASCALS PRESSURE
007008 GP07 (METERS/SECOND)2 GEOPOTENTIAL
008042 VSIGX FLAG TABLE EXTENDED VERTICAL SOUNDING SIGNIFICANCE
010008 GP10 (METERS/SECOND)2 GEOPOTENTIAL
011001 WDIR DEGREES TRUE WIND DIRECTION
011002 WSPD METERS/SECOND WIND SPEED
012101 TMDB DEGREES KELVIN TEMPERATURE/DRY BULB TEMPERATURE
012103 TMDP DEGREES KELVIN DEW POINT TEMPERATURE

uprair_field_list_single-vs-replicate.txt

The main issues with doing this is the large number of levels that are reported for some sondes and the irregular quality of some of the profiles. Note some profiles are full, reporting all expected values, others are just repeats of what is in an old radiosonde or rawinsonde report and many parameters are missing. So there has to be a lot of checking for special cases. Another difficulty with this data is that we have to check to ensure that a particular observation is not used twice once through the prepbufr file and once through the uprair file.

[CoryMartin-NOAA]

FYI @CatherineThomas-NOAA and I are co-hosting a NOAA Hollings student this summer to help tackle a similar issue with AI/ML (more intelligent thinning of the profiles)

[jderber-NOAA]

Yes. Daryl mentioned about the Hollings student. This work should assist the student in getting going with the work. I was not going to spend much time on optimally specifying the error (or the combination of the observations into superobs) and will leave this for the student.

[jderber-NOAA]

First difficulty was to get the BUFR Library to read the new soundings. There were many (10000) levels in some of the soundings and the standard BUFR library would run out of memory. Jeff Ator suggested using the Dynamic Allocation (DA) version of the BUFR library and setting a couple of parameters before calling the first openbf. Russ helped with this and verified that the DA version reproduced the standard BUFR library. Use of this data allowed the reading of the data.

A new routine (read_hdraob) was defined to read the new data file and put them into the standard output file types produced by the other read routines. The new data was put under data types 119, 219 and a few under 218. The type 119 was use for temperature, q and ps. 219 was used for winds when pressure was available. 218 was used for a few profiles that only had height for a vertical location. These profiles appeared to be just pibals put into the new format.

The code was tested by producing a record for certain radiosondes, find the corresponding prepbufr radiosonde record and looking to see if the results were similar. They were not identical because the input values are not identical (e.g., Td rather than q, speed and direction rather than u,v), more accurate values in the input file and in some cases certain values were not available from the new file (e.g., quality marks, etc.) Default values were put in place of the missing values (e.g., qm = 0). Comparisons are below.

[jderber-NOAA]

For PS, the output record for each observation includes:

            cdata_all(1,iout)=poe                     ! surface pressure error (cb)
             cdata_all(2,iout)=dlon                    ! grid relative longitude
             cdata_all(3,iout)=dlat                    ! grid relative latitude

             cdata_all(4,iout)=plevs(k)                ! pressure (in cb)

             cdata_all(5,iout)=obsdat(4,k)/grav        ! surface height
             cdata_all(6,iout)=obsdat(7,k)             ! surface temperature
             cdata_all(7,iout)=rstation_id             ! station id
             cdata_all(8,iout)=t4dv                    ! time
             cdata_all(9,iout)=nc                      ! type
             cdata_all(10,iout)=pqm(k)                 ! quality mark
             cdata_all(11,iout)=obserr(1,k)*one_tenth  ! original obs error (cb)
             cdata_all(12,iout)=usage                  ! usage parameter
             cdata_all(13,iout)=idomsfc                ! dominate surface type
             cdata_all(14,iout)=dlon_earth_deg         ! earth relative longitude (degrees)
             cdata_all(15,iout)=dlat_earth_deg         ! earth relative latitude (degrees)
             cdata_all(16,iout)=stnelev                ! station elevation (m)
             cdata_all(17,iout)=zz                     ! terrain height at ob location
             cdata_all(18,iout)='88888888'             ! provider name
             cdata_all(19,iout)='HDRAOB'               ! subprovider name
             cdata_all(20,iout)=var_jb(1,k)            ! non linear qc b parameter

For a single observation ps record for one radiosonde (station 94170)

0.110000002384186 606.527850666667 331.373679464446 100.549999250844 19.9864377743674 299.850000000000 6.013561573914827E-154 2.25111111110697 2.00000000000000 0.000000000000000E+000 0.110000002384186 0.000000000000000E+000 1.00000000000000 141.920590000000 -12.6776000000000 21.0000000000000 24.1550097691471 7.117503049684752E-038 6.063266241113909E-154 0.000000000000000E+000

For same station from prepbufr file

0.110000000000000 606.525333333333 331.363432795501 100.600000000000 22.0000000000000 303.050000000000 6.013561573914827E-154 2.25000000000000 3.00000000000000 2.00000000000000 0.110000000000000 0.000000000000000E+000 1.00000000000000 141.920000000000 -12.6800000000000 22.0000000000000 24.1176010397446 7.117503049684752E-038 6.013471087152871E-154 0.000000000000000E+000

[jderber-NOAA]

Note that elements 9,10,19 are expected to be different.

9 -- is the location on convinfo file of observation type. These ps are given obs type 119 for the new data and typ 120 for prepbufr data.

10. -- is the quality mark. The quality mark comes from prepbufr file. For the new data is set with the read_routine (with different meaning).
11. A character string of 'HDRAOB' is added as the subprovider flag for the uprair data. For prepbufr missing.

[jderber-NOAA]

Most other numbers are close. Some small differences in the numbers are expected since reporting precision, reporting quantities and other details are different.

[jderber-NOAA]

For q observations, the output record contains,

            cdata_all(1,iout)=qoe                     ! q error
             cdata_all(2,iout)=dlon                    ! grid relative longitude
             cdata_all(3,iout)=dlat                    ! grid relative latitude
             cdata_all(4,iout)=dlnpob                  ! ln(pressure in cb)
             cdata_all(5,iout)=qobcon                  ! q ob
             cdata_all(6,iout)=rstation_id             ! station id
             cdata_all(7,iout)=t4dv+obsdat(1,k)*r60inv ! time
             cdata_all(8,iout)=nc                      ! type
             cdata_all(9,iout)=0.2_r_kind              ! q max error
             cdata_all(10,iout)=obsdat(7,k)            ! dry temperature (obs is tv)
             cdata_all(11,iout)=qqm(k)                 ! quality mark
             cdata_all(12,iout)=obserr(2,k)*one_tenth  ! original obs error
             cdata_all(13,iout)=usage                  ! usage parameter
             cdata_all(14,iout)=idomsfc                ! dominate surface type
             cdata_all(15,iout)=dlon_earth_deg         ! earth relative longitude (degrees)
             cdata_all(16,iout)=dlat_earth_deg         ! earth relative latitude (degrees)
             cdata_all(17,iout)=stnelev                ! station elevation (m)
             cdata_all(18,iout)=levdat(2,k)            ! observation height (m)
             cdata_all(19,iout)=zz                     ! terrain height at ob location
             cdata_all(20,iout)='88888888'             ! provider name
             cdata_all(21,iout)='HDRAOB'               ! subprovider name
             cdata_all(22,iout)=2                      ! cat
             cdata_all(23,iout)=var_jb(2,k)            ! non linear qc b parameter

For a single record at level 5 for station chosen so that the levels for the station are similar (the same number of level) (station 91815)

0.221079645071228 1287.05956266667 592.845841571361 4.52720863706780 1.599801542651064E-003 6.013589892468061E-154 4.26666666666667 27.0000000000000 0.200000000000000 264.350000000000 0.000000000000000E+000 0.200000000000000 0.000000000000000E+000 3.00000000000000 301.420210000000 48.5650500000000 57.0000000000000 702.523287769014 94.8129684438317 7.117503049684752E-038 6.063266241113909E-154 2.00000000000000 0.000000000000000E+000

prepbufr

0.220387185134036 1287.04633600000 592.792644267955 4.52720864451838 1.679000000000000E-003 6.013589892468061E-154 2.28571999073029 28.0000000000000 0.200000000000000 264.280332280232 2.00000000000000 0.200000000000000 0.000000000000000E+000 3.00000000000000 301.417110000000 48.5525900000000 60.0000000000000 703.000000000000 92.7809568407753 7.117503049684752E-038 6.013471087152871E-154 1.00000000000000 0.000000000000000E+000

The lat,lon,and time (2,3,7,15,16) are expected to be a little different because of the observed balloon drift that has been included. The qob (5) in the new file is converted from dew point temperature. Because of the balloon drift the surface information (14,15,16,17,19) differs. As with the surface pressure data, the convinfo type (8), the quality mark(11) and the subprovider name ('HDRAOB'). Cat also comes from the prepbufr file so it is set to 2.

In general the numbers are close and the code appears to be working properly.

[jderber-NOAA]

temperature observation

             cdata_all(1,iout)=toe                     ! temperature error
             cdata_all(2,iout)=dlon                    ! grid relative longitude
             cdata_all(3,iout)=dlat                    ! grid relative latitude
             cdata_all(4,iout)=dlnpob                  ! ln(pressure in cb)

             cdata_all(5,iout)=obsdat(7,k)             ! temperature ob.
             cdata_all(6,iout)=rstation_id             ! station id
             cdata_all(7,iout)=t4dv+obsdat(1,k)*r60inv ! time
             cdata_all(8,iout)=nc                      ! type
             cdata_all(9,iout)=qtflg                   ! qtflg (virtual temperature flag)
             cdata_all(10,iout)=tqm(k)                 ! quality mark
             cdata_all(11,iout)=obserr(3,k)            ! original obs error
             cdata_all(12,iout)=usage                  ! usage parameter
             cdata_all(13,iout)=idomsfc                ! dominate surface type
             cdata_all(14,iout)=tsavg                  ! skin temperature
             cdata_all(15,iout)=ff10                   ! 10 meter wind factor
             cdata_all(16,iout)=sfcr                   ! surface roughness
             cdata_all(17,iout)=dlon_earth_deg         ! earth relative longitude (degrees)
             cdata_all(18,iout)=dlat_earth_deg         ! earth relative latitude (degrees)
             cdata_all(19,iout)=stnelev                ! station elevation (m)
             cdata_all(20,iout)=levdat(2,k)            ! observation height (m)
             cdata_all(21,iout)=zz                     ! terrain height at ob location
             cdata_all(22,iout)='88888888'             ! provider name
             cdata_all(23,iout)='HDRAOB'               ! subprovider name
             cdata_all(24,iout)=2                      ! cat
             cdata_all(25,iout)=var_jb(3,k)            ! non linear qc for T

(Single record for station 91815)

0.994858376465778 1287.05956266667 592.845841571361 4.52720863706780 264.350000000000 6.013589892468061E-154 4.26666666666667 11.0000000000000 1.00000000000000 0.000000000000000E+000 0.899999976158142 0.000000000000000E+000 3.00000000000000 271.676317808503 1.00342750840921 0.000000000000000E+000 301.420210000000 48.5650500000000 57.0000000000000 702.523287769014 94.8129684438317 7.117503049684752E-038 7.117503049684752E-038 2.00000000000000 0.000000000000000E+000

Prepbufr

1.10193592567018 1287.04633600000 592.792644267955 4.52720864451838 264.550000000000 6.013589892468061E-154 2.28571999073029 12.0000000000000 0.000000000000000E+000 2.00000000000000 1.00000000000000 0.000000000000000E+000 3.00000000000000 272.228827921212 1.00249501981468 0.000000000000000E+000 301.417110000000 48.5525900000000 60.0000000000000 703.000000000000 92.7809568407753 7.117503049684752E-038 6.013471087152871E-154 1.00000000000000 0.000000000000000E+000

Similar comments to those for q. Note for temperature, the sensible temperature is reported.

[jderber-NOAA]

UV

             cdata_all(1,iout)=woe                     ! wind error
             cdata_all(2,iout)=dlon                    ! grid relative longitude
             cdata_all(3,iout)=dlat                    ! grid relative latitude
             cdata_all(4,iout)=dlnpob                  ! ln(pressure in cb)
             cdata_all(5,iout)=oelev                   ! height of observation
             cdata_all(6,iout)=uob                     ! u obs
             cdata_all(7,iout)=vob                     ! v obs
             cdata_all(8,iout)=rstation_id             ! station id
             cdata_all(9,iout)=t4dv+obsdat(1,k)*r60inv ! time
             cdata_all(10,iout)=nc                     ! type
             cdata_all(11,iout)=stnelev                ! station elevation
             cdata_all(12,iout)=wqm(k)                 ! quality mark
             cdata_all(13,iout)=obserr(5,k)            ! original obs error
             cdata_all(14,iout)=usage                  ! usage parameter
             cdata_all(15,iout)=idomsfc                ! dominate surface type
             cdata_all(16,iout)=tsavg                  ! skin temperature
             cdata_all(17,iout)=ff10                   ! 10 meter wind factor
             cdata_all(18,iout)=sfcr                   ! surface roughness
             cdata_all(19,iout)=dlon_earth_deg         ! earth relative longitude (degrees)
             cdata_all(20,iout)=dlat_earth_deg         ! earth relative latitude (degrees)     
             cdata_all(21,iout)=zz                     ! terrain height at ob location
             cdata_all(22,iout)='88888888'             ! provider name
             cdata_all(23,iout)='HDRAOB'               ! subprovider name
             cdata_all(24,iout)=2                      ! cat
             cdata_all(25,iout)=var_jb(5,k)            ! non linear qc parameter
             cdata_all(26,iout)=one                    ! hilbert curve weight, modified later

Again, one sample record at level 5 for station 71815

1.65809733803421 1287.05956266667 592.845841571361 4.52720863706780 702.523287769014 -9.79685510466092 4.77824550000094 6.013589892468061E-154 4.26666666666667 69.0000000000000 57.0000000000000 0.000000000000000E+000 1.50000000000000 0.000000000000000E+000 3.00000000000000 271.676317808503 1.00342750840921 0.000000000000000E+000 301.420210000000 48.5650500000000 94.8129684438317 7.117503049684752E-038 6.063266241113909E-154 2.00000000000000 0.000000000000000E+000

Prepbufr

1.68636172351236 1287.04633600000 592.792644267955 4.52720864451838 100000000000.000 -9.80000000000000 4.60000000000000 6.013589892468061E-154 2.28571999073029 70.0000000000000 60.0000000000000 2.00000000000000 1.50000000000000 0.000000000000000E+000 3.00000000000000 272.228827921212 1.00249501981468 0.000000000000000E+000 301.417110000000 48.5525900000000 92.7809568407753 7.117503049684752E-038 6.013471087152871E-154 1.00000000000000 0.000000000000000E+000

Similar comments as for t and q. Note that the height of the observation is missing in the prepbufr data.

[jderber-NOAA]

As another test, we will look at the fits to obs for types 120, 220, 221 versus the new data types 119, 218 and 219.

ps o-g 01 ps asm 119 0000 223 0.111E+00 0.841E+00 0.318E+00 0.316E+00 o-g 01 ps asm 120 0000 577 0.395E-01 0.108E+01 0.442E+00 0.423E+00

t o-g ptop 0.100E+04 0.900E+03 0.800E+03 0.600E+03 0.400E+03 0.300E+03 0.250E+03 0.200E+03 0.150E+03 0.100E+03 0.500E+02 0.000E+00 o-g it obs use typ styp pbot 0.120E+04 0.100E+04 0.900E+03 0.800E+03 0.600E+03 0.400E+03 0.300E+03 0.250E+03 0.200E+03 0.150E+03 0.100E+03 0.200E+04

o-g 01 t asm 119 0000 count 1409 17655 22508 56699 77193 53101 32206 38772 49959 67582 100103 696545 o-g 01 t asm 119 0000 bias 0.426E+00 0.293E+00 0.412E+00 0.375E+00 0.504E+00 0.222E+00 0.137E+00 -0.103E+00 -0.496E-01 -0.101E+00 0.365E-02 0.186E+00 o-g 01 t asm 119 0000 rms 0.220E+01 0.178E+01 0.197E+01 0.148E+01 0.146E+01 0.126E+01 0.129E+01 0.162E+01 0.151E+01 0.127E+01 0.149E+01 0.150E+01 o-g 01 t asm 119 0000 cpen 0.417E+00 0.319E+00 0.340E+00 0.119E+00 0.707E-01 0.414E-01 0.203E-01 0.291E-01 0.345E-01 0.386E-01 0.366E-01 0.595E-01 o-g 01 t asm 119 0000 qcpen 0.389E+00 0.299E+00 0.295E+00 0.109E+00 0.665E-01 0.408E-01 0.200E-01 0.289E-01 0.334E-01 0.379E-01 0.346E-01 0.556E-01 o-g 01 t asm 120 0000 count 724 1666 1910 3279 3467 2808 959 1599 1638 2477 2827 29165 o-g 01 t asm 120 0000 bias 0.346E+00 0.134E+00 0.170E+00 0.193E+00 0.972E-01 -0.420E-01 -0.189E+00 -0.103E-01 0.772E-01 0.594E-01 -0.273E-01 0.105E+00 o-g 01 t asm 120 0000 rms 0.225E+01 0.214E+01 0.176E+01 0.121E+01 0.942E+00 0.956E+00 0.130E+01 0.125E+01 0.132E+01 0.123E+01 0.199E+01 0.156E+01 o-g 01 t asm 120 0000 cpen 0.680E+00 0.121E+01 0.161E+01 0.117E+01 0.725E+00 0.743E+00 0.790E+00 0.611E+00 0.789E+00 0.117E+01 0.220E+01 0.112E+01 o-g 01 t asm 120 0000 qcpen 0.653E+00 0.111E+01 0.145E+01 0.111E+01 0.701E+00 0.697E+00 0.748E+00 0.604E+00 0.777E+00 0.114E+01 0.201E+01 0.107E+01

current fit of q data, units in per-cent of guess q-sat o-g ptop 0.100E+04 0.950E+03 0.900E+03 0.850E+03 0.800E+03 0.700E+03 0.600E+03 0.500E+03 0.400E+03 0.300E+03 0.000E+00 0.000E+00 o-g it obs use typ styp pbot 0.120E+04 0.100E+04 0.950E+03 0.900E+03 0.850E+03 0.800E+03 0.700E+03 0.600E+03 0.500E+03 0.400E+03 0.300E+03 0.200E+04

o-g 01 q asm 119 0000 count 1409 7942 9714 10578 11930 26291 30408 35505 41688 53096 110800 339361 o-g 01 q asm 119 0000 bias 0.389E+00 -0.168E+01 -0.188E+01 -0.293E+01 -0.899E+00 -0.248E+01 -0.535E+01 -0.857E+01 -0.108E+02 -0.147E+02 -0.500E+01 -0.704E+01 o-g 01 q asm 119 0000 rms 0.132E+02 0.139E+02 0.143E+02 0.180E+02 0.194E+02 0.195E+02 0.223E+02 0.222E+02 0.220E+02 0.216E+02 0.105E+02 0.181E+02 o-g 01 q asm 119 0000 cpen 0.572E-01 0.563E-01 0.403E-01 0.531E-01 0.470E-01 0.387E-01 0.399E-01 0.329E-01 0.249E-01 0.188E-01 0.359E-02 0.232E-01 o-g 01 q asm 119 0000 qcpen 0.570E-01 0.562E-01 0.403E-01 0.509E-01 0.456E-01 0.371E-01 0.364E-01 0.307E-01 0.243E-01 0.186E-01 0.358E-02 0.223E-01 o-g 01 q asm 120 0000 count 723 604 1062 1207 703 1947 1322 1404 1982 1937 0 12891 o-g 01 q asm 120 0000 bias 0.264E+01 0.148E+00 0.501E+00 0.231E+01 0.245E+01 -0.106E+00 -0.888E+00 0.453E+00 -0.170E+00 0.315E+00 0.000E+00 0.510E+00 o-g 01 q asm 120 0000 rms 0.161E+02 0.186E+02 0.144E+02 0.161E+02 0.173E+02 0.164E+02 0.178E+02 0.169E+02 0.173E+02 0.188E+02 0.000E+00 0.171E+02 o-g 01 q asm 120 0000 cpen 0.128E+00 0.258E+00 0.214E+00 0.298E+00 0.333E+00 0.324E+00 0.362E+00 0.344E+00 0.332E+00 0.271E+00 0.000E+00 0.298E+00 o-g 01 q asm 120 0000 qcpen 0.126E+00 0.232E+00 0.214E+00 0.289E+00 0.331E+00 0.314E+00 0.349E+00 0.341E+00 0.319E+00 0.255E+00 0.000E+00 0.288E+00

o-g ptop 0.100E+04 0.900E+03 0.800E+03 0.600E+03 0.400E+03 0.300E+03 0.250E+03 0.200E+03 0.150E+03 0.100E+03 0.500E+02 0.000E+00 o-g it obs use typ styp pbot 0.120E+04 0.100E+04 0.900E+03 0.800E+03 0.600E+03 0.400E+03 0.300E+03 0.250E+03 0.200E+03 0.150E+03 0.100E+03 0.200E+04

o-g 01 uv asm 218 0000 count 5 48 39 81 64 46 34 33 42 57 90 692 o-g 01 uv asm 218 0000 bias 0.536E+01 0.631E-01 -0.141E+01 -0.169E+01 -0.118E+01 -0.768E+00 -0.108E+01 0.167E+01 0.597E-01 -0.109E+01 0.876E+00 -0.291E+00 o-g 01 uv asm 218 0000 rms 0.594E+01 0.359E+01 0.549E+01 0.497E+01 0.492E+01 0.557E+01 0.605E+01 0.463E+01 0.468E+01 0.402E+01 0.424E+01 0.464E+01 o-g 01 uv asm 218 0000 cpen 0.239E+01 0.115E+01 0.243E+01 0.221E+01 0.141E+01 0.985E+00 0.952E+00 0.662E+00 0.788E+00 0.874E+00 0.546E+00 0.110E+01 o-g 01 uv asm 218 0000 qcpen 0.213E+01 0.110E+01 0.202E+01 0.195E+01 0.139E+01 0.983E+00 0.939E+00 0.662E+00 0.752E+00 0.874E+00 0.544E+00 0.103E+01 o-g 01 uv asm 219 0000 count 1392 17578 22318 56238 76794 52966 32158 38453 49677 67221 98490 689984 o-g 01 uv asm 219 0000 bias -0.203E+00 -0.724E-01 0.764E+00 0.593E+00 0.742E+00 -0.223E+00 0.298E+00 0.593E+00 0.735E+00 0.746E+00 0.735E+00 0.588E+00 o-g 01 uv asm 219 0000 rms 0.294E+01 0.397E+01 0.501E+01 0.543E+01 0.646E+01 0.768E+01 0.759E+01 0.678E+01 0.526E+01 0.511E+01 0.511E+01 0.567E+01 o-g 01 uv asm 219 0000 cpen 0.229E+00 0.315E+00 0.270E+00 0.163E+00 0.940E-01 0.569E-01 0.432E-01 0.484E-01 0.344E-01 0.362E-01 0.397E-01 0.692E-01 o-g 01 uv asm 219 0000 qcpen 0.224E+00 0.285E+00 0.235E+00 0.143E+00 0.780E-01 0.502E-01 0.399E-01 0.440E-01 0.332E-01 0.351E-01 0.377E-01 0.623E-01 o-g 01 uv asm 220 0000 count 670 1921 1773 3203 3099 2532 935 1682 1803 3016 4642 34767 o-g 01 uv asm 220 0000 bias -0.473E+00 0.551E+00 0.741E+00 0.472E+00 0.524E+00 0.379E+00 0.489E+00 0.514E+00 0.549E+00 0.642E+00 0.680E+00 0.493E+00 o-g 01 uv asm 220 0000 rms 0.319E+01 0.368E+01 0.342E+01 0.365E+01 0.384E+01 0.403E+01 0.448E+01 0.443E+01 0.460E+01 0.495E+01 0.515E+01 0.458E+01 o-g 01 uv asm 220 0000 cpen 0.751E+00 0.130E+01 0.131E+01 0.133E+01 0.908E+00 0.581E+00 0.529E+00 0.594E+00 0.844E+00 0.125E+01 0.146E+01 0.106E+01 o-g 01 uv asm 220 0000 qcpen 0.703E+00 0.119E+01 0.126E+01 0.128E+01 0.888E+00 0.574E+00 0.526E+00 0.590E+00 0.827E+00 0.120E+01 0.140E+01 0.101E+01 o-g 01 uv asm 221 0000 count 31 221 139 152 65 36 6 9 11 15 3 692 o-g 01 uv asm 221 0000 bias -0.108E+01 0.445E+00 0.397E+00 0.202E+00 -0.494E+00 0.220E+00 0.908E+00 0.222E+01 0.722E+00 0.871E+00 0.402E+01 0.329E+00 o-g 01 uv asm 221 0000 rms 0.273E+01 0.336E+01 0.384E+01 0.386E+01 0.493E+01 0.532E+01 0.288E+01 0.456E+01 0.464E+01 0.601E+01 0.680E+01 0.409E+01 o-g 01 uv asm 221 0000 cpen 0.113E+01 0.160E+01 0.177E+01 0.160E+01 0.169E+01 0.120E+01 0.271E+00 0.648E+00 0.983E+00 0.228E+01 0.336E+01 0.164E+01 o-g 01 uv asm 221 0000 qcpen 0.110E+01 0.150E+01 0.163E+01 0.150E+01 0.162E+01 0.107E+01 0.271E+00 0.648E+00 0.983E+00 0.222E+01 0.336E+01 0.153E+01

Note that the rms fits for the new data tend to be a little better than the prepbufr files. Of course these are not the same sets of data, but there is considerable overlap. Also note that for 119 and 219 there are many more observations because of the density of the observations in the vertical. The penalty terms are smaller for the new data because the obs errors have been modified so that the average error in a layer is equal to the specified error.

[jderber-NOAA]

As mentioned before, the quality marks are not set in new data set. Thus, the default quality mark is set to 0. The following conditions result in the quality mark being changed.

uvob: Set to 1. Height not from direct measurement.

uvob,tob,qob,psob: Set to 2. Change in time missing.

uvob,tob,qob,psob: Set to 2. Change in lat/long missing.

tob: Set to 12 (don't use). Temperature < 100. or Temperature > 400.

uvob: Set to 12(don't use). Direction > 360. or < 0. .or. Speed < 0. or Speed > 300.

[jderber-NOAA]

The next step was to ensure that profiles are not used twice since many are in both the uprair file and the prepbufr file. To do this, a list of station numbers was saved for each observation type for those profiles that are used. These lists are then transferred to all processors. In setup(t),(q),(ps),and (w), the list is search for a matching station number for observation types 120, 220 and 221 If the new uprair station matches that from the prepbufr file, the profile from the prepbufr file is moved to being monitored. This assumes that the uprair profile is as good or better than that from the prepbufr file.

With these changes, the output for the appropriate removal of the prepbufr profiles to the monitored category.

At this point the preliminary version seems to be working and the branch (hdraob) is saved.

[jderber-NOAA]

Some stats concerning the contents of the contents of the input .

ps data

3/359 profiles with invalid station IDs.
8/698437 levels removed due to invalid pressures when determining ob error. 19/347 profiles removed due to invalid lat/long 60/328 with no time change for profile. (set to zero) 39/328 with no change in lat/long (set to zero) 0/328 with pressure too small.

q data 4/359 invalid station id 3/697494 invalid pressure in estimation of observation error. 613/341138 invalid change in time (set to zero) 576/341138 invalid change in lat/lon 1463/340824 invalid Td

t data

4/359 with invalid station number 3/697494 levels with an invalid pressure for profile when estimating errors 1383/697494 invalid change in time (set to zero) 1699/697494 invalid change in lat/long (set to zero) 944/697491 invalid pressures

uv 4/359 invalid station number 8/698437 invalid pressure when estimating ob error 1606/698409 invalid time change (set to zero) 1920/698409 invalid lat/lon change (set to zero) 7484/698409 invalid direction 12/690925 invalid speed

[jderber-NOAA]

Note by using the new bufrlib, there is a message printed everytime a bufr file is closed when it is not open. This was happening many times. Modification of the code was made to remove the messages. Also, closes were added to the files after the the closbf. This appears to have released some memory and it uses a bit less.

[jderber-NOAA]

---

Note revised figures can be found which include the descent data and have some fixed errors in the code are found in the attached presentation near the end of the issue.

---

Some plots comparing prepbufr data vs hd raob data. Note the large difference just above 100mb. If we plot the original data, the same difference shows up.

For q, currently cannot plot on same plot (for an unknown reason)

These plots contain all data, even if the use flag is not true. May wish to remove some of this data. For q in the high resolution data, q is not used if the temperature is less than 215K (and all points above). If you look at the temperature plot all points up to about 60mb will be used. For q in the prepbufr file, no data above 300mb is used. Maybe the same criterion should be used for the high resolution data?

[jderber-NOAA]

Updated to head of trunk. Contains restart changes (which were partly in this branch). So now regression test can be properly performed.

[jderber-NOAA]

Did regression test. Passed.

[RussTreadon-NOAA]

WCOSS_D test

Check out and install NOAA-EMC/GSI master at 39ed5fa and jderber-NOAA:master at 5d76654 on Venus. Run 2021041300 gdas case using two global_gsi.x executables. 5d76654 global_gsi.x generates b4b results with respect from 39ed5fa global_gsi.x. Wall times from two runs are comparable. 5d76654 ran about 69 seconds longer than 39ed5fa. Another run of 5d76654 might yield a wall time closer to 39ed5fa.

[jderber-NOAA]

Daryl asked me to look at using the RAOB descent data. It turns out this data was also in the same data file. In fact the descent data was being misused as high resolution radiosonde data as ascent data. There was no real issue with using the data this way except it was not separated the data was not given a different type. Two new types for the descent data 118 (for temp, and q) and 217 for uv. No surface pressure data was created from the descent data. Note that it is assumed that there are no uv descent observations with only height (not pressure) reported.

Usually the descent data has a GTS bulletin header of IUD or IUX. However it was found that it was more reliable to use the change in time as a better measure of whether it is ascent or descent data. For ascent data, the time starts at the launch of the radiosonde. For descent data, the time starts when the balloon bursts. The data for both is ordered from higher pressure to lower pressure. Thus, if the time of the first observation is greater than the last observation, it is a descent observation. A message is printed if the header is not consistent with a descent or ascent observation. If no change in time is available it is assumed to be a ascent profile.

Changes to allow the use of this data were incorporated and tested. The code which separates out this data into types 118 and 217. Appears to be working properly.

[jderber-NOAA]

A presentation with revised figures including descent data and with a few errors fixed are found in the attached presentation Inclusion of high resolution radiosonde data.pptx