Temporal and spatial accuracy when overlay aircraft data in a tile plot - testing VERDI 2.1.4 20221109 builds

[yadongxuEPA]

Describe the bug Test VERDI 2.1.4 20221109 builds on Atmos When overlay aircraft data as observations in a tile plot for model data, the observational timestamps and variable values do not match up with the ones in model data.

To Reproduce Steps to reproduce the behavior:

1. Launch VERDI GUI
2. Load two datasets: model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110729” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” from /work/MOD3DEV/dkj/Aircraft
3. Select "NO2" or "O3" from variables list of the above model data and click "Tile Plot"
4. In "Tile Plot", click on Plot >> Add overlay >> Observations
5. In the pop-up window, select the corresponding variables ("NO2" or "O3") from the aircraft data's variables list
6. Keep other settings in the pop-up window as default (Stroke Size == 1; Shape Size ==8; Symbol == CIRCLE)
7. Click "Add variable", then click "OK"

Expected behavior The aircraft observations should be displayed as one or more circles at the timestamps when they are available and those circles' fill color should represent the selected variable value in the aircraft data.

Screenshots Animated tile plot when overlay O3 shown as below:

Animated tile plot when overlay NO2 shown as below:

Additional context With Daiwen's example R script, I extracted the above aircraft data into a csv file and you can find it on Atmos under /work/MOD3APP/verdi/VERDI_2.1.4_testing_files/extracted_aircraft_data_in_csv

The columns in this csv file look like this: "timeon","datetime","timeoff","Latitude","Longitude","NO2_ppbv","O3_ppbv" 2011-07-28 13:56:08,2011-07-28 13:56:13,2011-07-28 13:56:18,39.0855,-76.7542,-9999,-9999 2011-07-28 13:56:18,2011-07-28 13:56:23,2011-07-28 13:56:28,39.0851,-76.7542,-9999,-9999 2011-07-28 13:56:28,2011-07-28 13:56:33,2011-07-28 13:56:38,39.0851,-76.7543,-9999,-9999 2011-07-28 13:56:38,2011-07-28 13:56:43,2011-07-28 13:56:48,39.0851,-76.7547,-9999,-9999 2011-07-28 13:56:48,2011-07-28 13:56:53,2011-07-28 13:56:58,39.0853,-76.7575,-9999,-9999 2011-07-28 13:56:58,2011-07-28 13:57:03,2011-07-28 13:57:08,39.0855,-76.7624,-9999,-9999 2011-07-28 13:57:08,2011-07-28 13:57:13,2011-07-28 13:57:18,39.0858,-76.7683,-9999,-9999 2011-07-28 13:57:18,2011-07-28 13:57:23,2011-07-28 13:57:28,39.0862,-76.7746,-9999,-9999 2011-07-28 13:57:28,2011-07-28 13:57:33,2011-07-28 13:57:38,39.0865,-76.7814,-9999,-9999 2011-07-28 13:57:38,2011-07-28 13:57:43,2011-07-28 13:57:48,39.0869,-76.7885,-9999,-9999 2011-07-28 13:57:48,2011-07-28 13:57:53,2011-07-28 13:57:58,39.0872,-76.7957,-9999,-9999 2011-07-28 13:57:58,2011-07-28 13:58:03,2011-07-28 13:58:08,39.0854,-76.8023,-9999,-9999 2011-07-28 13:58:08,2011-07-28 13:58:13,2011-07-28 13:58:18,39.0811,-76.8066,-9999,-9999 2011-07-28 13:58:18,2011-07-28 13:58:23,2011-07-28 13:58:28,39.0755,-76.8065,-9999,-9999 2011-07-28 13:58:28,2011-07-28 13:58:33,2011-07-28 13:58:38,39.0707,-76.8025,-9999,55.89 2011-07-28 13:58:38,2011-07-28 13:58:43,2011-07-28 13:58:48,39.0674,-76.7963,-9999,56.64 2011-07-28 13:58:48,2011-07-28 13:58:53,2011-07-28 13:58:58,39.0642,-76.7902,-9999,57.42 2011-07-28 13:58:58,2011-07-28 13:59:03,2011-07-28 13:59:08,39.0609,-76.7841,-9999,58.12 2011-07-28 13:59:08,2011-07-28 13:59:13,2011-07-28 13:59:18,39.0574,-76.778,-9999,58.69 2011-07-28 13:59:18,2011-07-28 13:59:23,2011-07-28 13:59:28,39.0539,-76.7718,-9999,59.29 2011-07-28 13:59:28,2011-07-28 13:59:33,2011-07-28 13:59:38,39.0502,-76.7657,-9999,60.06 2011-07-28 13:59:38,2011-07-28 13:59:43,2011-07-28 13:59:48,39.0462,-76.7597,-9999,61.11 2011-07-28 13:59:48,2011-07-28 13:59:53,2011-07-28 13:59:58,39.0427,-76.7533,-9999,61.63 2011-07-28 13:59:58,2011-07-28 14:00:03,2011-07-28 14:00:08,39.0401,-76.746,-9999,62.21 2011-07-28 14:00:08,2011-07-28 14:00:13,2011-07-28 14:00:18,39.0376,-76.7387,-9999,62.88 2011-07-28 14:00:18,2011-07-28 14:00:23,2011-07-28 14:00:28,39.0351,-76.7314,-9999,62.94 2011-07-28 14:00:28,2011-07-28 14:00:33,2011-07-28 14:00:38,39.0325,-76.7241,-9999,63.02 2011-07-28 14:00:38,2011-07-28 14:00:43,2011-07-28 14:00:48,39.0297,-76.7168,-9999,63.31 2011-07-28 14:00:48,2011-07-28 14:00:53,2011-07-28 14:00:58,39.0269,-76.7095,-9999,63.34 2011-07-28 14:00:58,2011-07-28 14:01:03,2011-07-28 14:01:08,39.0243,-76.702,-9999,63.8 2011-07-28 14:01:08,2011-07-28 14:01:13,2011-07-28 14:01:18,39.0232,-76.6939,-9999,64.1 2011-07-28 14:01:18,2011-07-28 14:01:23,2011-07-28 14:01:28,39.0237,-76.6854,-9999,64.22 ...

1) We can see that there is a red filled circle and a few black filled circles showed up in NJ (under PA). They are persistent from the first timestamp (July 29 2011 00:00:00) to the last timestamp (July 29 2011 23:00:00) of the model data; however, the actual aircraft data only covers 2011-07-28 13:56:08 to 2011-07-28 15:56:43. 2) Notice that the NO2 and O3 values in aircraft data are in "ppbv", but the units in model data are "ppmV" , the filled circle should not be red if it displays the instantaneous values. We need to look into why other circles are filled with black color (Is it how the code handle "-9999" ?)

[yadongxuEPA]

Daiwen,

Can you confirm if the timeStamps in the aircraft data is also in UTC ? I did set the "time.vars" to "UTC" before runing read.ICARTT() function. Thanks, Yadong @dkang2

[dkang2]

@yadongxuEPA

The time stamps in the aircraft data are the seconds after midnight "UTC". I have R code to work with it.

[yadongxuEPA]

There is another issue we need to pay attention to when overlay aircraft data as observations in a tile plot for model data, that is to figure out which vertical layer a specific instantaneous aircraft observation should locate. Aircraft observations were taken at lower altitude (see below the Y-axis).

P3B_Discover-AQ_15s_CMAQ_Overlay_2011-07-28_O3.pdf They should not appear in the upper layers of the model data in a tile plot.

In this case, we need to deal with layer vs altitude matching. However, the altitude values for each layer are not straightforward from the WRF file itself. According to Daiwen's shared R script, the layer data need to be extracted from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/mcip/METCRO3D_20110711

[yadongxuEPA]

Tested VERDI 2.1.4 20221206 builds on Atmos. Found that some improvements have been made but the issue has not resolved, see the details as below: Test 1: Overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110729” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base Now VERDI can correctly match the timestamps and display an error message to remind user that "OBS data does not contain readings ..." (There is a typo in the error message "Observactional" should be "Observational" )

Test 2: Overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base When came to the timestamps (hour 13, 14, 15 UTC) when the aircraft Obs data was available, nothing displayed in the title plot but an error message popped-up as below:

[yadongxuEPA]

Tested VERDI 2.1.4 20230103 builds on Atmos. Found that some improvements have been made but the following issues need to be worked on: 1) Model data and aircraft data need to be matched hour by hour. For example, when overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base the aircraft Obs data was only available at hour 13, 14, 15 UTC, we need to pick out those observations for each hour to match with the model data at that hour. 2) We need to figure out which vertical layer each aircraft observation is located and only display them in the corresponding layer. In general, those overlaid circles should not appear in lower (for example, layer 1) layers, nor upper layers (for example, layer 34).

[yadongxuEPA]

Tested VERDI 2.1.4 20230201 builds on Atmos. Found that some improvements have been made in terms of timesteps matching between model data and aircraft data. Use “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” as an example, now only the aircraft measurements for a specific hour (hour 13, 14, 15 UTC) showed up in the overlaid tile plot.

We still need to work on vertical layer matching for each aircraft observation. For example, the aircraft data points should not show up in the tile plot when we switch to layer 35 shown as below.

[lizadams]

I was able to visualize the O3 data following Yadong's instructions above. However, if I tried to visualize both O3 and NO2, then VERDI seemed to either freeze or give a memory error.

./verdi.sh Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "TimerQueue" java.lang.OutOfMemoryError: GC overhead

Output from top shows high cpu and memory usage

PID    COMMAND      %CPU  TIME     #TH    #WQ  #PORT MEM    PURG   CMPRS  PGRP  PPID  STATE    BOOSTS           %CPU_ME %CPU_OTHRS UID  FAULTS    COW    MSGSENT    MSGRECV    SYSBSD
79918  java         98.1  05:00.05 31/1   3    409   3797M+ 0B     195M   79914 79914 running  *46[88]          0.02332 0.00000    502  1480477+  1861   556303+    115840+    612153+

I had to use Force Quit on the mac to get VERDI to stop.

The model data file covered the CONUS and contained 6 variables so it was a 2.7G file. I used m3xtract and m3wndw to create a file with only O3 and NO2, and a reduced spatial dimension that covers only columns 352-362 and rows 147-157, which is within the area that the observational data is located. The file name and size is now:

ls -lht 2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.O3.NO2.wndw.20110728
-rw-rw-r-- 1 lizadams rc_cep-emc_psx 839K Feb 15 13:27 2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.O3.NO2.wndw.20110728

Also added the observational dataset to the VERDI so it will be available in future builds under VERDI_2.1.4/data/obs/discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict

[lizadams]

working on extracting the same windowed domain for the 3.7G METCRO3D_20110711 file. I am getting an error.

Program version: 
     $Id:: m3xtract.f 117 2019-06-15 14:56:29Z coats               $

     Value for PROMPTFLAG not defined;returning default:   TRUE
     Enter logical name for  INPUT FILE [INFILE] >>
     Error opening file at path-name:
     netCDF error number -128  processing file "INFILE"

Found this error described in the CMAS User Forum: https://forum.cmascenter.org/t/error-abort-in-subroutine-m3wndw/2199/3

Need to use ncatted to replace the following attribute. (or some value that will allow the m3wndw program to run successfully.

 :IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 1 2017-06-10 18:05:20Z coats $   

with

 :IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 200 2021-05-10 14:06:20Z coats $ 

The METCRO3D_20110728 file didn't have the same issue. It looks like perhaps the source of the problem was this attribute: _NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.18" ; comparing the difference between diff METCRO3D_20110711.header METCRO3D_20110728.header

<       :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.18" ;
<       :IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 1 2017-06-10 18:05:20Z coats $                        " ;
---
>       :IOAPI_VERSION = "$Id:: init3.F 1 2014-03-14 20:22:54Z coats                  $        

I didn't have the same issue using m3wndw on the METCRO3D_20110728 file.

I've added the METCRO3D_20110728 file windowed to the region of interest to the VERDI repo. The next build should contain all of the files required to do vertical layer matching for the aircraft observation testing.

TODO:

• [ ] look for another *.ict file that contains a different variable name such as ALTP for the altitude rather than GPSAlt that is in the discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict that we already have.

It looks like the ict files that contain merge in the name: merge are for flights covering a different location, different lat, lon values. I tried extracting only the July 28 data from the 0701-0729 file, but I can't get VERDI to plot the data, as the obs values are located outside of the CONUS. I am also not sure that any of these files contained data for O3 or NO2, as some of the values are -99999.

• [ ] upload another *.ict file that contains different units for the altitude km versus m

I will need help identifying files that contain these different variations of variables and units that covers the same time and region.

[lizadams]

@yadongxuEPA Following the list of todo items in the comment above, I took an additional look at the files that contained ALTP for the altitude rather than GPSAlt.

Here is a slice of the data from discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict I chose one row of the data for the JDAY 209.

The latitude, longitude values are: 37.936733, 284.536545

VERDI didn't plot this data correctly. When I reached timestep 14, the following pop-up window appeared.

I used awk to change any -999999 value to 60 for the O3_ppbv column, and was able to get past this error message. So that indicates that VERDI needs to handle NaN when doing the Obs Overlay.

For the merge file, the Latitude and Longitude values are in all CAPPS, and I don't think that VERDI is currently recognizing that, and the Logitude value may need to be adjusted by subtracting 360 degrees: 284.536545 - 360 = -75.463455

This location on Google Maps is near Wallops Field in Virginia. https://www.google.com/maps/place/37%C2%B056'12.2%22N+75%C2%B027'48.4%22W/@37.9367372,-75.4984739,13z/data=!4m4!3m3!8m2!3d37.936733!4d-75.463455

 Fractional_Day,  UTC, JDAY, INDEX, FLIGHT, LOCAL_SUN_TIME, LATITUDE, LONGITUDE, ALTP, PRESSURE, TEMPERATURE, THETA, O3COLUMN, SZA, WNS, WND, FMS_TAS, FMS_SAT, FMS_GRD_SPD, Heading, FMS_TRK, IRS_PITCH, IRS_ROLL, IRS_VERT_ACC, ADC_IAS, GPS_ALT, A_DewPoint, A_CabinPressure, A_SurfTemp, A_TotalTemp, A_JNO2_Nadir, A_JNO2_Zenith, A_RadarAlt, C_DiffPressure, C_PotTempDegK, C_MachNumber, C_CabAltitude, C_VaporPresWater, C_SatVaporPresWater, C_SatVaporPresIce, C_MixingRatio, C_RelHumidity, H2O(DLH), NO, NOy, NO2_NCAR, O3_ppbv, Carbon_Monoxide_mixing_ratio, Methane_mixing_ratio, Carbon_dioxide_mixing_ratio, NO2_LIF, PNs_TD-LIF, ANs_TD-LIF, HNO3_TD-LIF, CH2O_DFGAS, Abs470tot, Abs532tot, Abs660tot, nAPS_total, sAPS_total, vAPS_total, nAPS_fine(Daero_lt_1.0um), sAPS_fine(Daero_lt_1.0um), vAPS_fine(Daero_lt_1.0um), nAPS_coarse(Daero_ge_1.0um), sAPS_coarse(Daero_ge_1.0um), vAPS_coarse(Daero_ge_1.0um), BlackCarbonMassConcentration, CN>3nm, CN>10nm, nonvolCN>10nm, Scat450tot, Scat550tot, Scat700tot, Scat450sub, Scat550sub, Scat700sub, nUHSAS(60-1000_nm), sUHSAS(60-1000_nm), vUHSAS(60-1000_nm), nLAS(0.9-7.7_um), sLAS(0.9-7.7_um), vLAS(0.9-7.7_um), Angstrom_Exponent_of_Scattering_at_450and700nm, Angstrom_Exponent_of_Scattering_at_450and550nm, Angstrom_Exponent_of_Absorption_at_450and700nm, Angstrom_Exponent_of_Absorption_at_450and550nm, SingleScatteringAlbedo_at_450nm, SingleScatteringAlbedo_at_550nm, SingleScatteringAlbedo_at_700nm, SingleScatteringAlbedo_at_550nm(ambient), nSMPS, sSMPS, vSMPS, Acetaldehyde, Acetonitrile, Acetone, Methanol, Toluene, Isoprene, MVK_MAC, Xylenes_ETB, Acetic_acid, Monoterpenes, RHamb, RHdry, RHwet, SCdry, SCwet, gamma, fRH_80_20, SCamb, EXTamb532, EXTdry532, SCamb532, SCdry532, ABSdry532, Chloride, Nitrite, Nitrate, Sulfate, Sodium, Ammonium, Potassium, Magnesium, Calcium, Water-Soluble_Organic_Carbon_Mass, discoveraq-SiteFlag-1sec, ProfileSequenceNum
       209.60079,51908,   209,          130002.00,         13.000000,  9.388130778,   37.936733,   284.536545,   -0.03989832049,   1019.03,   302.0799939,   300.4573479,   0,   38.83436106,    -999999,   -999999,  68.27133241,   301.6499939,   70.78179961,   212.01,   212.7,   0.47,   0.44,   0.029,   68.77033921,   -0.01392936017,   295.1499939,   1019.528127,   319.0499939,   304.3899939,   0.000598,   0.007919,   0.000161239202,   27.516,   300.467,   0.195,   -0.0519494116,   26.5538,   40.0809,   52.7245,   16.64112,   66.25,   24926.9,    -999999,   -999999,   -999999,   -999999,   100.0,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  18.79,   0.25,   0,   0,   0,   0,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  43.26,   4.78,   0.19,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  66.25,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,                0,                0

[dkang2]

@lizadams The Latitude and Logitude are 37.936733 and 284.536545, respectively.

[yadongxuEPA]

Hi Liz,

It is very likely that different files contain different variable names for altitude. I saw both "ALTP" and "GPS_ALT" in the header you extracted. Don't know how they are related though.

Best, Yadong

From: Liz Adams @.> Sent: Thursday, February 16, 2023 10:32 AM To: CEMPD/VERDI @.> Cc: Xu, Yadong @.>; Mention @.> Subject: Re: [CEMPD/VERDI] Temporal and spatial accuracy when overlay aircraft data in a tile plot - testing VERDI 2.1.4 20221109 builds (Issue #312)

@yadongxuEPAhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FyadongxuEPA&data=05%7C01%7Cxu.yadong%40epa.gov%7C27e3f3edf8d44a7671ce08db1032feb2%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C638121583427151353%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0U1Vp0U%2FJLJBfyKafhBdtRMBxM6kPbqE%2FmaCkJwGQE8%3D&reserved=0 Following the list of todo items in the comment above, I took an additional look at the files that contained ALTP for the altitude rather than GPSAlt.

Here is a slice of the data from discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict I chose one row of the data for the JDAY 209.

Fractional_Day, UTC, JDAY, INDEX, FLIGHT, LOCAL_SUN_TIME, LATITUDE, LONGITUDE, ALTP, PRESSURE, TEMPERATURE, THETA, O3COLUMN, SZA, WNS, WND, FMS_TAS, FMS_SAT, FMS_GRD_SPD, Heading, FMS_TRK, IRS_PITCH, IRS_ROLL, IRS_VERT_ACC, ADC_IAS, GPS_ALT, A_DewPoint, A_CabinPressure, A_SurfTemp, A_TotalTemp, A_JNO2_Nadir, A_JNO2_Zenith, A_RadarAlt, C_DiffPressure, C_PotTempDegK, C_MachNumber, C_CabAltitude, C_VaporPresWater, C_SatVaporPresWater, C_SatVaporPresIce, C_MixingRatio, C_RelHumidity, H2O(DLH), NO, NOy, NO2_NCAR, O3_ppbv, Carbon_Monoxide_mixing_ratio, Methane_mixing_ratio, Carbon_dioxide_mixing_ratio, NO2_LIF, PNs_TD-LIF, ANs_TD-LIF, HNO3_TD-LIF, CH2O_DFGAS, Abs470tot, Abs532tot, Abs660tot, nAPS_total, sAPS_total, vAPS_total, nAPS_fine(Daero_lt_1.0um), sAPS_fine(Daero_lt_1.0um), vAPS_fine(Daero_lt_1.0um), nAPS_coarse(Daero_ge_1.0um), sAPS_coarse(Daero_ge_1.0um), vAPS_coarse(Daero_ge_1.0um), BlackCarbonMassConcentration, CN>3nm, CN>10nm, nonvolCN>10nm, Scat450tot, Scat550tot, Scat700tot, Scat450sub, Scat550sub, Scat700sub, nUHSAS(60-1000_nm), sUHSAS(60-1000_nm), vUHSAS(60-1000_nm), nLAS(0.9-7.7_um), sLAS(0.9-7.7_um), vLAS(0.9-7.7_um), Angstrom_Exponent_of_Scattering_at_450and700nm, Angstrom_Exponent_of_Scattering_at_450and550nm, Angstrom_Exponent_of_Absorption_at_450and700nm, Angstrom_Exponent_of_Absorption_at_450and550nm, SingleScatteringAlbedo_at_450nm, SingleScatteringAlbedo_at_550nm, SingleScatteringAlbedo_at_700nm, SingleScatteringAlbedo_at_550nm(ambient), nSMPS, sSMPS, vSMPS, Acetaldehyde, Acetonitrile, Acetone, Methanol, Toluene, Isoprene, MVK_MAC, Xylenes_ETB, Acetic_acid, Monoterpenes, RHamb, RHdry, RHwet, SCdry, SCwet, gamma, fRH_80_20, SCamb, EXTamb532, EXTdry532, SCamb532, SCdry532, ABSdry532, Chloride, Nitrite, Nitrate, Sulfate, Sodium, Ammonium, Potassium, Magnesium, Calcium, Water-Soluble_Organic_Carbon_Mass, discoveraq-SiteFlag-1sec, ProfileSequenceNum

   209.60079,51908,   209,          130002.00,         13.000000,  9.388130778,   37.936733,   284.536545,   -0.03989832049,   1019.03,   302.0799939,   300.4573479,   0,   38.83436106,    -999999,   -999999,  68.27133241,   301.6499939,   70.78179961,   212.01,   212.7,   0.47,   0.44,   0.029,   68.77033921,   -0.01392936017,   295.1499939,   1019.528127,   319.0499939,   304.3899939,   0.000598,   0.007919,   0.000161239202,   27.516,   300.467,   0.195,   -0.0519494116,   26.5538,   40.0809,   52.7245,   16.64112,   66.25,   24926.9,    -999999,   -999999,   -999999,   -999999,   100.0,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  18.79,   0.25,   0,   0,   0,   0,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  43.26,   4.78,   0.19,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  66.25,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,                0,                0

- Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FCEMPD%2FVERDI%2Fissues%2F312%23issuecomment-1433275466&data=05%7C01%7Cxu.yadong%40epa.gov%7C27e3f3edf8d44a7671ce08db1032feb2%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C638121583427151353%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gIz8%2FMUjJ0JgtSCmGxyH3AV82R0bvercdF6liTKOOak%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAROL2T44PFEZDJPSB5VGL6LWXZCAFANCNFSM6AAAAAAR562L5M&data=05%7C01%7Cxu.yadong%40epa.gov%7C27e3f3edf8d44a7671ce08db1032feb2%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C638121583427151353%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=kWHw%2FRU12XRbyH%2BJNvWT%2BLpCxyJ0Mmlba132%2FgnlCRE%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[dkang2]

Another thought, it is probably good to use the pressure levels (or at least as an option). In general, the pressure levels should be in both the observations and model outputs.

[lizadams]

Attaching a gzipped version of the file where I extracted JDAY 209 or 2011209. In case it helps troubleshoot this issue.

discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict.gz

[lizadams]

@dkang2 Is there a formula to convert pressure levels to layer height? I did find this document for a method to compare satellite data to CMAQ output. https://www.ldeo.columbia.edu/~amfiore/haqast_TT_files/HAQAST_SIP_TT_GT_Technical_Guidance_Comparison_of_CMAQ_to_OMI-V1.0.pdf

[dkang2]

@lizadams There are some algorithms to convert pressure levels to layer height. It can be interpolated using the bottom and top pressure values and the sigma values, p=sigma*(psfc-ptop)+ptop to find the bottom and top pressures at each layer. It becomes more complicated when using the hybrid vertical structure. However, I think that any interpolation should be good enough to make the comparisons, especially the aircraft observations are mainly in the lower to middle troposphere. In the LTNG_DEFN.F in CMAQ code, the lightning NO is matched to the model layer in this way (find the BOTTOM and TOP variables). The most simple and accurate method is still using the ZF variable.

[yadongxuEPA]

Reference

Agree with Daiwen's comments. Yadong

[yadongxuEPA]

Tested VERDI 2.1.4 20230224 builds on Atmos. Found that we still have timesteps matching issue between model data and aircraft data for some aircraft data files. When used “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”, the timesteps matching worked ok. But when switched to use another aircraft file "discoveraq-mrg01-p3b_merge_20110701_R3_thru20110729.ict", I got the following error message:

[yadongxuEPA]

Retested the timesteps matching issue with VERDI 2.1.4 20230911 builds on Atmos. Found that we don't have error messages showed up when load the merged file such as "discoveraq-mrg01-p3b_merge_20110701_R3_thru20110729.ict", but no data points showed up in the map after we add them as observations.

Furthermore, there was an error message popped-up when used the aircraft data for one specific day such as “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”.

[yadongxuEPA]

Retested the timesteps matching issue with VERDI 2.1.4 20230924 builds on Atmos. Found that this issue has improved and but the timesteps matching is not consistent with the ones we saw in our previous tests.
When use “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”. We found that only two hours (T14 and T15 UTC have O3 observations)

But our previous testing results showed that we have three hours (T13, T14, T15 UTC) with O3 observations, I also checked the csv files extracted from the *.ict file and confirmed that we do have several data points within T13.
DaiWen, Can you double-check if “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” contains O3 observations at T13 UTC? Thanks, Yadong

[yadongxuEPA]

Continued testing the timesteps matching issue with VERDI 2.1.4 20230924 builds on Atmos. When use the merged aircraft data “discoveraq-mrgLARGE-pilsIC-p3b_merge_20110701_R3_thru20110729.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”. Found that the timesteps when observations showed up in the map are not consistent with what we see from the extracted data in CSV file. In VERDI, we can see observational data points appeared at 2011-07-28 T00 UTC through 2011-07-28 T13 UTC.