Obs overlays not working in VERDI 2.1.4 20221109 version

[jherwehe]

Describe the bug The observations overlay feature of VERDI 2.1.4 became broken in the 20221109 version (see screen capture below), but was working in the previous 20221007 version. This is a repeat of closed Issue #308.

To Reproduce Steps to reproduce the behavior (as tested on the Atmos Linux system):

1. Open the 20221109 version of VERDI 2.1.4.
2. Load model output file /work/MOD3DEV/jherwehe/verdi/test/bkx/cmaq/wr401/task4/2018/cctm/extr/201808/combine_aconc_v53_intel18.0_2018ff_12US1_20180817
3. Load associated observations file /work/MOD3DEV/jherwehe/verdi/test/wtt/obs_data/Release_Obs_Dev/Tools/Obs_Overlay/AQS_Hourly_PAVE_VERDI_overlay_20180817.ncf
4. Create a tile plot of the O3 field.
5. Click on "Plot", then click on "Add Overlay", then click on "Observations", select O3 from the "Observation Details" box, click "Add Variable", and finish by clicking the "OK" button.
6. See error shown in screen capture below.

Expected behavior The obs overlay feature worked fine in the previous 20221007 version of VERDI 2.1.4. The 20221007 version also worked correctly when loading an entire year of observations to use with the same test model output file above, such as when using observations file /work/MOD3DEV/jherwehe/verdi/test/wtt/obs_data/Release_Obs_Dev/Tools/Obs_Overlay/AQS_Hourly_PAVE_VERDI_overlay_2018.ncf

Screenshots

Tested on the Atmos HPC system:

• OS: Red Hat Enterprise Linux Server release 7.9 (Maipo)
• Linux atmos5 3.10.0-1160.80.1.el7.x86_64 # 1 SMP Sat Oct 8 18:13:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[dkang2]

@jherwehe Thanks for testing and identifying this problem. I guess that the addition of the aircraft data overlay messed up the overlay code again, the same as last time.

[yadongxuEPA]

Confirmed this issue with another set of dataset with the model output file: data/model/CCTM46_P16.baseO2a.36k.O3MAX and observational data file: data/obs/AQS_overlay_2002.2hr.2002152.ncf, got the same error message:

[yadongxuEPA]

Retested VERDI_2.1.4_linux64_20221206.tar.gz on Atmos, found that the issue has resolved.

[yadongxuEPA]

Confirmed this issue with two sets of dataset with VERDI_2.1.4_linux64_20230103.tar.gz on Atmos, found that the previous error messages came back, this issue is not resolved in the new builds.

[yadongxuEPA]

Confirmed this issue with two sets of dataset with VERDI_2.1.4_linux64_20230201.tar.gz on Atmos, found that this issue is not resolved in the new builds. We still cannot overlay non-aircraft observations with CMAQ outputs. Strangely, we got two different error messages shown as below. Please check.

[dkang2]

Just did a quick test for the 20230224 build, the overlay worked well on Atmos.

[yadongxuEPA]

Confirmed this issue with two sets of dataset with VERDI_2.1.4_linux64_20230224.tar.gz on Atmos, found that this issue has resolved in the new builds (with timestamps matching and observational data points color scale matching all work properly)

[dkang2]

The overlay function stopped working again with the 20230924 build.

[yadongxuEPA]

The overlay function stopped working again with the 20230924 build.

Hi Daiwen,

Thanks for retesting this issue. I got the same error messages when tested 20230924 builds on Atmos5. Yadong

[yadongxuEPA]

Confirmed this issue with VERDI_2.1.4_linux64_20231012.tar.gz on Atmos, found that this issue has resolved in the new builds.

[jherwehe]

This issue is only partially resolved with VERDI_2.1.4_linux64_20231012.tar.gz. The observations do plot now, but they are not scaled correctly (which can also be seen in Yadong's screenshot above). Here is an example of the problem for August 17, 2018 at 20UTC:

Using the VERDI_2.1.4_linux64_20221206.tar.gz version, the correctly scaled O3 obs overlays for the same time for these data should appear as seen in the screenshot below:

[yadongxuEPA]

Tony, It seems that you pushed more updates to Atmos today into 20231012 builds. I retested the issue as Jerry mentioned above and found the same problem. Please check into it and push the updates to a new date so that we can track what builds we have tested. Thanks! Yadong

[jherwehe]

Retested this issue with VERDI_2.1.4_linux64_20231013.tar.gz on Atmos and found that this issue has been resolved in this latest build; the obs overlays now appear to reflect the correct color scale.

[lizadams]

Tested using VERDI_2.1.5_mac_20240103.tar.gz

Obtained the data from the following locations: combine_aconc_v53_intel18.0_2018ff_12US1_20180817 https://drive.google.com/file/d/1HhYp49voK4j36ISwFu-cabrjqUqJitnL/view?usp=drive_link

AQS_Hourly_PAVE_VERDI_overlay_2018.ncf https://drive.google.com/file/d/1Qwb6ay-MwacaY3mSNywu3n35RstO-JKC/view?usp=drive_link

Used the following command line option to create the tile plot, and then used the GUI to add the observation overlay.

./verdi.sh -f $cwd/../combine_aconc_v53_intel18.0_2018ff_12US1_20180817 -s "O3[1]" -legendBins "0, 14, 28, 42, 56, 70, 84, 98, 112" -ts 21 -g tile -f $cwd/../AQS_Hourly_PAVE_VERDI_overlay_2018.ncf -s "O3[2]"

Confirmed that it works.

However, I noticed that the command line isn't using the "Tile Plot default" and is using the Newton RGB (AVS) so the colors look different than Yadong's plot above.

I changed the Color map to use the Tile Plot default, and this caused the obs overlay to be cleared, so I had to reload it. Once the Tile Plot default color map is used, it matches Yadong's plot.

[lizadams]

@yadongxuEPA - Is this issue fixed in the new build for 2.1.5? If so, should we add it to the release notes?

[yadongxuEPA]

@yadongxuEPA - Is this issue fixed in the new build for 2.1.5? If so, should we add it to the release notes?

No, we don't need to add this issue in the release note even if it is fully resolved. Because it was working properly in VERDI 2.1.4 to start with, then some code changes made for overlaying aircraft data caused the issue.

[yadongxuEPA]

@yadongxuEPA - Is this issue fixed in the new build for 2.1.5? If so, should we add it to the release notes?

No, we don't need to add this issue in the release note even if it is fully resolved. Because it was working properly in VERDI 2.1.4 to start with, then some code changes made for overlaying aircraft data caused the issue.

Just want to add, the public users should not have VERDI 2.1.4 20221109 builds.

[lizadams]

tested on a mac using VERDI_2.1.5_mac_20240124.tar.gz

Using the following config file (changed extension from .cfg to .txt to allow github to upload the file: TilePlot_default_0_to_112.txt

Used the command

./verdi.command -f $cwd/../combine_aconc_v53_intel18.0_2018ff_12US1_20180817 -s "O3[1]" -ts 21 -configFile $cwd/data/configs/TilePlot_default_0_to_112.txt -g tile -f $cwd/../AQS_Hourly_PAVE_VERDI_overlay_2018.ncf -s "O3[2]"

Then in the GUI, used Controls>Plot>Add Overlay> Select O3 > Add variable > select OK

The plot appears to match the correct plot provided by @jherwehe

So, I think this issue is fully solved now.

[yadongxuEPA]

Retested this issue with VERDI_2.1.5_linux64_20240124.tar.gz on Atmos and confirmed that it has resolved.