Tile plot for 3D grid data with layer mean or sum quantities shown in 2D space

[dkang2]

Is your feature request related to a problem? Please describe. Yes. When VERDI is used to visualized 3D tile plots, sometimes it is useful to be able to show the statistics (mean or sum) of the vertical layers (all or a number of layers). For example, when shown the 3D lightning NO emissions in the following plot: Right now, it can display the tile plots for any individual layer by changing the Layer number. However, it would be useful, if it can compute the mean or total of all or some of the layers for each grid cell and display it as a tile plot. Describe the solution you'd like The "Use Layer Range" panel can be used to choose all or some of the layers, and the "Stats" box can provide the function option of either "mean" or "sum", but in this case, it would generate the mean or sum of the vertical layers as chosen from the "Use Layer Range" panel for each grid cell and generate a tile plot for that stat.

Describe alternatives you've considered None

Additional context Refer to the screenshot image from VERDI.

[yadongxuEPA]

Tested with VERDI_2.1.5_linux64_20240717.tar.gz on Atmos, found that this issue has partially resolved. used this data file: /home/yxu01/data_files_for_VERDI_testing/CCTM_LTNGEMIS3D_WWLLNs_20160701.nc We see that "layer_mean" and "layer_sum" were added into the "Stats" dropdown menu. I did spot check for the values at two grid cells (one at [col=123,row=43) in FL, the other at [col=68,row=76] in CO).

Remaining issue 1) the calculated "layer_mean" values are far away from the ones calculated in Excel ("layer_sum" is close enough)

Remaining issue 2) the probed output at the lower left corner displayed the wrong col and row (after selecting "Stats" > "layer_mean" or "layer_sum") (the reason might be the code of the newly fixed grid-cell shifted toward north-east when probed is not pulled into this new feature)

[lizadams]

I've tested on the mac for the dataset and cell that Yadong tested in Florida, and am observing the same issue with layer_mean using VERDI_2.1.5_mac_20240717.tar.gz.

./verdi.command -f $cwd/data/model/CCTM_LTNGEMIS3D_WWLLNs_20160701.nc

I tried taking the layer sum at all of the cells next to cell, row (123, 43), and I can't find a mean that matches the reported value by VERDI. I was thinking that perhaps it was reporting the value for (124,44), but that doesn't seem to be the case.

https://docs.google.com/spreadsheets/d/1-SSRXkJAY2tSXYQjCocoa8u-DPhPqp2mTz6PgEcqNl8/edit?gid=0#gid=0

[lizadams]

I added the lightning dataset as a compressed file to help with future testing. CCTM_LTNGEMIS3D_WWLLNs_20160701.nc.gz https://github.com/CEMPD/VERDI/pull/348

[dkang2]

In addition, the layer_mean seems no change over time steps, while the layer_sum changes with time steps. When the layer_mean and layer_sum functions are chosen, the Layer: option box at the top of the plot should be disabled. Please also test the values with different layer options (say from Layer 10 to Layer 15).

[dkang2]

Actually, it is the layer_mean option changes over time steps, but the layer_sum doesn't.

[dkang2]

The layer_sum generated unrealistically large negative numbers when probing. See the attached plot:

[dkang2]

Also noticed that when starting the 20240717 build, the following message is generated: ./verdi.sh -f /home/yxu01/data_files_for_VERDI_testing/CCTM_LTNGEMIS3D_WWLLNs_20160701.nc SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

[yadongxuEPA]

Tested with VERDI_2.1.5_linux64_20240725.tar.gz on Atmos, found that this issue has partially resolved. used this data file: /home/yxu01/data_files_for_VERDI_testing/CCTM_LTNGEMIS3D_WWLLNs_20160701.nc

Found that the "layer_mean" calculated values matched with the spot-checked values and "Layer" spinner was grayed-out when the "Stats" is selected at "layer_mean" and "layer_sum".

But the following issues have not resolved:

1) the probed output at the lower left corner displayed the wrong col and row (after selecting "Stats" > "layer_mean" or "layer_sum") 2) "layer_sum" does not change at different timesteps

3) Got these Java library error messages when bring up VERDI GUI or run it from command line

[dkang2]

Just quickly tested the 20240731 build, the layersum stat is partially resolved with changing timesteps. However, when the Use Layer Range is selected, the values don't seem to change by changing different layers, for instance, set Min: 10 and Max: 20. The layer_sum and layer_mean values should respond to different selections of layer range. In addition, the title should change from "Layer 1 NO[1]" to Layer_mean or Layer_sum NO[layerX-layerY]. As I'm pondering with the plot, another feature would be to add the layer_sum and layer_mean as variables in the variables list such as NO_layer_sum_Lx-Ly. With this, if there is another file that contains the column values, the difference plots between the calculated stats (sum or mean) and the values from another file can be created. We can discuss more on our next meeting for clarification. In addition, the SLF4J problem is still there as before.

[dkang2]

Just tested the 20240824 build for the layer stats. It partially worked by allowing choosing the layer ranges, but the stats are not right. As shown in the plot below, when the layers range 8-18 is selected, the values are all 0s and at the terminal it throws Error messages: 2024.08.26 08:53:13.523 [Thread-4] ERROR anl.verdi.plot.gui.FastTilePlot - Error occurred during computing statistics java.lang.ArrayIndexOutOfBoundsException: Index 11 out of bounds for length 11 at gov.epa.emvl.GridCellStatistics.computeLayers(GridCellStatistics.java:398) ~[?:?] at gov.epa.emvl.GridCellStatistics.computeStatistics(GridCellStatistics.java:130) ~[?:?] at anl.verdi.plot.gui.FastTilePlot.computeStatistics(FastTilePlot.java:1796) ~[?:?] at anl.verdi.plot.gui.FastTilePlot.copySubsetLayerData(FastTilePlot.java:1969) ~[?:?] at anl.verdi.plot.gui.FastTilePlot.access$14(FastTilePlot.java:1922) ~[?:?] at anl.verdi.plot.gui.FastTilePlot$5.run(FastTilePlot.java:575) ~[?:?] at java.lang.Thread.run(Thread.java:1583) [?:?]

In addition, when the layer statistics are selected, probably the ">0.12" box can be disabled or hidden from the scene:

[yadongxuEPA]

Tested with VERDI_2.1.5_linux64_20240824.tar.gz on Atmos, found that this issue has improved but only partially resolved. used this data file: /home/yxu01/data_files_for_VERDI_testing/CCTM_LTNGEMIS3D_WWLLNs_20160701.nc the following issues have not resolved:

1. the probed output at the lower left corner displayed the wrong col and row (after selecting "Stats" > "layer_mean" or "layer_sum")
2. "layer_sum" and "layer_mean" do not change according to user's layer range settings.
3. did not receive the error messages as described by Daiwen in the command window, but there were lots print-out messages as below

[lizadams]

Tested with VERDI_2.1.5_mac_20240824.tar.gz

When I choose select layer range from 34-35, the top two layers in the model, and then select layer_sum or layer_mean, VERDI throws an error that I think is related to the off-by-one index issue.

2024.08.28 14:14:08.973 [AWT-EventQueue-0] ERROR anl.verdi.plot.gui.FastTilePlot - Error occurred during computing statistics java.lang.ArrayIndexOutOfBoundsException: Index 33 out of bounds for length 2

It is looking for layer 33 instead of layer 34.

Created a plot of NO, and then used Controls > Set Rows and Columns to a subdomain region: Rows: 34 to 38 Columns 128 to 131

I can successfully use the statistics menu options of layer_mean and layer_sum if I create a tile plot without restricting the number of layers.

I also noted that when you change the columns and rows to create a subdomain, the min and max on the legend do not change, to use a min and max over this subdomain region, so I changed the min, max values using configure plot.

Note that the index of the cell value when not using statistics menu is correct.

I then changed statistics to none, and recorded the value of NO in the grid cell for each layer (probing the grid cell after changing the layer) Found that the layer_sum and layer_mean matched a calculation done in this spreadsheet.

https://docs.google.com/spreadsheets/d/1en5RTdqVo228ENhsIJoqUghZ8OMNmP8AvDpXE7b7pg8/edit?usp=sharing

However, due to problem with the grid cell indexing, the statistics don't work for a restricted number of layers.

[dkang2]

The 20240904 build partially solved the problem. When calculate the layer stats, some occasions work, some others don't. For example, if all the layers and when the min layer between 1-13, it works. As long as the min layer is selected above 14, the layer stats values become all 0s (both layer_sum and layer_mean). Without going through the layers, it is difficult to find out and the errors seem odd.

[dkang2]

Further check the "Use Layer Range", it seems even more weird: Here is the table the stats work (w) or not (x) Min Max w/x 14 <=26 x 14 >26 w 15 <=28 x 15 > 28 w 16 <= 30 x 16 >30 w

To generalize the relationship, it holds: 2n - 1, that is for Min layer n, when the max layer is >= 2n - 1 the layer stats start to work, if the max layer is less than 2n - 1, it failed. For min layer = 1, it works for all layers, but for n = 2, it only fails when max layer is also 2, works all other layers. For n =5, it works when max is larger than 8.

[yadongxuEPA]

Tested with VERDI_2.1.5_linux64_20240904.tar.gz on Atmos, found that this issue has improved but only partially resolved. used this data file: /home/yxu01/data_files_for_VERDI_testing/CCTM_LTNGEMIS3D_WWLLNs_20160701.nc

1) Confirmed that the issue of probed output at the lower left corner displayed the wrong col and row has resolved. 2) the "layer_sum" and "layer_mean" displayed incorrect values for user's layer range settings. For example, when "Use Layer Range" set at "min=9, max=20", the "layer_sum" at cell (123,43) should be 0.871 instead of 0.664 as shown in this screenshot.

when "Use Layer Range" set at "min=15, max=30", the "layer_sum" at cell (123,43) should be 34.760 instead of 8.929 as shown in this screenshot.