search

CEMPD / VERDI

This is the repo for the VERDI project, written in java.
GNU General Public License v3.0
16 stars 13 forks source link

Reading netCDF files with unconventional dimension names #344

Open dkang2 opened 4 months ago

dkang2 commented 4 months ago

Is your feature request related to a problem? Please describe. Yes, occasionally in some netCDF files there are variables that have dimensions other than those normally recognized by VERDI (columns, rows, layers, or south-north, west-east, and bottom-top), such at categories of crop types, soil types, or species types. Usually the variables have the dimensions as var(Time, var-cat, south-north, west-east). When this kind of files are opened by VERDI, the variables with the unconventional dimensions are ignored (no showing in the variable list).

Describe the solution you'd like In conventional IOAPI (TSTEP, LAY, ROW, COL) or wrf netCDF (Times, bottom_top, south_north, west_east) files, verdi would automatically recognize the variables as 4-D time-space variables. But in the files with the unconventional dimension names, the first (Time) and the second and third (spatial) dimensions are generally the same. The only difference is the forth (LAY in case of IOAPI, bottom_top in case of wrf netCDF) dimension. One proposed solution is to treat the third (time-independent) or forth (time-dependent) unconventional dimension as the vertical layers as long as the other dimensions are conformable.

Describe alternatives you've considered None.

Additional context An example file can be found at: /work/MOD3DEV/dkj/Verdi/example_files/beld4_CMAQ12KM_2011.nc and look for variables with land_cat, crop_cat, and tree-cat as one of the dimensions.

yadongxuEPA commented 1 month ago

Tested with VERDI_2.1.5_linux64_20240717.tar.gz on Atmos, found that this issue has not resolved. used this data file: /work/MOD3DEV/dkj/Verdi/example_files/beld4_CMAQ12KM_2011.nc

Found no differences in the app behaviors compared with loading the data file into "VERDI_2.1.5_linux64_20240524.tar.gz", the hidden variables (with unconventional dimension names) were not listed under "Variables" dropdown window. image image

yadongxuEPA commented 1 month ago

Add more details to the expected behavior for the above test case.
Use "ncdump", we can see these dimensions & variables in the data file: image testing_20240717_issue_344

We expected to see "TREEF", "CROPF" to be added under the variable dropdown window. Apparently, the older version VERDI (20240524 builds) already had code structure to replace "layer" with "land_cat" because we see "LANDUSEF" is listed and available for tile plot.

testing_20240717_issue_344_1

yadongxuEPA commented 1 month ago

Tested with VERDI_2.1.5_linux64_20240725.tar.gz on Atmos, found that this issue has not resolved. used this data file: /work/MOD3DEV/dkj/Verdi/example_files/beld4_CMAQ12KM_2011.nc

Although "TREEF", "CROPF" are added to the variable list, but "tree_cat" and "crop_cat" have not been imported as the fourth dimension to replace "layer".
testing_20240725_issue_344_1 testing_20240725_issue_344_2

dkang2 commented 1 month ago

Just tested the 20240814 build on linux (atmos) and found that the issue has been nicely resolved. In addition, the SLF4J error has also resolved.

yadongxuEPA commented 2 weeks ago

Tested with VERDI_2.1.5_linux64_20240824.tar.gz on Atmos, found that this issue has resolved. used this data file: /work/MOD3DEV/dkj/Verdi/example_files/beld4_CMAQ12KM_2011.nc image image