search

E3SM-Project / e3sm_diags

E3SM Diagnostics package
https://e3sm-project.github.io/e3sm_diags
BSD 3-Clause "New" or "Revised" License
40 stars 32 forks source link

Unstructured grid files are read for lat-lot set by mistake, need more informative error msg #165

Open oksanaguba opened 6 years ago

oksanaguba commented 6 years ago

On Edison, I have 3 very similar runs. They all have the same duration, output, etc. However, initially diagnostics with lat-lon was producing different plots for all 3. Many plots were missing. For example, all ANN plots were missing from 2 of 3 runs, while all 3 runs have the same data structures.

Turns out that ncclimo produces files with structured (latlon) and unstructured grids (ncol). I did not know which are needed, so, I pull all of them in the folder where diagnostics was running. However, diagnostics would pick different files for observations for different runs. It would open file with unstructured grid for one run and file with structured grid for another. Turns out that in function _findfile(), value

listdir_files = os.listdir(path_name)

was used to find the first match for a file. The first match is not necessarily a structured file, how it should be (and listdir_files does not seem to have any particular order). In the function, filename match has to satisfy case name and season, but nothing else. In 2 out of 3 runs, first match would be a file with unstructured grid, and diagnostics would produce an error, no plot.

The best solution for me was to move all unstructured files (the ones with mask *000*_climo.nc) . There are still some errors in the output (at least some of them are related to non existing fields in observations), but at least sets of diagnostics plots now match for my 3 runs, as supposed to.

Initially, the error about unstructured files looked like this

test file: /scratch1/scratchdirs/onguba/vsplit-diag/may09-f2/vsplit-f2-6y_ANN_000112_000611_climo.nc
reference file: /global/project/projectdirs/acme/acme_diags/obs_for_acme_diags/GPCP_v2.2_ANN_climo.nc
Variable: PRECT
Selected region: global
no domain selector
Error in acme_diags.driver.lat_lon_driver
Traceback (most recent call last):
  File "/global/homes/o/onguba/.conda/envs/e3sm_diags_env/lib/python2.7/site-packages/acme_diags/acme_diags_driver.py", line 79, in run_diag
    single_result = module.run_diag(parameters)
  File "/global/homes/o/onguba/.conda/envs/e3sm_diags_env/lib/python2.7/site-packages/acme_diags/driver/lat_lon_driver.py", line 235, in run_diag
    mv1_domain, mv2_domain, parameter.regrid_tool, parameter.regrid_method)
  File "/global/homes/o/onguba/.conda/envs/e3sm_diags_env/lib/python2.7/site-packages/acme_diags/driver/utils.py", line 164, in regrid_to_lower_res
    if len(axes1[1]) <= len(axes2[1]):
IndexError: list index out of range
oksanaguba commented 6 years ago

Could I make a suggestion to have it fixed in the repo in case someone else gets the same error but they don't know about unstructured files? Is it possible to check whether file has structured grid or not?

chengzhuzhang commented 6 years ago

Hello @oksanaguba,

Thank you for bringing in our attention this issue! As you discovered, we don't have a mechanism to check for structured/unstructured grid files yet. Since our package is modeled after NCAR's amwg package. We assumed most users are familiar with the input data requirement for the package (https://e3sm-project.github.io/acme_diags/docs/html/index.html search for input data requirement). We will try add a check function for data dimensions, and throw a clear error message than currently we have. Thanks a lot for your feedback!