search

CDAT / cdms

8 stars 10 forks source link

cdscan KeyError: 'height' #421

Open mzelinka opened 3 years ago

mzelinka commented 3 years ago

Describe the bug I can read in a netcdf file without issue, but when I try to read in data from a cdscan-generated xml that points to that same file, I get KeyError: 'height'.

To Reproduce Steps to reproduce the behavior (on feedback2.llnl.gov): 1) I am able to open this netcdf file without any problem: f=cdms2.open('/p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/tas_Amon_KIOST-ESM_abrupt-4xCO2_r1i1p1f1_gr1_185001-200012.nc')

2) Generate an xml pointing to this file via cdscan: In [69]: import os

In [70]: os.system('cdscan -x temp.xml /p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/tas_Amon_KIOST-ESM_abrupt-4xCO2_r1i1p1f1_gr1_185001-200012.nc')
Finding common directory ... Common directory: /p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/ Scanning files ... /p/css03/esgf_publish/CMIP6/CMIP/KIOST/KIOST-ESM/abrupt-4xCO2/r1i1p1f1/Amon/tas/gr1/v20191106/tas_Amon_KIOST-ESM_abrupt-4xCO2_r1i1p1f1_gr1_185001-200012.nc Setting reference time units to days since 1850-01-01 temp.xml written Out[70]: 0

3) Now, try to open this xml file: In [71]: f=cdms2.open('temp.xml')

KeyError Traceback (most recent call last)

in ----> 1 f=cdms2.open('temp.xml') ~/workdir2/miniconda3/envs/cdat82/lib/python3.7/site-packages/cdms2/dataset.py in openDataset(uri, mode, template, dods, dpath, hostObj) 570 dpath = head 571 --> 572 dataset = Dataset(uri, mode, datanode, None, dpath) 573 return dataset 574 ~/workdir2/miniconda3/envs/cdat82/lib/python3.7/site-packages/cdms2/dataset.py in __init__(self, uri, mode, datasetNode, parent, datapath) 762 # Collect named children (having attribute 'id') into dictionaries 763 if datasetNode is not None: --> 764 coordsaux = self._convention_.getDsetnodeAuxAxisIds(datasetNode) 765 766 for node in list(datasetNode.getIdDict().values()): ~/workdir2/miniconda3/envs/cdat82/lib/python3.7/site-packages/cdms2/convention.py in getDsetnodeAuxAxisIds(self, dsetnode) 147 # Check the rank, don't include if not 1D or 2D (e.g., 148 # scalar coordinate) --> 149 domnode = dsetdict[item].getDomain() 150 if domnode.getChildCount() not in [1, 2]: 151 continue KeyError: 'height' **Expected behavior** I expected no problem opening this xml file, just as I experienced no problem opening the netcdf that it points to. ## Environment Information
`conda info`

``` (cdat82) -bash-4.1$ conda info active environment : cdat82 active env location : /work/zelinka1/workdir2/miniconda3/envs/cdat82 shell level : 2 user config file : /work/zelinka1/.condarc populated config files : /work/zelinka1/.condarc conda version : 4.8.3 conda-build version : not installed python version : 3.7.4.final.0 virtual packages : __glibc=2.12 base environment : /work/zelinka1/workdir2/miniconda3 (writable) channel URLs : https://conda.anaconda.org/cdat/label/v82/linux-64 https://conda.anaconda.org/cdat/label/v82/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /work/zelinka1/workdir2/miniconda3/pkgs /work/zelinka1/.conda/pkgs envs directories : /work/zelinka1/workdir2/miniconda3/envs /work/zelinka1/.conda/envs platform : linux-64 user-agent : conda/4.8.3 requests/2.24.0 CPython/3.7.4 Linux/2.6.32-754.28.1.el6.x86_64 rhel/6.10 glibc/2.12 UID:GID : 25073:1026 netrc file : None offline mode : False ```

`conda config --show-sources`

``` (cdat82) -bash-4.1$ conda config --show-sources ==> /work/zelinka1/.condarc <== ssl_verify: False channel_priority: strict channels: - cdat/label/v82 - conda-forge - defaults ```

`conda list --show-channel-urls`

``` (cdat82) -bash-4.1$ conda list --show-channel-urls # packages in environment at /work/zelinka1/workdir2/miniconda3/envs/cdat82: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 0_gnu conda-forge alabaster 0.7.12 py_0 conda-forge astroid 2.3.3 py37_1 conda-forge attrdict 2.0.1 py_0 conda-forge attrs 19.3.0 py_0 conda-forge autopep8 1.5 py_0 conda-forge babel 2.8.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge basemap 1.2.1 py37hd759880_1 conda-forge beautifulsoup4 4.8.2 py37_0 conda-forge bleach 3.1.0 py_0 conda-forge bokeh 1.4.0 py37_0 conda-forge bzip2 1.0.8 h516909a_2 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge cartopy 0.17.0 py37hd759880_1006 conda-forge cdat 8.2 3 cdat/label/v82 cdat_info 8.2 py_7 conda-forge cdms2 3.1.4 pypi_0 pypi cdp 1.6.0 py_0 conda-forge cdtime 3.1.2 py37h7eb8c7e_6 conda-forge cdutil 8.2 py_2 cdat/label/v82 certifi 2020.6.20 py37hc8dfbb8_0 conda-forge cffi 1.13.2 py37h8022711_0 conda-forge cftime 1.2.1 py37h03ebfcd_0 conda-forge chardet 3.0.4 py37_1003 conda-forge cibots 0.3 py37_1001 conda-forge click 7.0 py_0 conda-forge climlab 0.7.6 py37hc7d476f_1 conda-forge cloudpickle 1.3.0 py_0 conda-forge cryptography 2.8 py37h72c5cf5_1 conda-forge curl 7.68.0 hf8cf82a_0 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 2.10.1 py_0 conda-forge dask-core 2.10.1 py_0 conda-forge dbus 1.13.6 he372182_0 conda-forge decorator 4.4.1 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distarray 2.12.2 py_1 conda-forge distributed 2.10.0 py_0 conda-forge docopt 0.6.2 py_1 conda-forge docutils 0.16 py37_0 conda-forge dv3d 8.2 py_0 cdat/label/v82 entrypoints 0.3 py37_1000 conda-forge eofs 1.4.0 py_0 conda-forge esmf 7.1.0 hcda5390_1008 conda-forge esmpy 7.1.0 py37h5ca1d4c_3 conda-forge expat 2.2.9 he1b5a44_2 conda-forge ffmpeg 4.2 h167e202_0 conda-forge flake8 3.7.9 py37_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.0 he983fc9_1 conda-forge fsspec 0.6.2 py_0 conda-forge future 0.18.2 py37_0 conda-forge g2clib 1.6.0 hf3f1b0b_9 conda-forge genutil 8.2 py37hc1659b7_3 conda-forge geos 3.7.2 he1b5a44_2 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge ghostscript 9.22 hf484d3e_1001 conda-forge glib 2.58.3 py37h6f030ca_1002 conda-forge gmp 6.2.0 he1b5a44_1 conda-forge gnutls 3.6.5 hd3a4fd2_1002 conda-forge gst-plugins-base 1.14.5 h0935bb2_2 conda-forge gstreamer 1.14.5 h36ae1b5_2 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.5 nompi_h3c11f04_1104 conda-forge heapdict 1.0.1 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.8 py37_1000 conda-forge imagesize 1.2.0 py_0 conda-forge importlib_metadata 1.5.0 py37_0 conda-forge ipykernel 5.1.4 py37h5ca1d4c_0 conda-forge ipython 7.12.0 py37h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 py_0 conda-forge isort 4.3.21 py37_0 conda-forge jasper 1.900.1 h07fcdf6_1006 conda-forge jedi 0.16.0 py37_0 conda-forge jeepney 0.4.2 py_0 conda-forge jinja2 2.11.1 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge jsonschema 3.2.0 py37_0 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter_client 5.3.4 py37_1 conda-forge jupyter_console 6.0.0 py_0 conda-forge jupyter_core 4.6.2 py37_0 conda-forge keyring 21.1.0 py37_0 conda-forge kiwisolver 1.1.0 py37hc9558a2_0 conda-forge krb5 1.16.4 h2fd8d38_0 conda-forge lame 3.100 h14c3975_1001 conda-forge lazy-object-proxy 1.4.3 py37h516909a_1 conda-forge ld_impl_linux-64 2.33.1 h53a641e_8 conda-forge libblas 3.8.0 11_openblas conda-forge libcblas 3.8.0 11_openblas conda-forge libcdms 3.1.2 h7dcd879_5 conda-forge libcf 1.0.3 py37hb508d2e_1 conda-forge libclang 9.0.1 default_hde54327_0 conda-forge libcurl 7.68.0 hda55be3_0 conda-forge libdrs 3.1.2 h6e990d7_7 conda-forge libdrs_f 3.1.2 h322d3c2_6 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc 7.2.0 h69d50b8_2 conda-forge libgcc-ng 9.2.0 h24d8f2e_2 conda-forge libgfortran-ng 7.3.0 hdf63c60_5 conda-forge libgomp 9.2.0 h24d8f2e_2 conda-forge libiconv 1.15 h516909a_1005 conda-forge liblapack 3.8.0 11_openblas conda-forge libllvm9 9.0.1 hc9558a2_0 conda-forge libnetcdf 4.6.2 h303dfb8_1003 conda-forge libopenblas 0.3.6 h6e990d7_6 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libsodium 1.0.17 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.2.0 hdf63c60_2 conda-forge libtiff 4.1.0 hc3755c2_3 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxkbcommon 0.10.0 he1b5a44_0 conda-forge libxml2 2.9.10 hee79883_0 conda-forge libxslt 1.1.33 h31b3aaa_0 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.5.0 py37he3881c9_1 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge markupsafe 1.1.1 py37h516909a_0 conda-forge matplotlib 3.1.3 py37_0 conda-forge matplotlib-base 3.1.3 py37h250f245_0 conda-forge mccabe 0.6.1 py_1 conda-forge mechanicalsoup 0.12.0 py_0 conda-forge mistune 0.8.4 py37h516909a_1000 conda-forge mpi 1.0 mpich conda-forge mpich 3.3.2 hc856adb_0 conda-forge msgpack-python 0.6.2 py37hc9558a2_0 conda-forge nb_conda 2.2.1 py37_2 conda-forge nb_conda_kernels 2.2.2 py37_0 conda-forge nbconvert 5.6.1 py37_0 conda-forge nbformat 5.0.4 py_0 conda-forge ncurses 6.1 hf484d3e_1002 conda-forge netcdf-fortran 4.4.5 h0789656_1004 conda-forge netcdf4 1.5.1.2 py37h73a1b54_1 conda-forge nettle 3.4.1 h1bed415_1002 conda-forge notebook 6.0.3 py37_0 conda-forge nspr 4.25 he1b5a44_0 conda-forge nss 3.47 he751ad9_0 conda-forge numpy 1.17.5 py37h95a1406_0 conda-forge numpydoc 0.9.2 py_0 conda-forge olefile 0.46 py_0 conda-forge openblas 0.3.6 h6e990d7_6 conda-forge openh264 1.8.0 hdbcaa40_1000 conda-forge openssl 1.1.1g h516909a_0 conda-forge output_viewer 1.3.1 py_1 conda-forge owslib 0.19.1 py_0 conda-forge packaging 20.1 py_0 conda-forge pandas 1.0.1 py37hb3f55d8_0 conda-forge pandoc 2.9.1.1 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.6.1 py_0 conda-forge partd 1.1.0 py_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pexpect 4.8.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 7.0.0 py37hefe7db6_0 conda-forge pip 20.0.2 py_2 conda-forge proj4 5.2.0 he1b5a44_1006 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.10 py_0 conda-forge psutil 5.6.7 py37h516909a_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py37_1 conda-forge pydap 3.2.2 py37_1000 conda-forge pyepsg 0.4.0 py_0 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.5.2 py_0 conda-forge pykdtree 1.3.1 py37hc1659b7_1002 conda-forge pylint 2.4.4 py37_0 conda-forge pyopenssl 19.1.0 py37_0 conda-forge pyparsing 2.4.6 py_0 conda-forge pyproj 1.9.6 py37h516909a_1002 conda-forge pyqt 5.12.3 py37hcca6a23_1 conda-forge pyqt5-sip 4.19.18 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pyrsistent 0.15.7 py37h516909a_0 conda-forge pyshp 2.1.0 py_0 conda-forge pysocks 1.7.1 py37_0 conda-forge pyspharm 1.0.9 py37h7eb8c7e_1004 conda-forge python 3.7.6 h357f687_2 conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.7 1_cp37m conda-forge pytz 2019.3 py_0 conda-forge pyyaml 5.3 py37h516909a_0 conda-forge pyzmq 18.1.1 py37h1768529_0 conda-forge qt 5.12.5 hd8c4c69_1 conda-forge qtawesome 0.7.0 py_0 conda-forge qtconsole 4.6.0 py_0 conda-forge qtpy 1.9.0 py_0 conda-forge readline 8.0 hf8c457e_0 conda-forge regrid2 3.1.4 pypi_0 pypi requests 2.22.0 py37_1 conda-forge requests-cache 0.5.2 pypi_0 pypi rope 0.16.0 py_0 conda-forge scipy 1.4.1 py37h921218d_0 conda-forge secretstorage 3.1.2 py37_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 45.2.0 py37_0 conda-forge shapely 1.6.4 py37hec07ddf_1006 conda-forge six 1.14.0 py37_0 conda-forge snowballstemmer 2.0.0 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge soupsieve 1.9.4 py37hc8dfbb8_1 conda-forge sphinx 2.4.1 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge spyder 3.3.6 py37_1 conda-forge spyder-kernels 0.5.2 py37_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge tblib 1.6.0 py_0 conda-forge terminado 0.8.3 py37_0 conda-forge testpath 0.4.4 py_0 conda-forge tk 8.6.10 hed695b0_0 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py37h516909a_4 conda-forge traitlets 4.3.3 py37_0 conda-forge typed-ast 1.4.1 py37h516909a_0 conda-forge udunits2 2.2.27.6 h4e0c4b3_1001 conda-forge urllib3 1.25.7 py37_0 conda-forge vcs 8.2 py_2 cdat/label/v82 vcsaddons 8.2 py37h516909a_1 cdat/label/v82 vtk-cdat 8.2.0.8.2 py37h3df8362_0 cdat/label/v82 wcwidth 0.1.8 py_0 conda-forge webencodings 0.5.1 py_1 conda-forge webob 1.8.6 py_0 conda-forge wheel 0.34.2 py_1 conda-forge widgetsnbextension 3.5.1 py37_0 conda-forge windspharm 1.7.0 py37_1001 conda-forge wk 8.2 py_0 cdat/label/v82 wrapt 1.11.2 py37h516909a_0 conda-forge wurlitzer 2.0.0 py37_0 conda-forge x264 1!152.20180806 h14c3975_0 conda-forge xarray 0.15.0 py_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.2.2 h516909a_1 conda-forge zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 py_0 conda-forge zipp 2.2.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.4 h3b9ef0a_1 conda-forge ```

mzelinka commented 3 years ago

@durack1 @pochedls You guys may want to follow this

durack1 commented 3 years ago

@mzelinka you might want to bump your env up to cdat/label/v8.2.1 just to make sure this issue remains, which it most likely does

pochedls commented 3 years ago

I just checked and I do not have this issue with CDAT8.2.1. Maybe this fix helped: https://github.com/CDAT/cdms/issues/397.

mzelinka commented 3 years ago

Upgrading to the latest CDAT via these instructions did not resolve this for me. Same error.

pochedls commented 3 years ago

Sorry - I was incorrect. I was mistakenly reading the netcdf file not the xml file. This does not work in CDAT8.2 or CDAT8.2.1.

pochedls commented 3 years ago

@mzelinka - I've hit this bug now, too. I have modified my trimmodellist functionality to skip these files (and print a warning for now)...