search

CDAT / cdms

8 stars 10 forks source link

cdms fails on axis mismatch #342

Open pochedls opened 5 years ago

pochedls commented 5 years ago

Bug description cdms is failing when I try to open a CMIP5 xml file with "CDMSError: axis length 1140 does not match corresponding dimension 1739."

To Reproduce

On Crunchy:

$ conda activate cdat8
$ ipython
In [1]: fn = '/p/user_pub/xclim/CMIP5/ScenarioMIP/rcp85/ocnBgchem/mon/spco2/CMIP5.ScenarioMIP.rcp85.NCC.NorESM1-ME.r1i1p1.mon.spco2.ocnBgchem.glb-2d-gu.v20130926.0000100.0.xml'
In [2]: import cdms2
In [3]: f = cdms2.open(fn)
In [4]: data = f('spco2')

---------------------------------------------------------------------------
CDMSError                                 Traceback (most recent call last)
<ipython-input-5-954d69680bff> in <module>()
----> 1 data = f('spco2')

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/cudsinterface.py in __call__(self, id, *args, **kwargs)
     32             except (AttributeError, TypeError):
     33                 raise CDMSError("No such variable, " + id)
---> 34         return v(*args, **kwargs)
     35
     36     def __getitem__(self, key):

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/avariable.py in __call__(self, *args, **kwargs)
    168                                    squeeze=squeeze,
    169                                    order=order,
--> 170                                    grid=grid)
    171
    172     select = __call__

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/selectors.py in unmodified_select(self, variable, raw, squeeze, order, grid)
    205             # result is variable when there are no components, for example.
    206                 return result.subRegion(squeeze=squeeze, order=order,
--> 207                                         grid=grid, raw=raw)
    208         else:
    209             return result

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/avariable.py in subRegion(self, *specs, **keys)
    822                  'grid': grid,
    823                  }
--> 824             return self.subSlice(*slicelist, **d)
    825
    826         #

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/avariable.py in subSlice(self, *specs, **keys)
    666                                        grid=resultgrid,
    667                                        attributes=self.attributes,
--> 668                                        id=self.id)
    669             if grid is not None:
    670                 order2 = grid.getOrder()

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/tvariable.py in __init__(self, data, typecode, copy, savespace, mask, fill_value, grid, axes, attributes, id, copyaxes, dtype, order, no_update_from, **kargs)
    201         if axes is not None:
    202             # Note: clobbers the grid, so set the grid after.
--> 203             self.initDomain(axes, copyaxes=copyaxes)
    204         if grid is not None:
    205             self.setGrid(grid)

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/tvariable.py in initDomain(self, axes, copyaxes)
    362                     else:
    363                         # No sense copying a virtual axis.
--> 364                         self.setAxis(i, flataxes[i])
    365             if newgrid is not None:     # Do this after setting the axes, so the grid is consistent
    366                 self.setGrid(newgrid)

~/bin/anaconda2/envs/cdat8/lib/python3.6/site-packages/cdms2/tvariable.py in setAxis(self, n, axis, savegrid)
    397             raise CDMSError(
    398                 "axis length %d does not match corresponding dimension %d" %
--> 399                 (len(axis), axislen))
    400         if not isinstance(axis, AbstractAxis):
    401             raise CDMSError("copydimension, other not a slab.")

CDMSError: axis length 1140 does not match corresponding dimension 1739

Environment Information

`conda info`

``` $ conda info active environment : cdat8 active env location : /export_backup/pochedley1/bin/anaconda2/envs/cdat8 shell level : 1 user config file : /export_backup/pochedley1/.condarc populated config files : /export_backup/pochedley1/.condarc conda version : 4.5.11 conda-build version : 3.10.9 python version : 3.5.5.final.0 base environment : /export_backup/pochedley1/bin/anaconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/pro/linux-64 https://repo.anaconda.com/pkgs/pro/noarch package cache : /export_backup/pochedley1/bin/anaconda3/pkgs /export_backup/pochedley1/.conda/pkgs envs directories : /export_backup/pochedley1/bin/anaconda2/envs /export_backup/pochedley1/bin/anaconda3/envs /export_backup/pochedley1/.conda/envs platform : linux-64 user-agent : conda/4.5.11 requests/2.19.1 CPython/3.5.5 Linux/2.6.32-754.14.2.el6.x86_64 rhel/6.10 glibc/2.12 UID:GID : 89674:1026 netrc file : /export_backup/pochedley1/.netrc offline mode : False ```

`conda config --show-sources`

``` conda config --show-sources ==> /export_backup/pochedley1/.condarc <== ssl_verify: False envs_dirs: - /export_backup/pochedley1/bin/anaconda2/envs/ ```

`conda list --show-channel-urls`

``` $ conda list --show-channel-urls # packages in environment at /export_backup/pochedley1/bin/anaconda2/envs/cdat8: # # Name Version Build Channel alabaster 0.7.11 py_3 conda-forge appdirs 1.4.3 py_1 conda-forge asn1crypto 0.24.0 py36_2 conda-forge astroid 1.6.5 py36_0 conda-forge attrs 18.1.0 py_1 conda-forge automat 0.7.0 py36_0 conda-forge autopep8 1.3.4 py_0 conda-forge babel 2.6.0 py_1 conda-forge backcall 0.1.0 py_0 conda-forge basemap 1.1.0 py36h50ae964_5 conda-forge blas 1.0 mkl defaults bleach 2.1.3 py_0 conda-forge bokeh 0.13.0 py36_0 conda-forge bzip2 1.0.6 h470a237_2 conda-forge ca-certificates 2018.11.29 ha4d7672_0 conda-forge cdat 8.0 0 cdat cdat_info 8.0 py36_0 conda-forge cdms2 3.0.1 py36h6091dcd_1 conda-forge cdp 1.3.3 py36_0 cdat cdtime 3.0 py36h6091dcd_1 conda-forge cdutil 8.0 py36_1 conda-forge certifi 2018.11.29 py36_1000 conda-forge cffi 1.11.5 py36h5e8e0c9_1 conda-forge chardet 3.0.4 py36_3 conda-forge cibots 0.2 py36_0 conda-forge clapack 3.2.1 0 conda-forge click 6.7 py_1 conda-forge cloudpickle 0.5.3 py_0 conda-forge constantly 15.1.0 py_0 conda-forge cryptography 2.3.1 py36hdffb7b8_0 conda-forge cryptography-vectors 2.3.1 py36_0 conda-forge curl 7.61.0 h93b3f91_1 conda-forge cycler 0.10.0 py_1 conda-forge cytoolz 0.9.0.1 py36h470a237_0 conda-forge dask 0.18.2 py_0 conda-forge dask-core 0.18.2 py_0 conda-forge dbus 1.13.0 h3a4f0e9_0 conda-forge decorator 4.3.0 py_0 conda-forge distarray 2.12.2 py36_0 conda-forge distributed 1.22.1 py36_0 conda-forge docutils 0.14 py36_0 conda-forge dv3d 8.0 py36_0 cdat entrypoints 0.2.3 py36_1 conda-forge eofs 1.3.0 py_1 conda-forge esmf 7.1.0r 1 conda-forge esmpy 7.1.0r py36_1 conda-forge expat 2.2.5 hfc679d8_1 conda-forge ffmpeg 3.4.1 0 conda-forge flake8 3.5.0 py36_0 conda-forge fontconfig 2.13.0 h65d0f4c_5 conda-forge freetype 2.9.1 h6debe1e_0 conda-forge future 0.16.0 py36_2 conda-forge g2clib 1.6.0 3 conda-forge genutil 8.0 py36_1 conda-forge geos 3.6.2 hfc679d8_2 conda-forge gettext 0.19.8.1 0 conda-forge glib 2.55.0 h464dc38_2 conda-forge gmp 6.1.2 hfc679d8_0 conda-forge gst-plugins-base 1.12.5 hde13a9d_0 conda-forge gstreamer 1.12.5 h61a6719_0 conda-forge hdf4 4.2.13 0 conda-forge hdf5 1.10.1 2 conda-forge heapdict 1.0.0 py36_0 conda-forge html5lib 1.0.1 py_0 conda-forge hyperlink 17.3.1 py_0 conda-forge icu 58.2 hfc679d8_0 conda-forge idna 2.7 py36_2 conda-forge imagesize 1.0.0 py_1 conda-forge incremental 17.5.0 py_0 conda-forge intel-openmp 2018.0.3 0 defaults ipykernel 4.8.2 py36_0 conda-forge ipython 6.5.0 py36_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.4.0 py_0 conda-forge isort 4.3.4 py36_0 conda-forge jasper 1.900.1 4 conda-forge jedi 0.12.1 py36_0 conda-forge jeepney 0.3.1 py_0 conda-forge jinja2 2.10 py_1 conda-forge joblib 0.12.2 py_0 conda-forge jpeg 9c h470a237_0 conda-forge jsonschema 2.6.0 py36_1 conda-forge jupyter 1.0.0 py_1 conda-forge jupyter_client 5.2.3 py_1 conda-forge jupyter_console 5.2.0 py36_0 conda-forge jupyter_core 4.4.0 py_0 conda-forge keyring 13.2.1 py36_0 conda-forge kiwisolver 1.0.1 py36h2d50403_2 conda-forge krb5 1.14.6 0 conda-forge lapack 3.6.1 1 conda-forge lazy-object-proxy 1.3.1 py36h470a237_0 conda-forge libcdms 3.0.1 h9ac9557_2 conda-forge libcf 1.0.1 py36_5 conda-forge libdrs 3.0.1 h6e3784b_3 conda-forge libdrs_f 3.0.1 h2cd7f18_3 conda-forge libffi 3.2.1 hfc679d8_4 conda-forge libgcc 7.2.0 h69d50b8_2 defaults libgcc-ng 7.3.0 hdf63c60_0 conda-forge libgfortran 3.0.0 1 conda-forge libgfortran-ng 7.2.0 hdf63c60_3 conda-forge libiconv 1.15 h470a237_2 conda-forge libnetcdf 4.6.1 2 conda-forge libopenblas 0.2.20 h9ac9557_7 defaults libpng 1.6.35 ha92aebf_0 conda-forge libprotobuf 3.6.0 hd28b015_0 conda-forge libsodium 1.0.16 0 conda-forge libssh2 1.8.0 h5b517e9_2 conda-forge libstdcxx-ng 7.2.0 hdf63c60_3 conda-forge libtiff 4.0.9 he6b73bb_1 conda-forge libuuid 2.32.1 h470a237_0 conda-forge libxcb 1.13 h470a237_2 conda-forge libxml2 2.9.8 h422b904_3 conda-forge locket 0.2.0 py_2 conda-forge markupsafe 1.0 py36_0 conda-forge matplotlib 2.2.3 py36h8e2386c_0 conda-forge mccabe 0.6.1 py_1 conda-forge mesalib 17.3.9 hdd5ec5b_0 conda-forge mistune 0.8.3 py36h470a237_2 conda-forge mkl 2018.0.3 1 defaults mkl_fft 1.0.6 py36_0 conda-forge mkl_random 1.0.1 py36_0 conda-forge mpi 1.0 mpich conda-forge mpich 3.2.1 h26a2512_4 conda-forge msgpack-python 0.5.6 py36h2d50403_2 conda-forge msu 1.0 mysql-connector-python 2.2.3 py36_0 conda-forge nb_conda 2.2.1 py36_0 conda-forge nb_conda_kernels 2.1.1 py36_1 conda-forge nbconvert 5.3.1 py_1 conda-forge nbformat 4.4.0 py_1 conda-forge ncurses 6.1 hfc679d8_1 conda-forge netcdf-fortran 4.4.4 7 conda-forge notebook 5.6.0 py36_0 conda-forge numpy 1.15.0 py36h1b885b7_0 defaults numpy-base 1.15.0 py36h3dfced4_0 defaults numpydoc 0.8.0 py_1 conda-forge olefile 0.45.1 py36_0 defaults openssl 1.0.2p h14c3975_1002 conda-forge ossuuid 1.6.2 hfc679d8_0 conda-forge output_viewer 1.2.5 py36_0 cdat packaging 17.1 py_0 conda-forge palettable 3.1.1 py_0 conda-forge pandas 0.23.4 py36hf8a1672_0 conda-forge pandoc 2.2.2 1 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.3.1 py_0 conda-forge partd 0.3.8 py_1 conda-forge pcre 8.41 h470a237_2 conda-forge pexpect 4.6.0 py36_0 conda-forge pickleshare 0.7.4 py36_0 conda-forge pillow 5.2.0 py36heded4f4_0 defaults pip 18.0 py36_1 conda-forge proj4 4.9.3 5 conda-forge prometheus_client 0.3.0 py_0 conda-forge prompt_toolkit 1.0.15 py36_0 conda-forge protobuf 3.6.0 py36hfc679d8_0 conda-forge psutil 5.4.7 py36_0 conda-forge pthread-stubs 0.4 h470a237_1 conda-forge ptyprocess 0.6.0 py36_0 conda-forge pyasn1 0.4.4 py_0 conda-forge pyasn1-modules 0.2.1 py_0 conda-forge pycodestyle 2.3.1 py36_0 conda-forge pycparser 2.18 py_1 conda-forge pyflakes 1.6.0 py36_0 conda-forge pygments 2.2.0 py_1 conda-forge pyhamcrest 1.9.0 py_2 conda-forge pylibconfig2 0.2.5 pylint 1.9.2 py36_0 conda-forge pyopenssl 18.0.0 py36_0 conda-forge pyparsing 2.2.0 py_1 conda-forge pyproj 1.9.5.1 py36_0 conda-forge pyqt 5.6.0 py36h8210e8a_6 conda-forge pyshp 1.2.12 py_0 conda-forge pysocks 1.6.8 py36_1 conda-forge pysolar 0.8 py_0 conda-forge pyspharm 1.0.9 py36_3 conda-forge python 3.6.6 h5001a0f_0 conda-forge python-dateutil 2.7.3 py_0 conda-forge pytz 2018.5 py_0 conda-forge pyyaml 3.12 py36_1 conda-forge pyzmq 17.1.2 py36hae99301_0 conda-forge qt 5.6.2 hf70d934_9 conda-forge qtawesome 0.4.4 pyh8a2030e_1 conda-forge qtconsole 4.4.0 py36h8a2030e_0 conda-forge qtpy 1.4.2 pyh8a2030e_1 conda-forge readline 7.0 haf1bffa_1 conda-forge requests 2.19.1 py36_1 conda-forge rope 0.10.7 py_1 conda-forge scandir 1.9.0 py36h470a237_0 conda-forge scipy 1.1.0 py36hfc37229_0 defaults secretstorage 3.0.1 py36_0 conda-forge send2trash 1.5.0 py_0 conda-forge service_identity 17.0.0 py_0 conda-forge setuptools 40.0.0 py36_1 conda-forge simplegeneric 0.8.1 py_1 conda-forge sip 4.18 py36_1 conda-forge six 1.11.0 py36_1 conda-forge snowballstemmer 1.2.1 py_1 conda-forge sortedcontainers 2.0.4 py_1 conda-forge sphinx 1.7.5 py36_0 conda-forge sphinxcontrib-websupport 1.0.1 py36_0 conda-forge spyder 3.3.1 py36_1 conda-forge spyder-kernels 0.2.6 py_0 conda-forge sqlite 3.24.0 h2f33b56_0 conda-forge tblib 1.3.2 py_1 conda-forge terminado 0.8.1 py36_0 conda-forge testpath 0.3.1 py36_0 conda-forge thermo 8.0 py36_0 cdat tk 8.6.8 0 conda-forge toolz 0.9.0 py_0 conda-forge tornado 5.1 py36h470a237_1 conda-forge tqdm 4.25.0 py36h28b3542_0 defaults traitlets 4.3.2 py36_0 conda-forge twisted 18.7.0 py36h14c3975_1 defaults udunits2 2.2.27.6 h3a4f0e9_0 conda-forge urllib3 1.23 py36_1 conda-forge vcs 8.0 py36_0 cdat vcsaddons 8.0 py36_0 cdat vtk-cdat 8.0.1.8.0 py36_mesalib_1 [mesalib] cdat wcwidth 0.1.7 py_1 conda-forge webencodings 0.5 py36_0 conda-forge wheel 0.31.1 py36_1 conda-forge widgetsnbextension 3.4.0 py36_0 conda-forge windspharm 1.6.0 py36_0 conda-forge wk 8.0 py36_0 cdat wrapt 1.10.11 py36_0 conda-forge x264 20131218 0 conda-forge xorg-libxau 1.0.8 h470a237_6 conda-forge xorg-libxdmcp 1.1.2 h470a237_7 conda-forge xz 5.2.4 h470a237_1 conda-forge yaml 0.1.7 h470a237_1 conda-forge zeromq 4.2.5 hfc679d8_5 conda-forge zict 0.1.3 py_0 conda-forge zlib 1.2.11 h470a237_3 conda-forge zope.interface 4.5.0 py36h470a237_0 conda-forge ```

pochedls commented 5 years ago

@durack1 @dnadeau4 - Have you seen this one before? I think I have, but I can't remember how I got past it.

durack1 commented 5 years ago

@pochedls is it possible a file(s) disappeared between when the cdscan xml was generated and when you tried to read it?

@doutriaux1

doutriaux1 commented 5 years ago

@durack1 yep that's my first guess.

pochedls commented 5 years ago

Hi @durack1 and @doutriaux1 - I don't think this is the case - the xml software is supposed to delete xmls when the underlying data changes. I just regenerated the xml for the CMIP5 data and I get the same error: 

$ conda activate cdat8
$ cdscan -x test.xml /p/css03/scratch/unknown-status/cmip5/output1/NCC/NorESM1-ME/rcp85/mon/ocnBgchem/Omon/r1i1p1/v20130926/spco2/*.nc
$ ipython
In [1]: import cdms2
In [2]: f = cdms2.open('test.xml')
In [3]: data = f('spco2')
.
.
.
CDMSError: axis length 1140 does not match corresponding dimension 1739
durack1 commented 5 years ago

@pochedls thanks for updating this, as it's a bug that occurs between cdscan and cdms2 reads this should be a high priority fix!

@dnadeau4 @doutriaux1

dnadeau4 commented 5 years ago

@pochedls Can you tell me if it works with py2?

pochedls commented 5 years ago

@dnadeau4 - I tried this with cdat8 / Python 2 and I get the same error message.