search

CDAT / cdms

8 stars 10 forks source link

Error when high-temporal-resolution (3hourly) data reading in #419

Closed lee1043 closed 3 years ago

lee1043 commented 3 years ago

Describe the bug A clear and concise description of what the bug is.

Noticed that when using cdms2 to open xml for 3 hourly data, it does not correctly road the data to array.

To Reproduce In gates:

import cdms2
import cdtime
f = cdms2.open('/p/user_pub/pmp/pmp_results/pmp_v1.1.2/additional_xmls/latest/v20200420/cmip6/historical/atmos/3hr/pr/cmip6.historical.MPI-ESM1-2-HR.r1i1p1f1.3hr.pr.xml')
d = f('pr')

Expected behavior The actual data should be loaded as d, 3d array (time, lon, lat)

Screenshots or traceback I am getting following error instead:

ncvargets: ncid 65536: NetCDF: HDF error
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/export/lee1043/anaconda3/envs/cdat82_20200819_py37/lib/python3.7/site-packages/cdms2/cudsinterface.py", line 34, in __call__
    return v(*args, **kwargs)
  File "/export/lee1043/anaconda3/envs/cdat82_20200819_py37/lib/python3.7/site-packages/cdms2/avariable.py", line 197, in __call__
    grid=grid)
  File "/export/lee1043/anaconda3/envs/cdat82_20200819_py37/lib/python3.7/site-packages/cdms2/selectors.py", line 211, in unmodified_select
    grid=grid, raw=raw)
  File "/export/lee1043/anaconda3/envs/cdat82_20200819_py37/lib/python3.7/site-packages/cdms2/avariable.py", line 969, in subRegion
    return self.subSlice(*slicelist, **d)
  File "/export/lee1043/anaconda3/envs/cdat82_20200819_py37/lib/python3.7/site-packages/cdms2/avariable.py", line 732, in subSlice
    d = self.expertSlice(slicelist)
  File "/export/lee1043/anaconda3/envs/cdat82_20200819_py37/lib/python3.7/site-packages/cdms2/variable.py", line 602, in expertSlice
    chunk = var.getitem(*tuple(slicelist))
OSError: NetCDF: HDF error

Desktop (please complete the following information):

Environment Information

`conda info`

``` active environment : cdat82_20200819_py37 active env location : /export/lee1043/anaconda3/envs/cdat82_20200819_py37 shell level : 2 user config file : /export/lee1043/.condarc populated config files : conda version : 4.8.3 conda-build version : 3.18.8 python version : 3.7.3.final.0 virtual packages : __glibc=2.17 base environment : /export/lee1043/anaconda3 (writable) channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /export/lee1043/anaconda3/pkgs /export/lee1043/.conda/pkgs envs directories : /export/lee1043/anaconda3/envs /export/lee1043/.conda/envs platform : linux-64 user-agent : conda/4.8.3 requests/2.24.0 CPython/3.7.3 Linux/3.10.0-1127.13.1.el7.x86_64 rhel/7.8 glibc/2.17 UID:GID : 55431:1026 netrc file : None offline mode : False ```

`conda list --show-channel-urls`

``` # packages in environment at /export/lee1043/anaconda3/envs/cdat82_20200819_py37: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge argon2-cffi 20.1.0 py37h8f50634_1 conda-forge attrs 19.3.0 py_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.1 py_0 conda-forge bleach 3.1.5 pyh9f0ad1d_0 conda-forge bokeh 2.1.1 py37hc8dfbb8_0 conda-forge boost-cpp 1.70.0 h8e57a91_2 conda-forge brotlipy 0.7.0 py37h8f50634_1000 conda-forge bzip2 1.0.8 h516909a_2 conda-forge c-ares 1.16.1 h516909a_0 conda-forge ca-certificates 2020.6.20 hecda079_0 conda-forge cdat 8.2.1 0 cdat/label/v8.2.1 cdat_info 8.2.1 pyh9f0ad1d_1 conda-forge cdms2 3.1.5 pypi_0 pypi cdp 1.6.0 py_0 conda-forge cdtime 3.1.4 py37hd741776_0 conda-forge cdutil 8.2.1 pyh9f0ad1d_0 conda-forge certifi 2020.6.20 py37hc8dfbb8_0 conda-forge cffi 1.14.1 py37h2b28604_0 conda-forge chardet 3.0.4 py37hc8dfbb8_1006 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge cloudpickle 1.5.0 py_0 conda-forge cryptography 3.0 py37hb09aad4_0 conda-forge curl 7.71.1 he644dc0_5 conda-forge cycler 0.10.0 py_2 conda-forge cytoolz 0.10.1 py37h516909a_0 conda-forge dask 2.23.0 py_0 conda-forge dask-core 2.23.0 py_0 conda-forge dbus 1.13.6 he372182_0 conda-forge decorator 4.4.2 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distarray 2.12.2 py_1 conda-forge distributed 2.23.0 py37hc8dfbb8_0 conda-forge dv3d 8.2.1 pyh9f0ad1d_0 cdat/label/v8.2.1 entrypoints 0.3 py37hc8dfbb8_1001 conda-forge esmf 8.0.1 mpi_openmpi_hda7c4e6_0 conda-forge esmpy 8.0.1 mpi_openmpi_py37ha8a4988_0 conda-forge expat 2.2.9 he1b5a44_2 conda-forge ffmpeg 4.2.3 h167e202_0 conda-forge fontconfig 2.13.1 h86ecdb6_1001 conda-forge freetype 2.10.2 he06d7ca_0 conda-forge fsspec 0.8.0 py_0 conda-forge future 0.18.2 py37hc8dfbb8_1 conda-forge g2clib 1.6.0 hf3f1b0b_9 conda-forge genutil 8.2.1 py37h161383b_0 conda-forge gettext 0.19.8.1 hc5be6a0_1002 conda-forge ghostscript 9.22 hf484d3e_1001 conda-forge glib 2.65.0 h6f030ca_0 conda-forge gmp 6.2.0 he1b5a44_2 conda-forge gnutls 3.6.13 h79a8f9a_0 conda-forge gst-plugins-base 1.14.5 h0935bb2_2 conda-forge gstreamer 1.14.5 h36ae1b5_2 conda-forge hdf4 4.2.13 hf30be14_1003 conda-forge hdf5 1.10.6 mpi_openmpi_hac320be_1 conda-forge heapdict 1.0.1 py_0 conda-forge icu 64.2 he1b5a44_1 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 1.7.0 py37hc8dfbb8_0 conda-forge importlib_metadata 1.7.0 0 conda-forge ipykernel 5.3.4 py37h43977f1_0 conda-forge ipython 7.17.0 py37hc6149b9_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 py_0 conda-forge jasper 1.900.1 h07fcdf6_1006 conda-forge jedi 0.15.2 py37_0 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge jpeg 9d h516909a_0 conda-forge jsoncpp 1.8.4 hc9558a2_1002 conda-forge jsonschema 3.2.0 py37hc8dfbb8_1 conda-forge jupyter 1.0.0 py_2 conda-forge jupyter_client 6.1.6 py_0 conda-forge jupyter_console 6.1.0 py_1 conda-forge jupyter_core 4.6.3 py37hc8dfbb8_1 conda-forge kiwisolver 1.2.0 py37h99015e2_0 conda-forge krb5 1.17.1 hfafb76e_2 conda-forge lame 3.100 h14c3975_1001 conda-forge lazy-object-proxy 1.5.1 py37h8f50634_0 conda-forge lcms2 2.11 hbd6801e_0 conda-forge ld_impl_linux-64 2.34 hc38a660_9 conda-forge libblas 3.8.0 17_openblas conda-forge libcblas 3.8.0 17_openblas conda-forge libcdms 3.1.2 h054cd8a_112 conda-forge libcf 1.0.3 py37hda0e254_108 conda-forge libclang 9.0.1 default_hde54327_0 conda-forge libcurl 7.71.1 hcdd3856_5 conda-forge libdrs 3.1.2 hc2e2db3_112 conda-forge libdrs_f 3.1.2 hae7e664_110 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_0 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 9.3.0 h24d8f2e_15 conda-forge libgfortran-ng 7.5.0 hdf63c60_15 conda-forge libgomp 9.3.0 h24d8f2e_15 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.8.0 17_openblas conda-forge libllvm9 9.0.1 he513fc3_1 conda-forge libnetcdf 4.7.4 mpi_openmpi_h2b047e1_5 conda-forge libnghttp2 1.41.0 hab1572f_1 conda-forge libopenblas 0.3.10 pthreads_hb3c22a3_4 conda-forge libpng 1.6.37 hed695b0_2 conda-forge libsodium 1.0.18 h516909a_0 conda-forge libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.3.0 hdf63c60_15 conda-forge libtiff 4.1.0 hc7e4089_6 conda-forge libuuid 2.32.1 h14c3975_1000 conda-forge libwebp-base 1.1.0 h516909a_3 conda-forge libxcb 1.13 h14c3975_1002 conda-forge libxkbcommon 0.10.0 he1b5a44_0 conda-forge libxml2 2.9.10 hee79883_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.2 he1b5a44_3 conda-forge markupsafe 1.1.1 py37h8f50634_1 conda-forge matplotlib 3.3.1 0 conda-forge matplotlib-base 3.3.1 py37hd478181_0 conda-forge mesalib 18.3.1 h590aaf7_0 conda-forge mistune 0.8.4 py37h8f50634_1001 conda-forge mpi 1.0 openmpi conda-forge mpi4py 3.0.3 py37hbfacf26_1 conda-forge msgpack-python 1.0.0 py37h99015e2_1 conda-forge nbconvert 5.6.1 py37hc8dfbb8_1 conda-forge nbformat 5.0.7 py_0 conda-forge ncurses 6.2 he1b5a44_1 conda-forge netcdf-fortran 4.5.3 mpi_openmpi_h2cce2cb_0 conda-forge nettle 3.4.1 h1bed415_1002 conda-forge notebook 6.1.3 py37hc8dfbb8_0 conda-forge nspr 4.27 he1b5a44_1 conda-forge nss 3.55 he751ad9_0 conda-forge numpy 1.19.1 py37h8960a57_0 conda-forge olefile 0.46 py_0 conda-forge openblas 0.3.10 pthreads_hf183345_4 conda-forge openh264 2.1.1 h8b12597_0 conda-forge openmpi 4.0.4 hdf1f1ad_0 conda-forge openssh 8.3p1 h5957347_0 conda-forge openssl 1.1.1g h516909a_1 conda-forge output_viewer 1.3.1 py_1 conda-forge packaging 20.4 pyh9f0ad1d_0 conda-forge pandas 1.1.0 py37h3340039_0 conda-forge pandoc 2.10.1 h516909a_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.5.2 py_0 defaults partd 1.1.0 py_0 conda-forge pcre 8.44 he1b5a44_0 conda-forge pexpect 4.8.0 py37hc8dfbb8_1 conda-forge pickleshare 0.7.5 py37hc8dfbb8_1001 conda-forge pillow 7.2.0 py37h718be6c_1 conda-forge pip 20.2.2 py_0 conda-forge prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge prompt-toolkit 3.0.6 py_0 conda-forge prompt_toolkit 3.0.6 0 conda-forge psutil 5.7.2 py37h8f50634_0 conda-forge pthread-stubs 0.4 h14c3975_1001 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pygments 2.6.1 py_0 conda-forge pyopenssl 19.1.0 py_1 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyqt 5.12.3 py37h8685d9f_3 conda-forge pyqt5-sip 4.19.18 pypi_0 pypi pyqtchart 5.12 pypi_0 pypi pyqtwebengine 5.12.1 pypi_0 pypi pyrsistent 0.16.0 py37h8f50634_0 conda-forge pysocks 1.7.1 py37hc8dfbb8_1 conda-forge python 3.7.8 h6f2ec95_1_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python_abi 3.7 1_cp37m conda-forge pytz 2020.1 pyh9f0ad1d_0 conda-forge pyyaml 5.3.1 py37h8f50634_0 conda-forge pyzmq 19.0.2 py37hac76be4_0 conda-forge qt 5.12.5 hd8c4c69_1 conda-forge qtconsole 4.7.6 pyh9f0ad1d_0 conda-forge qtpy 1.9.0 py_0 conda-forge readline 8.0 he28a2e2_2 conda-forge regrid2 3.1.5 pypi_0 pypi requests 2.24.0 pyh9f0ad1d_0 conda-forge scipy 1.5.2 py37hb14ef9d_0 conda-forge send2trash 1.5.0 py_0 conda-forge setuptools 49.6.0 py37hc8dfbb8_0 conda-forge six 1.15.0 pyh9f0ad1d_0 conda-forge sortedcontainers 2.2.2 pyh9f0ad1d_0 conda-forge sqlite 3.33.0 h4cf870e_0 conda-forge tbb 2020.1 hc9558a2_0 conda-forge tbb-devel 2020.1 hc9558a2_0 conda-forge tblib 1.6.0 py_0 conda-forge terminado 0.8.3 py37hc8dfbb8_1 conda-forge testpath 0.4.4 py_0 conda-forge tk 8.6.10 hed695b0_0 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.4 py37h8f50634_1 conda-forge traitlets 4.3.3 py37hc8dfbb8_1 conda-forge typing_extensions 3.7.4.2 py_0 conda-forge udunits2 2.2.27.6 h4e0c4b3_1001 conda-forge urllib3 1.25.10 py_0 conda-forge vcs 8.2.1 pyh9f0ad1d_0 cdat/label/v8.2.1 vcsaddons 8.2.1 py37h8f50634_0 cdat/label/v8.2.1 vtk-cdat 8.2.0.8.2.1 py37_mesalibh95575ff_0 [mesalib] cdat/label/v8.2.1 wcwidth 0.2.5 pyh9f0ad1d_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.35.1 pyh9f0ad1d_0 conda-forge widgetsnbextension 3.5.1 py37hc8dfbb8_1 conda-forge wk 8.2.1 pyh9f0ad1d_0 cdat/label/v8.2.1 x264 1!152.20180806 h14c3975_0 conda-forge xorg-kbproto 1.0.7 h14c3975_1002 conda-forge xorg-libice 1.0.10 h516909a_0 conda-forge xorg-libsm 1.2.3 h84519dc_1000 conda-forge xorg-libx11 1.6.11 h516909a_0 conda-forge xorg-libxau 1.0.9 h14c3975_0 conda-forge xorg-libxdmcp 1.1.3 h516909a_0 conda-forge xorg-libxt 1.2.0 h516909a_0 conda-forge xorg-xproto 7.0.31 h14c3975_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge zeromq 4.3.2 he1b5a44_3 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.1.0 py_0 conda-forge zlib 1.2.11 h516909a_1007 conda-forge zstd 1.4.5 h6597ccf_2 conda-forge ```

lee1043 commented 3 years ago

Maybe it was a temporary issue perhaps something related to network. It is working now with no error so I am closing the issue.

jasonb5 commented 3 years ago

@lee1043 I just experience an issue running ls on the directory, which has cleared up now, so I think you're correct with it being related to the network.

lee1043 commented 3 years ago

@jasonb5 good to know! Thank you for your comment.

Docetom commented 3 years ago

@lee1043, please can you explain how i can use the cdutil.averager to average hourly data into daily mean? I want to use cdat so that I can retain the metadata in the file. The file has JJAS hourly data so the expected shape of my file should be (122,9,181,281). I also do not want to use CDO nor Pandas to do this averaging.

Here is what I have tried to do

import cdms2,cdutil f = cdms2.open('ERA5_hourly_plevels_geopotential_jjas_1960.nc') g_p = f('z') g_p.shape (2928, 9, 181, 281); ## AXIS_dim = (time, level,latitude, longitude)

cdutil.setTimeBoundsDaily(g_p, frquency =24) aa = MV2.average(g_p, axis=0)

kindly help me out. thanks

lee1043 commented 3 years ago

Hi @Docetom, in my best knowledge CDAT/cdutil unfortunately does not offer the capability of averaging an hourly data set into a daily mean. I think you may have to write a customized function for your own purpose.

@CDAT/cdms-developers Please correct me if I am wrong.

Docetom commented 3 years ago

Ok. Thanks.

On Mon, Dec 21, 2020, 3:56 AM Jiwoo Lee notifications@github.com wrote:

Hi @Docetom https://github.com/Docetom, in my best knowledge CDAT/cdutil unfortunately does not offer the capability of averaging an hourly data set into a daily mean. I think you may have to write a customized function for your own purpose.

@CDAT/cdms-developers https://github.com/orgs/CDAT/teams/cdms-developers Please correct me if I am wrong.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CDAT/cdms/issues/419#issuecomment-748679535, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR7RDAAM6ND2T6HUEVQ3RDDSVZ22LANCNFSM4QFK6LDA .