Opening grb files fails

HelixPiano commented 1 year ago

What happened?

Hello everyone, I am not sure if this a bug in xarray or a bug with cfgrib, I will therefore crosspost it.

I have a grb file with the dimension 30316x160x392 , filetype float32 and filesize of around 3.7GB.

df= xr.open_dataset("129.grb", engine="cfgrib") works initially. The problem is when I call df.max() it maxes out the RAM of my PC and fails to return any result. RAM usage before df.max() call: 3.5/16GB

If I run df= xr.load_dataset("129.grb", engine="cfgrib") instead I get an error message:

What did you expect to happen?

Work with the file as usual

Minimal Complete Verifiable Example

I can't provide a file of this size easily

MVCE confirmation

[ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[ ] Complete example — the example is self-contained, including all data and the text of any traceback.
[ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[x] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.3.2\plugins\python-ce\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\backends\api.py", line 264, in load_dataset
    return ds.load()
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\dataset.py", line 760, in load
    v.load()
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\variable.py", line 539, in load
    self._data = self._data.get_duck_array()
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\indexing.py", line 695, in get_duck_array
    self._ensure_cached()
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\indexing.py", line 689, in _ensure_cached
    self.array = as_indexable(self.array.get_duck_array())
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\indexing.py", line 663, in get_duck_array
    return self.array.get_duck_array()
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\indexing.py", line 550, in get_duck_array
    array = self.array[self.key]
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\xarray_plugin.py", line 156, in __getitem__
    return xr.core.indexing.explicit_indexing_adapter(
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\xarray\core\indexing.py", line 857, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\xarray_plugin.py", line 165, in _getitem
    return self.array[key]
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\dataset.py", line 354, in __getitem__
    message = self.index.get_field(message_ids[0])  # type: ignore
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\messages.py", line 484, in get_field
    return ComputedKeysAdapter(self.fieldset[message_id], self.computed_keys)
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\messages.py", line 344, in __getitem__
    return self.message_from_file(file, offset=item)
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\messages.py", line 340, in message_from_file
    return Message.from_file(file, offset, **kwargs)
  File "C:\Users\xxx\miniconda3\envs\cnn\lib\site-packages\cfgrib\messages.py", line 93, in from_file
    file.seek(offset)
OSError: [Errno 22] Invalid argument

Anything else we need to know?

No response

Environment

absl-py 1.4.0 pypi_0 pypi
ansicon 1.89.0 py310h5588dad_6 conda-forge anyio 3.6.2 pyhd8ed1ab_0 conda-forge arrow 1.2.3 pyhd8ed1ab_0 conda-forge arrow-cpp 12.0.0 h57928b3_1_cpu conda-forge attrs 23.1.0 pyh71513ae_0 conda-forge aws-c-auth 0.6.26 h0e8f9f7_6 conda-forge aws-c-cal 0.5.26 h3200b0f_0 conda-forge aws-c-common 0.8.17 hcfcfb64_0 conda-forge aws-c-compression 0.2.16 h50f2975_6 conda-forge aws-c-event-stream 0.2.20 h03f8742_6 conda-forge aws-c-http 0.7.7 h1ea9cf2_3 conda-forge aws-c-io 0.13.21 h09d0bb2_3 conda-forge aws-c-mqtt 0.8.6 h9028ce9_15 conda-forge aws-c-s3 0.2.8 h1059ad0_4 conda-forge aws-c-sdkutils 0.1.9 h50f2975_1 conda-forge aws-checksums 0.1.14 h50f2975_6 conda-forge aws-crt-cpp 0.19.9 hd59d7e4_5 conda-forge aws-sdk-cpp 1.10.57 h44051fd_10 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge blas 2.116 mkl conda-forge blas-devel 3.9.0 16_win64_mkl conda-forge blessed 1.19.1 pyh95a074a_2 conda-forge bokeh 3.1.0 pyhd8ed1ab_0 conda-forge boltons 23.0.0 pyhd8ed1ab_0 conda-forge bottleneck 1.3.7 py310h9b08ddd_0 conda-forge brotli 1.0.9 hcfcfb64_8 conda-forge brotli-bin 1.0.9 hcfcfb64_8 conda-forge brotlipy 0.7.0 py310h8d17308_1005 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge c-ares 1.18.1 h8ffe710_0 conda-forge ca-certificates 2023.5.7 h56e8100_0 conda-forge cachecontrol 0.12.11 pyhd8ed1ab_1 conda-forge cachetools 5.3.0 pypi_0 pypi certifi 2023.5.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h628cb3f_3 conda-forge cfgrib 0.9.10.3 pyhd8ed1ab_0 conda-forge cftime 1.6.2 py310h9b08ddd_1 conda-forge charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge cleo 2.0.1 pyhd8ed1ab_0 conda-forge click 8.1.3 win_pyhd8ed1ab_2 conda-forge cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge conda 23.3.1 py310h5588dad_0 conda-forge conda-package-handling 2.0.2 pyh38be061_0 conda-forge conda-package-streaming 0.7.0 pyhd8ed1ab_1 conda-forge contourpy 1.0.7 py310h232114e_0 conda-forge crashtest 0.4.1 pyhd8ed1ab_0 conda-forge croniter 1.3.14 pyhd8ed1ab_0 conda-forge cryptography 40.0.2 py310h6e82f81_0 conda-forge cuda 11.7.1 0 nvidia cuda-cccl 11.7.91 0 nvidia cuda-command-line-tools 11.7.1 0 nvidia cuda-compiler 11.7.1 0 nvidia cuda-cudart 11.7.99 0 nvidia cuda-cudart-dev 11.7.99 0 nvidia cuda-cuobjdump 11.7.91 0 nvidia cuda-cupti 11.7.101 0 nvidia cuda-cuxxfilt 11.7.91 0 nvidia cuda-demo-suite 12.0.140 0 nvidia cuda-documentation 12.0.140 0 nvidia cuda-libraries 11.7.1 0 nvidia cuda-libraries-dev 11.7.1 0 nvidia cuda-memcheck 11.8.86 0 nvidia cuda-nsight-compute 12.0.1 0 nvidia cuda-nvcc 11.7.99 0 nvidia cuda-nvdisasm 12.0.140 0 nvidia cuda-nvml-dev 11.7.91 0 nvidia cuda-nvprof 12.0.146 0 nvidia cuda-nvprune 11.7.91 0 nvidia cuda-nvrtc 11.7.99 0 nvidia cuda-nvrtc-dev 11.7.99 0 nvidia cuda-nvtx 11.7.91 0 nvidia cuda-nvvp 12.0.146 0 nvidia cuda-runtime 11.7.1 0 nvidia cuda-sanitizer-api 12.0.140 0 nvidia cuda-toolkit 11.7.1 0 nvidia cuda-tools 11.7.1 0 nvidia cuda-version 12.0 hffde075_2 conda-forge cuda-visual-tools 11.7.1 0 nvidia curl 8.0.1 h68f0423_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.0 py310h8d17308_1 conda-forge dask 2023.4.1 pyhd8ed1ab_0 conda-forge dask-core 2023.4.1 pyhd8ed1ab_0 conda-forge dateutils 0.6.12 py_0 conda-forge deepdiff 6.3.0 pyhd8ed1ab_0 conda-forge distlib 0.3.6 pyhd8ed1ab_0 conda-forge distributed 2023.4.1 pyhd8ed1ab_0 conda-forge dulwich 0.20.50 py310h8d17308_0 conda-forge eccodes 2.29.0 h76b5364_1 conda-forge fastapi 0.88.0 pyhd8ed1ab_0 conda-forge filelock 3.12.0 pyhd8ed1ab_0 conda-forge findlibs 0.0.5 pyhd8ed1ab_0 conda-forge fonttools 4.39.3 py310h8d17308_0 conda-forge freeglut 3.2.2 h0e60522_1 conda-forge freetype 2.12.1 h546665d_1 conda-forge fsspec 2023.4.0 pyh1a96a4e_0 conda-forge gettext 0.21.1 h5728263_0 conda-forge gflags 2.2.2 ha925a31_1004 conda-forge glib 2.76.2 h12be248_0 conda-forge glib-tools 2.76.2 h12be248_0 conda-forge glog 0.6.0 h4797de2_0 conda-forge google-auth 2.17.3 pypi_0 pypi google-auth-oauthlib 1.0.0 pypi_0 pypi grpcio 1.54.0 pypi_0 pypi gst-plugins-base 1.22.0 h001b923_2 conda-forge gstreamer 1.22.0 h6b5321d_2 conda-forge h11 0.14.0 pyhd8ed1ab_0 conda-forge hdf4 4.2.15 h1b1b6ef_5 conda-forge hdf5 1.12.2 nompi_h57737ce_101 conda-forge html5lib 1.1 pyh9f0ad1d_0 conda-forge icu 70.1 h0e60522_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge ignite 0.4.11 py_0 pytorch importlib-metadata 6.6.0 pyha770c72_0 conda-forge importlib_metadata 6.6.0 hd8ed1ab_0 conda-forge importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge inquirer 3.1.3 pyhd8ed1ab_0 conda-forge intel-openmp 2023.1.0 h57928b3_46319 conda-forge itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge jaraco.classes 3.2.3 pyhd8ed1ab_0 conda-forge jasper 4.0.0 hc2e4405_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge jinxed 1.2.0 pyh95a074a_0 conda-forge joblib 1.2.0 pypi_0 pypi jpeg 9e hcfcfb64_3 conda-forge jsonpatch 1.32 pyhd8ed1ab_0 conda-forge jsonpointer 2.0 py_0 conda-forge jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge keyring 23.13.1 py310h5588dad_0 conda-forge kiwisolver 1.4.4 py310h232114e_1 conda-forge krb5 1.20.1 heb0366b_0 conda-forge lcms2 2.15 ha5c8aab_0 conda-forge lerc 4.0.0 h63175ca_0 conda-forge libabseil 20230125.0 cxx17_h63175ca_1 conda-forge libaec 1.0.6 h63175ca_1 conda-forge libarrow 12.0.0 h91bed3e_1_cpu conda-forge libblas 3.9.0 16_win64_mkl conda-forge libbrotlicommon 1.0.9 hcfcfb64_8 conda-forge libbrotlidec 1.0.9 hcfcfb64_8 conda-forge libbrotlienc 1.0.9 hcfcfb64_8 conda-forge libcblas 3.9.0 16_win64_mkl conda-forge libclang 15.0.7 default_h77d9078_1 conda-forge libclang13 15.0.7 default_h77d9078_1 conda-forge libcrc32c 1.1.2 h0e60522_0 conda-forge libcublas 11.10.3.66 0 nvidia libcublas-dev 11.10.3.66 0 nvidia libcufft 10.7.2.124 0 nvidia libcufft-dev 10.7.2.124 0 nvidia libcurand 10.3.1.50 h63175ca_0 conda-forge libcurand-dev 10.3.1.50 h63175ca_0 conda-forge libcurl 8.0.1 h68f0423_0 conda-forge libcusolver 11.4.0.1 0 nvidia libcusolver-dev 11.4.0.1 0 nvidia libcusparse 11.7.4.91 0 nvidia libcusparse-dev 11.7.4.91 0 nvidia libdeflate 1.17 hcfcfb64_0 conda-forge libevent 2.1.12 hf43717d_0 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libglib 2.76.2 he8f3873_0 conda-forge libgoogle-cloud 2.10.0 h00b2bdc_0 conda-forge libgrpc 1.54.1 h32da247_0 conda-forge libhwloc 2.9.1 h51c2c0f_0 conda-forge libiconv 1.17 h8ffe710_0 conda-forge liblapack 3.9.0 16_win64_mkl conda-forge liblapacke 3.9.0 16_win64_mkl conda-forge libnetcdf 4.9.1 nompi_hc41bf00_101 conda-forge libnpp 11.7.4.75 0 nvidia libnpp-dev 11.7.4.75 0 nvidia libnvjpeg 11.8.0.2 0 nvidia libnvjpeg-dev 11.8.0.2 0 nvidia libogg 1.3.4 h8ffe710_1 conda-forge libpng 1.6.39 h19919ed_0 conda-forge libprotobuf 3.21.12 h12be248_0 conda-forge libsqlite 3.40.0 hcfcfb64_1 conda-forge libssh2 1.10.0 h9a1e1f7_3 conda-forge libthrift 0.18.1 h9ce19ad_1 conda-forge libtiff 4.5.0 hf8721a0_2 conda-forge libutf8proc 2.8.0 h82a8f57_0 conda-forge libuv 1.44.2 h8ffe710_0 conda-forge libvorbis 1.3.7 h0e60522_0 conda-forge libwebp-base 1.3.0 hcfcfb64_0 conda-forge libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.10.4 hc3477c8_0 conda-forge libzip 1.9.2 h519de47_1 conda-forge libzlib 1.2.13 hcfcfb64_4 conda-forge lightning 2.0.2 pyhd8ed1ab_0 conda-forge lightning-cloud 0.5.34 pyhd8ed1ab_0 conda-forge lightning-utilities 0.8.0 pyhd8ed1ab_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lockfile 0.12.2 py_1 conda-forge lz4 4.3.2 py310hbbb2075_0 conda-forge lz4-c 1.9.4 hcfcfb64_0 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.4.3 pypi_0 pypi markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.2 py310h8d17308_0 conda-forge matplotlib 3.7.1 py310h5588dad_0 conda-forge matplotlib-base 3.7.1 py310h51140c5_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge menuinst 1.4.19 py310h5588dad_1 conda-forge mkl 2022.1.0 h6a75c08_874 conda-forge mkl-devel 2022.1.0 h57928b3_875 conda-forge mkl-include 2022.1.0 h6a75c08_874 conda-forge more-itertools 9.1.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.5 py310h232114e_0 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge netcdf4 1.6.3 nompi_py310hf10f414_100 conda-forge nsight-compute 2022.4.1.6 0 nvidia numpy 1.24.3 py310hd02465a_0 conda-forge oauthlib 3.2.2 pypi_0 pypi openjpeg 2.5.0 ha2aaf27_2 conda-forge openssl 3.1.0 hcfcfb64_3 conda-forge orc 1.8.3 hada7b9e_0 conda-forge ordered-set 4.1.0 pyhd8ed1ab_0 conda-forge orjson 3.8.12 py310h48ea969_0 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 2.0.1 py310h1c4a608_0 conda-forge parquet-cpp 1.5.1 2 conda-forge partd 1.4.0 pyhd8ed1ab_0 conda-forge pcre2 10.40 h17e33f8_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pillow 9.4.0 py310hdbb7713_1 conda-forge pip 23.1.2 pyhd8ed1ab_0 conda-forge pkginfo 1.9.6 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 2.6.2 pyhd8ed1ab_0 conda-forge pluggy 1.0.0 pyhd8ed1ab_5 conda-forge ply 3.11 py_1 conda-forge poetry 1.3.1 py310h5588dad_0 conda-forge poetry-core 1.4.0 pyhd8ed1ab_0 conda-forge poetry-plugin-export 1.3.1 pyhd8ed1ab_0 conda-forge protobuf 4.22.4 pypi_0 pypi psutil 5.9.5 py310h8d17308_0 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pyarrow 12.0.0 py310hd1a9178_1_cpu conda-forge pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pycosat 0.6.4 py310h8d17308_1 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pydantic 1.10.7 py310h8d17308_0 conda-forge pygments 2.15.1 pyhd8ed1ab_0 conda-forge pyjwt 2.7.0 pyhd8ed1ab_0 conda-forge pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyqt 5.15.7 py310h1fd54f2_3 conda-forge pyqt5-sip 12.11.0 py310h00ffb61_3 conda-forge pyrsistent 0.19.3 py310h8d17308_0 conda-forge pysocks 1.7.1 pyh0701188_6 conda-forge python 3.10.10 h4de0772_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-eccodes 1.5.1 py310h9b08ddd_0 conda-forge python-editor 1.0.4 py_0 conda-forge python-multipart 0.0.6 pyhd8ed1ab_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytorch 1.13.1 py3.10_cuda11.7_cudnn8_0 pytorch pytorch-cuda 11.7 h67b0de4_1 pytorch pytorch-lightning 2.0.2 pyhd8ed1ab_0 conda-forge pytorch-mutex 1.0 cuda pytorch pytz 2023.3 pyhd8ed1ab_0 conda-forge pywin32-ctypes 0.2.0 py310h5588dad_1006 conda-forge pyyaml 6.0 py310h8d17308_5 conda-forge qt-main 5.15.8 h720456b_6 conda-forge rapidfuzz 2.15.1 py310h00ffb61_0 conda-forge re2 2023.02.02 h63175ca_0 conda-forge readchar 4.0.5 pyhd8ed1ab_0 conda-forge requests 2.29.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pypi_0 pypi requests-toolbelt 0.10.1 pyhd8ed1ab_0 conda-forge rich 13.3.5 pyhd8ed1ab_0 conda-forge rsa 4.9 pypi_0 pypi ruamel.yaml 0.17.22 py310h8d17308_0 conda-forge ruamel.yaml.clib 0.2.7 py310h8d17308_1 conda-forge scikit-learn 1.2.2 pypi_0 pypi scipy 1.10.1 pypi_0 pypi setuptools 67.7.2 pyhd8ed1ab_0 conda-forge shellingham 1.5.1 pyhd8ed1ab_0 conda-forge sip 6.7.9 py310h00ffb61_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 hfb803bf_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge starlette 0.22.0 pyhd8ed1ab_0 conda-forge starsessions 1.3.0 pyhd8ed1ab_0 conda-forge tbb 2021.9.0 h91493d7_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge tensorboard 2.13.0 pypi_0 pypi tensorboard-data-server 0.7.0 pypi_0 pypi threadpoolctl 3.1.0 pypi_0 pypi tk 8.6.12 h8ffe710_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tomlkit 0.11.8 pyha770c72_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge torchaudio 0.13.1 pypi_0 pypi torchmetrics 0.11.4 pyhd8ed1ab_0 conda-forge torchvision 0.14.1 pypi_0 pypi tornado 6.3 py310h8d17308_0 conda-forge tqdm 4.65.0 pyhd8ed1ab_1 conda-forge traitlets 5.9.0 pyhd8ed1ab_0 conda-forge trove-classifiers 2023.5.2 pyhd8ed1ab_0 conda-forge typing-extensions 4.5.0 hd8ed1ab_0 conda-forge typing_extensions 4.5.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge ucrt 10.0.22621.0 h57928b3_0 conda-forge unicodedata2 15.0.0 py310h8d17308_0 conda-forge urllib3 1.26.15 pyhd8ed1ab_0 conda-forge uvicorn 0.22.0 py310h5588dad_0 conda-forge vc 14.3 hb25d44b_16 conda-forge vc14_runtime 14.34.31931 h5081d32_16 conda-forge virtualenv 20.21.1 pyhd8ed1ab_0 conda-forge vs2015_runtime 14.34.31931 hed1258a_16 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.5.1 pyhd8ed1ab_0 conda-forge websockets 11.0.3 py310h8d17308_0 conda-forge werkzeug 2.3.3 pypi_0 pypi wheel 0.40.0 pyhd8ed1ab_0 conda-forge win_inet_pton 1.1.0 pyhd8ed1ab_6 conda-forge xarray 2023.4.2 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.9 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xyzservices 2023.2.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.15.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hcfcfb64_4 conda-forge zstandard 0.19.0 py310h0009e47_1 conda-forge zstd 1.5.2 h12be248_6 conda-forge

welcome[bot] commented 1 year ago

Thanks for opening your first issue here at xarray! Be sure to follow the issue template! If you have an idea for a solution, we would really welcome a Pull Request with proposed changes. See the Contributing Guide for more. It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better. Thank you!

notoriusjack commented 1 year ago

I have the same issue where sometimes I get OSError: [Errno 22] Invalid argument.

My code loops through the time dimension called step and sometimes it fails when trying to get to the values of one of the data variables, in my case the variable u10.

print(clipped.u10) shows that the array is not empty but if I try clipped.u10.values that sometimes will give the error.


print(clipped.u10)

<xarray.DataArray 'u10' (step: 1, latitude: 112, longitude: 90)>
dask.array<where, shape=(1, 112, 90), dtype=float32, chunksize=(1, 112, 90), chunktype=numpy.ndarray>
Coordinates:
    time               datetime64[ns] 2023-07-26
  * step               (step) timedelta64[ns] 8 days 18:00:00
    heightAboveGround  float64 10.0
  * latitude           (latitude) float64 45.6 45.2 44.8 44.4 ... 2.0 1.6 1.2
  * longitude          (longitude) float64 114.8 115.2 115.6 ... 150.0 150.4
    valid_time         (step) datetime64[ns] dask.array<chunksize=(1,), meta=np.ndarray>
    spatial_ref        int32 0
Attributes: (12/30)
    GRIB_paramId:                             165
    GRIB_dataType:                            pf
    GRIB_numberOfPoints:                      405900
    GRIB_typeOfLevel:                         heightAboveGround
    GRIB_stepUnits:                           1
    GRIB_stepType:                            instant
    ...                                       ...
    GRIB_shortName:                           10u
    GRIB_totalNumber:                         51
    GRIB_units:                               m s**-1
    long_name:                                10 metre U wind component
    units:                                    m s**-1
    standard_name:                            eastward_wind

print(clipped.u10.values)

Traceback (most recent call last):
  File "C:\Users\Giacomo\PycharmProjects\HIT-v3\try_bufr_enfo.py", line 497, in <module>
    print(clipped.u10.values)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\dataarray.py", line 732, in values
    return self.variable.values
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\variable.py", line 614, in values
    return _as_array_or_item(self._data)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\variable.py", line 314, in _as_array_or_item
    data = np.asarray(data)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\array\core.py", line 1701, in __array__
    x = self.compute()
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\base.py", line 310, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\base.py", line 595, in compute
    results = schedule(dsk, keys, **kwargs)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\threaded.py", line 89, in get
    results = get_async(
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\local.py", line 319, in reraise
    raise exc
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\local.py", line 224, in execute_task
    result = _execute_task(task, data)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\optimization.py", line 992, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 151, in get
    result = _execute_task(task, cache)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\dask\array\core.py", line 126, in getter
    c = np.asarray(c)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\indexing.py", line 484, in __array__
    return np.asarray(self.get_duck_array(), dtype=dtype)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\indexing.py", line 487, in get_duck_array
    return self.array.get_duck_array()
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\indexing.py", line 664, in get_duck_array
    return self.array.get_duck_array()
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\indexing.py", line 551, in get_duck_array
    array = self.array[self.key]
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\xarray_plugin.py", line 155, in __getitem__
    return xr.core.indexing.explicit_indexing_adapter(
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\xarray\core\indexing.py", line 858, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\xarray_plugin.py", line 164, in _getitem
    return self.array[key]
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\dataset.py", line 358, in __getitem__
    message = self.index.get_field(message_ids[0])  # type: ignore
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\messages.py", line 484, in get_field
    return ComputedKeysAdapter(self.fieldset[message_id], self.computed_keys)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\messages.py", line 344, in __getitem__
    return self.message_from_file(file, offset=item)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\messages.py", line 340, in message_from_file
    return Message.from_file(file, offset, **kwargs)
  File "C:\Users\Giacomo\anaconda3\envs\HIT-v3\lib\site-packages\cfgrib\messages.py", line 93, in from_file
    file.seek(offset)
OSError: [Errno 22] Invalid argument

dcherian commented 1 year ago

I would ask over at cfgrib, This is a file reading issue and there's not much we can do about it.

pydata / xarray