Reading grib2 file causes an exception TypeError: illegal subscript type in Python3

[stitovich]

When i reading a grib2 file via: dataset = xarray.open_dataset('example.grib2', engine='pynio') and then trying get a variable with specific index: dataset.EXAMPLE_VARIABLE[1,2] i getting the following error: TypeError: illegal subscript type

Stacktrace:

Traceback (most recent call last): File "", line 1, in File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/common.py", line 100, in repr return formatting.array_repr(self) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/formatting.py", line 393, in array_repr summary.append(short_array_repr(arr.values)) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/dataarray.py", line 412, in values return self.variable.values File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/variable.py", line 396, in values return _as_array_or_item(self._data) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/variable.py", line 217, in _as_array_or_item data = np.asarray(data) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray return array(a, dtype, copy=False, order=order) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 418, in array self._ensure_cached() File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 415, in _ensure_cached self.array = NumpyIndexingAdapter(np.asarray(self.array)) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray return array(a, dtype, copy=False, order=order) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 399, in array return np.asarray(self.array, dtype=dtype) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/numpy/core/numeric.py", line 531, in asarray return array(a, dtype, copy=False, order=order) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 366, in array return np.asarray(array[self.key], dtype=None) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/conventions.py", line 389, in getitem self.scale_factor, self.add_offset, self._dtype) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/conventions.py", line 74, in mask_andscale values = np.array(array, dtype=dtype, copy=True) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/core/indexing.py", line 366, in array return np.asarray(array[self.key], dtype=None) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/xarray/backends/pynio.py", line 44, in getitem return array[key] File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/PyNIO/Nio.py", line 347, in getitem ret = get_variable(self.file, self.varname, xsel) File "/home/sergey/miniconda2/envs/pestforecast/lib/python3.6/site-packages/PyNIO/coordsel.py", line 68, in get_variable ret = file.file.variables[varname][xsel]

As i found problem in that items in xsel variable are instances of numpy.int64 So if i add a dirty hack:

if isinstance(xsel, tuple) and len(xsel) > 0 and isinstance(xsel[0], N.int64): xsel = tuple(map(lambda x: int(x) if x else x, xsel))

all works properly

Can you help me this problem?

[bladwig1]

We are not the developers of xarray. If the problem is with pynio, please create an example that demonstrates the problem outside of xarray.

[bladwig1]

OK, I think I understand the problem now. This works for Python 2, but fails for Python 3:

import Nio import numpy as np a = Nio.open_file("wrfout_d01_2008-09-29_07:00:00.nc") b = np.int64(10) c = (Ellipsis,) + (b,b) d = a.variables["P"][c]

Fails with:

Traceback (most recent call last): File "", line 1, in File "/Users/ladwig/miniconda2/envs/pynio3_build/lib/python3.5/site-packages/PyNIO/Nio.py", line 347, in getitem ret = get_variable(self.file, self.varname, xsel) File "/Users/ladwig/miniconda2/envs/pynio3_build/lib/python3.5/site-packages/PyNIO/coordsel.py", line 61, in get_variable ret = file.file.variables[varname][xsel] TypeError: illegal subscript type

The code is the same as in Python 2, so I'll have to do some digging to see why this changed in Python 3.x.

Thanks for reporting the issue.

[stitovich]

@bladwig1 Maybe you have any updates regarding this issue?

[bladwig1]

The issue is because PyInt_Check is removed in Python 3.x, and PyLong_Check fails with numpy integers, which worked in Python 2.x. I'll hopefully be able to turn my attention back to PyNIO in March.