aaronspring commented 5 years ago

I heard about the cfgrib lib which got implemented in xarray. But I don’t get it working. Workaround: cdo postprocessing

aaronspring commented 5 years ago

grb_ds = xr.open_dataset(grb_path, engine='cfgrib')

Can't create file '/work/mh0727/m300524/experiments/GE/hist/hist0001/outdata/echam6/hist0001_echam6_BOT_mm_1999.grb.887f8.idx'
Traceback (most recent call last):
  File "/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/cfgrib/messages.py", line 305, in from_indexpath_or_filestream
  File "/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/cfgrib/messages.py", line 243, in compat_create_exclusive
PermissionError: [Errno 13] Permission denied: '/work/mh0727/m300524/experiments/GE/hist/hist0001/outdata/echam6/hist0001_echam6_BOT_mm_1999.grb.887f8.idx'
Can't read index file '/work/mh0727/m300524/experiments/GE/hist/hist0001/outdata/echam6/hist0001_echam6_BOT_mm_1999.grb.887f8.idx'
Traceback (most recent call last):
  File "/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/cfgrib/messages.py", line 315, in from_indexpath_or_filestream
  File "/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/genericpath.py", line 55, in getmtime
    return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/work/mh0727/m300524/experiments/GE/hist/hist0001/outdata/echam6/hist0001_echam6_BOT_mm_1999.grb.887f8.idx'

---------------------------------------------------------------------------
DatasetBuildError                         Traceback (most recent call last)
<ipython-input-44-10bbce4c35e8> in <module>
----> 1 grb_ds = xr.open_dataset(grb_path, engine='cfgrib')

/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime)

/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/xarray/backends/cfgrib_.py in __init__(self, filename, lock, **backend_kwargs)

/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/cfgrib/dataset.py in open_file(path, grib_errors, **kwargs)

/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/cfgrib/dataset.py in build_dataset_components(stream, indexpath, filter_by_keys, errors, encode_cf, timestamp, log)

/work/mh0727/m300524/anaconda3/envs/my_jupyter/lib/python3.6/site-packages/cfgrib/dataset.py in enforce_unique_attributes(index, attributes_keys, filter_by_keys)

DatasetBuildError: multiple values for key 'centre'

https://www.ecmwf.int/sites/default/files/elibrary/2018/18727-cfgrib-easy-and-efficient-grib-file-access-xarray.pdf

aaronspring commented 5 years ago

cdopy with -f nc -t table returnXDataset. But this doesn’t work lazily

aaronspring commented 4 years ago

with pynio it works even lazily. conda install pynio

pwd=/work/mh0727/m300524/experiments/asp_esmControl_ens3175_m001/outdata/echam6

ds = xr.open_dataset("asp_esmControl_ens3175_m001_echam6_tracer_3188.grb",engine='pynio', chunks={'g4_lat_3':64})

missing: a nice mapping from table to variables as in cdo -t table in out. could be implemented via preprocess in open_mfdataset

aaronspring / pymistral

Open jsbach echam grb files #19

pwd=/work/mh0727/m300524/experiments/asp_esmControl_ens3175_m001/outdata/echam6