xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.
https://xcube.readthedocs.io/
MIT License
199 stars 18 forks source link

xcube serve demo returns KeyError for remote cube #490

Closed AliceBalfanz closed 3 years ago

AliceBalfanz commented 3 years ago

Describe the bug when starting a xcube server with the demo config and launching xcube viewer to view the demo data with: xcube serve -v -c ./examples/serve/demo/config.yml

a KeyError: 'xcube-examples/OLCI-SNS-RAW-CUBE-2.zarr' appears.

traceback:

[I 210713 14:33:02 perf:79] opened remote zarr dataset xcube-examples/OLCI-SNS-RAW-CUBE-2.zarr: took 494.76ms
tornado.application - ERROR - Uncaught exception GET /datasets?details=1&tiles=ol4 (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:8080', method='GET', uri='/datasets?details=1&tiles=ol4', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/tornado/web.py", line 1702, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/handlers.py", line 133, in get
    response = get_datasets(self.service_context,
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/controllers/catalogue.py", line 67, in get_datasets
    dataset_dict.update(get_dataset(ctx, ds_id, client, base_url, granted_scopes=granted_scopes))
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/controllers/catalogue.py", line 94, in get_dataset
    ds = ctx.get_dataset(ds_id)
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/context.py", line 208, in get_dataset
    ml_dataset, _ = self._get_dataset_entry(ds_id)
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/context.py", line 331, in _get_dataset_entry
    self._set_dataset_entry(self._create_dataset_entry(ds_id))
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/context.py", line 340, in _create_dataset_entry
    ml_dataset = self._open_ml_dataset(dataset_descriptor)
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/context.py", line 352, in _open_ml_dataset
    ml_dataset = ml_dataset_opener(self, dataset_descriptor)
  File "/home/alicja/Desktop/projects/xcube/xcube/webapi/context.py", line 690, in _open_ml_dataset_from_object_storage
    return open_ml_dataset_from_object_storage(path,
  File "/home/alicja/Desktop/projects/xcube/xcube/core/mldataset.py", line 666, in open_ml_dataset_from_object_storage
    ds = assert_cube(xr.open_zarr(store, consolidated=consolidated, **kwargs))
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/xarray/backends/zarr.py", line 666, in open_zarr
    ds = open_dataset(
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/xarray/backends/api.py", line 554, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/xarray/backends/zarr.py", line 324, in open_group
    zarr_group = zarr.open_group(store, **open_kwargs)
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/zarr/hierarchy.py", line 1163, in open_group
    if contains_array(store, path=path):
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/zarr/storage.py", line 95, in contains_array
    return key in store
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/zarr/storage.py", line 2019, in __contains__
    self._contains_cache = set(self._keys())
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/zarr/storage.py", line 2032, in _keys
    self._keys_cache = list(self._store.keys())
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/_collections_abc.py", line 805, in __len__
    return len(self._mapping)
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/fsspec/mapping.py", line 158, in __len__
    return len(self.fs.find(self.root))
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/fsspec/asyn.py", line 72, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in sync
    raise result[0]
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/fsspec/asyn.py", line 20, in _runner
    result[0] = await coro
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/s3fs/core.py", line 596, in _find
    self.dircache[par].append(o)
  File "/home/alicja/miniconda3/envs/xcube/lib/python3.9/site-packages/fsspec/dircache.py", line 62, in __getitem__
    return self._cache[item]  # maybe raises KeyError
KeyError: 'xcube-examples/OLCI-SNS-RAW-CUBE-2.zarr'

Last working version is xcube v0.8.1

To Reproduce Steps to reproduce the behavior:

  1. Checkout master
  2. start server with xcube serve -v -c ./examples/serve/demo/config.yml
  3. Launch xcube viewer
  4. See error

Expected behavior I expect no error - because with previous version the server configuration was working well.

AliceBalfanz commented 3 years ago

The error was due to a not update environment. Closing.