ratt-ru / dask-ms

Implementation of a dask/xarray dataset backed by a CASA MS
https://dask-ms.readthedocs.io
Other
19 stars 7 forks source link

xds_from_zarr should always open zarr groups in read mode #262

Closed sjperkins closed 2 years ago

sjperkins commented 2 years ago
sjperkins commented 2 years ago

@landmanbester @JSKenyon The changes in this PR now cause a consistent failure, and no spurious MAIN tables are created:

Traceback (most recent call last):
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/mapping.py", line 137, in __getitem__
    result = self.fs.cat(k)
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/spec.py", line 755, in cat
    return self.cat_file(paths[0], **kwargs)
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/spec.py", line 665, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/spec.py", line 1034, in open
    f = self._open(
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/implementations/local.py", line 162, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/implementations/local.py", line 260, in __init__
    self._open()
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/implementations/local.py", line 265, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/simon/code/dask-ms/github/dummy_dir/dummy_subtable/MAIN/.zmetadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/simon/code/dask-ms/daskms/experimental/zarr/__init__.py", line 419, in xds_from_zarr
    table_group = zarr.open_consolidated(store_map, mode="r")
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/convenience.py", line 1304, in open_consolidated
    meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/storage.py", line 2853, in __init__
    meta = json_loads(self.store[metadata_key])
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/storage.py", line 717, in __getitem__
    return self._mutable_mapping[key]
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/mapping.py", line 141, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gh261.py", line 19, in <module>
    incorrect_xds = xds_from_zarr("dummy_dir/dummy_subtable")
  File "/home/simon/code/dask-ms/daskms/experimental/zarr/__init__.py", line 421, in xds_from_zarr
    table_group = zarr.open_group(store_map, mode="r")
  File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/hierarchy.py", line 1347, in open_group
    raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''
landmanbester commented 2 years ago

Seems to have fixed the problem. Thanks @sjperkins

sjperkins commented 2 years ago

Thanks for the reproducer @JSKenyon. Merging.

JSKenyon commented 2 years ago

Thanks for the quick fix @sjperkins!