Closed sjperkins closed 2 years ago
@landmanbester @JSKenyon The changes in this PR now cause a consistent failure, and no spurious MAIN tables are created:
Traceback (most recent call last):
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/mapping.py", line 137, in __getitem__
result = self.fs.cat(k)
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/spec.py", line 755, in cat
return self.cat_file(paths[0], **kwargs)
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/spec.py", line 665, in cat_file
with self.open(path, "rb", **kwargs) as f:
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/spec.py", line 1034, in open
f = self._open(
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/implementations/local.py", line 162, in _open
return LocalFileOpener(path, mode, fs=self, **kwargs)
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/implementations/local.py", line 260, in __init__
self._open()
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/implementations/local.py", line 265, in _open
self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/simon/code/dask-ms/github/dummy_dir/dummy_subtable/MAIN/.zmetadata'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/simon/code/dask-ms/daskms/experimental/zarr/__init__.py", line 419, in xds_from_zarr
table_group = zarr.open_consolidated(store_map, mode="r")
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/convenience.py", line 1304, in open_consolidated
meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/storage.py", line 2853, in __init__
meta = json_loads(self.store[metadata_key])
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/storage.py", line 717, in __getitem__
return self._mutable_mapping[key]
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/fsspec/mapping.py", line 141, in __getitem__
raise KeyError(key)
KeyError: '.zmetadata'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "gh261.py", line 19, in <module>
incorrect_xds = xds_from_zarr("dummy_dir/dummy_subtable")
File "/home/simon/code/dask-ms/daskms/experimental/zarr/__init__.py", line 421, in xds_from_zarr
table_group = zarr.open_group(store_map, mode="r")
File "/home/simon/.cache/pypoetry/virtualenvs/dask-ms-jCyuTJVk-py3.8/lib/python3.8/site-packages/zarr/hierarchy.py", line 1347, in open_group
raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''
Seems to have fixed the problem. Thanks @sjperkins
Thanks for the reproducer @JSKenyon. Merging.
Thanks for the quick fix @sjperkins!
Closes #261
[x] Tests added / passed
If the pep8 tests fail, the quickest way to correct this is to run
autopep8
and thenflake8
andpycodestyle
to fix the remaining issues.[x] Fully documented, including
HISTORY.rst
for all changes and one of thedocs/*-api.rst
files for new APITo build the docs locally: