search

mehta-lab / waveorder

Wave optical models and inverse algorithms for label-agnostic imaging of density & orientation.
BSD 3-Clause "New" or "Revised" License
15 stars 4 forks source link

`WaveorderWriter` is not compatible with `napari-ome-zarr` reader #94

Closed talonchandler closed 1 year ago

talonchandler commented 2 years ago

I first noticed this issue during a recOrder session when I tried to drag and drop a zarr store saved by a recOrder acquisition into napari. The layers and shape populated correctly, but the actual pixel values were set to zero.

Digging further, I've realized that this is a WaveorderWriter issue. Here's a minimal example:

$ python
>>> import numpy as np
>>> from waveorder.io import WaveorderWriter
>>> writer = WaveorderWriter("./test")
>>> writer.create_zarr_root("test")
>>> writer.init_array(position=0, data_shape=(1,1,2,3,4), chunk_size=(1,1,2,3,4), chan_names=["TEST"])
>>> writer.write(np.random.random((1,1,2,3,4)), p=0)
>>> quit()
$ napari --plugin napari-ome-zarr test/test.zarr

results in this traceback:

18:06:35 WARNING version mismatch: detected: FormatV01, requested: FormatV04
18:06:35 ERROR Failed to load Row_0/Col_0/0/0
Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 137, in __getitem__
    result = self.fs.cat(k)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 755, in cat
    return self.cat_file(paths[0], **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 665, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 1034, in open
    f = self._open(
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 162, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 260, in __init__
    self._open()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 265, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'Q:/Talon/2022_11_23/recOrderPluginSnap_0/test/test.zarr/Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1376, in __getitem__
    return self.map[key]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 141, in __getitem__
    raise KeyError(key)
KeyError: 'Row_0/Col_0/0/0/.zarray'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 241, in _load_metadata_nosync
    meta_bytes = self._store[mkey]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1378, in __getitem__
    raise KeyError(key) from e
KeyError: 'Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\reader.py", line 548, in get_tile
    data = self.zarr.load(path)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\io.py", line 113, in load
    return da.from_zarr(self.__store, subpath)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\dask\array\core.py", line 3569, in from_zarr
    z = zarr.Array(mapper, read_only=True, path=component, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 215, in __init__
    self._load_metadata()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 232, in _load_metadata
    self._load_metadata_nosync()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 243, in _load_metadata_nosync
    raise ArrayNotFoundError(self._path)
zarr.errors.ArrayNotFoundError: array not found at path %r' 'Row_0/Col_0/0/0'
Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\Scripts\napari.exe\__main__.py", line 7, in <module>
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\__main__.py", line 446, in main
    _run()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\__main__.py", line 335, in _run
    run(gui_exceptions=True)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\_qt\qt_event_loop.py", line 402, in run
    app.exec_()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\contextlib.py", line 126, in __exit__
    next(self.gen)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\_qt\utils.py", line 459, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)
KeyboardInterrupt
(recorder-dev) PS Q:\Talon\2022_11_23\recOrderPluginSnap_0> python
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:51:29) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from waveorder.io import WaveorderWriter
>>> writer = WaveorderWriter("./test")
>>> writer.create_zarr_root("tes")
Creating new zarr store at ./test\tes.zarr
>>> writer.init_array(position=0, data_shape=(1,1,2,3,4), chunk_size=(1,1,2,3,4), chan_names=["TEST"])
>>> writer.write(np.random.random((1,1,2,3,4)), p=0)
>>> quit()
(recorder-dev) PS Q:\Talon\2022_11_23\recOrderPluginSnap_0> napari --plugin napari-ome-zarr test/tes.zarr
18:11:25 WARNING version mismatch: detected: FormatV01, requested: FormatV04
18:11:25 ERROR Failed to load Row_0/Col_0/0/0
Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 137, in __getitem__
    result = self.fs.cat(k)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 755, in cat
    return self.cat_file(paths[0], **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 665, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 1034, in open
    f = self._open(
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 162, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 260, in __init__
    self._open()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 265, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'Q:/Talon/2022_11_23/recOrderPluginSnap_0/test/tes.zarr/Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1376, in __getitem__
    return self.map[key]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 141, in __getitem__
    raise KeyError(key)
KeyError: 'Row_0/Col_0/0/0/.zarray'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 241, in _load_metadata_nosync
    meta_bytes = self._store[mkey]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1378, in __getitem__
    raise KeyError(key) from e
KeyError: 'Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\reader.py", line 548, in get_tile
    data = self.zarr.load(path)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\io.py", line 113, in load
    return da.from_zarr(self.__store, subpath)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\dask\array\core.py", line 3569, in from_zarr
    z = zarr.Array(mapper, read_only=True, path=component, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 215, in __init__
    self._load_metadata()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 232, in _load_metadata
    self._load_metadata_nosync()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 243, in _load_metadata_nosync
    raise ArrayNotFoundError(self._path)
zarr.errors.ArrayNotFoundError: array not found at path %r' 'Row_0/Col_0/0/0'

It seems related to our naming convention of Row_0/Col_0/Pos_000. Do we need to change this convention, or change the metadata so that ome-zarr can read our zarrs?

I think this is reasonably important because there are lots of cases when I'd like to use the ome-zarr reader to split zarr's channels into layers (it's native behaviour) instead of the recOrder-napari reader that opens as a single array (also convenient in some cases). It also causes confusion because the reader seems to work but has zeros---did the reconstruction get saved? (It does)

ziw-liu commented 2 years ago

It's possible that napari-ome-zarr throws KeyError: 'Row_0/Col_0/0/0/.zarray' because it expects a newer version of OME-NGFF format. Taking a look at its source, it does not use the fmt: Format = FormatV04 option of ome-zarr.reader.Reader: https://github.com/ome/napari-ome-zarr/blob/1d423d21a76b6cda8f5feb2b7292ace96df708ae/napari_ome_zarr/_reader.py#L33-L35

ziw-liu commented 2 years ago

Possibly introduced in: 0.2.0 | 2021-03-29 | Change chunk dimension separator to "/"

ziw-liu commented 2 years ago

Taking a look at its source, it does not use the fmt: Format = FormatV04 option of ome-zarr.reader.Reader

First line of the traceback indicates the problem:

18:06:35 WARNING version mismatch: detected: FormatV01, requested: FormatV04

While ome-zarr-py should be capable of reading data in earlier OME-NGFF versions, napari-ome-zarr does not utilize this feature.

ziw-liu commented 2 years ago

Related: https://github.com/ome/napari-ome-zarr/issues/39

ziw-liu commented 1 year ago

Now implemented as https://github.com/czbiohub/iohub/pull/31.

We can keep this issue until iohub is released and used by recOrder, and waveorder.io can be removed from waveorder v1.0.0.

talonchandler commented 1 year ago

iohub has been released, so this is fixed. Closing.