search

catalystneuro / lazy_ops

Lazy transposing and slicing of h5py and Zarr Datasets
BSD 3-Clause "New" or "Revised" License
3 stars 3 forks source link

Bug in lazy_transpose #30

Open HenryDayHall opened 7 months ago

HenryDayHall commented 7 months ago

Python version 3.11.7, lazy_ops version 0.2.0.

When a dataset has been transposed, the left hand side of an assignment doesn't recognise the transpose. For example;

import h5py
import numpy as np
import lazy_ops

sample_data = np.random.rand(100, 10, 5)
with h5py.File("sample.h5", "w") as f:
    f.create_dataset("data", data=sample_data)
read_data = h5py.File("sample.h5", "r")["data"]
sample_view = lazy_ops.DatasetView(read_data)
sample_transposed = sample_view.lazy_transpose([0, 2, 1])
sample_transposed[:, :, -1] = sample_transposed[:, :, -1] * 2

Gives 

#     sample_transposed[:, :, -1] = sample_transposed[:, :, -1] * 2
#     ~~~~~~~~~~~~~~~~~^^^^^^^^^^
#   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
#   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
#   File "/home/henry/Programs/miniconda3/envs/caloclouds/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 997, in __setitem__
#     mspace = h5s.create_simple(selection.expand_shape(mshape))
#                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/home/henry/Programs/miniconda3/envs/caloclouds/lib/python3.11/site-packages/h5py/_hl/selections.py", line 264, in expand_shape
#     raise TypeError("Can't broadcast %s -> %s" % (source_shape, self.array_shape))  # array shape
#     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# TypeError: Can't broadcast (100, 5) -> (100, 10)

It's not a very useful operation, as it presumably breaks the lazy loading. None the less, it probably is a bug, or at least wants a different error message.