zarr-developers / zarr-python

An implementation of chunked, compressed, N-dimensional arrays for Python.
http://zarr.readthedocs.io/
MIT License
1.43k stars 272 forks source link

2.18.1: pytest fails with `numcodecs` 0.12.1 #1891

Open kloczek opened 3 months ago

kloczek commented 3 months ago

Zarr version

2.18.1

Numcodecs version

0.12.1

Python Version

3.10.14

Operating System

Linux

Installation

From autogenerated from git tag tar ball.

Description

Looks like zarr test suite needs to be updated for latest numcodecs 0.12.1.

Steps to reproduce

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

Here is pytest output: ```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-zarr-2.18.1-2.fc37.x86_64/usr/lib64/python3.10/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-zarr-2.18.1-2.fc37.x86_64/usr/lib/python3.10/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/zarr-python-2.18.1 configfile: pyproject.toml collected 2621 items / 2 errors ========================================================================================== ERRORS =========================================================================================== _________________________________________________________________________ ERROR collecting zarr/tests/test_core.py __________________________________________________________________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/zarr-python-2.18.1/zarr/tests/test_core.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) zarr/tests/test_core.py:31: in from numcodecs.tests.common import greetings E ModuleNotFoundError: No module named 'numcodecs.tests' _________________________________________________________________________ ERROR collecting zarr/tests/test_sync.py __________________________________________________________________________ ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/zarr-python-2.18.1/zarr/tests/test_sync.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.10/importlib/__init__.py:126: in import_module return _bootstrap._gcd_import(name[level:], package, level) zarr/tests/test_sync.py:20: in from zarr.tests.test_core import TestArray zarr/tests/test_core.py:31: in from numcodecs.tests.common import greetings E ModuleNotFoundError: No module named 'numcodecs.tests' ================================================================================== short test summary info ================================================================================== ERROR zarr/tests/test_core.py ERROR zarr/tests/test_sync.py !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ===================================================================================== 2 errors in 3.79s ===================================================================================== ```

Additional output

List of installed modules in build env: ```console Package Version ----------------------------- ----------- alabaster 0.7.16 asciitree 0.3.3 Babel 2.15.0 build 1.2.1 charset-normalizer 3.3.2 defusedxml 0.7.1 docutils 0.20.1 exceptiongroup 1.1.3 idna 3.7 imagesize 1.4.1 importlib_metadata 7.1.0 iniconfig 2.0.0 installer 0.7.0 Jinja2 3.1.4 MarkupSafe 2.1.5 msgpack 1.0.8 numcodecs 0.12.1 numpy 1.26.5 numpydoc 1.7.0 packaging 24.0 pluggy 1.4.0 Pygments 2.18.0 pyproject_hooks 1.0.0 pytest 8.1.1 python-dateutil 2.9.0.post0 requests 2.31.0 setuptools 69.4.0 setuptools-scm 8.1.0 snowballstemmer 2.2.0 Sphinx 7.3.7 sphinx-automodapi 0.17.0 sphinx-copybutton 0.5.2 sphinx_design 0.5.0 sphinx-issues 3.0.1 sphinxcontrib-applehelp 1.0.8 sphinxcontrib-devhelp 1.0.6 sphinxcontrib-htmlhelp 2.0.5 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.7 sphinxcontrib-serializinghtml 1.1.10 tabulate 0.9.0 tokenize_rt 5.2.0 tomli 2.0.1 urllib3 2.2.1 wheel 0.43.0 zipp 3.18.2 ```

Please let me know if you need more details or want me to perform some diagnostics.

kloczek commented 3 months ago

BTW it could be another issue as well https://github.com/zarr-developers/zarr-python/issues/1891

toloudis commented 1 month ago

I've seen this issue also. I'm having to pin zarr < 2.18. In my case it's a problem with a dask array not passing a is_ndarray_like test deep inside set_items... https://github.com/bioio-devs/bioio/actions/runs/9812320870

joshmoore commented 1 month ago

@madsbk, does this ring a bell?

FAILED bioio/tests/writers/test_ome_zarr_writer_2.py::test_write_ome_zarr[e.zarr-shape0-3-scaling0-da_random_from_shape] - TypeError: memoryview: a bytes-like object is required, not 'Array'

        if not is_ndarray_like(buf):
            if isinstance(buf, array.array) and buf.typecode in "cu":
                # Guard condition, do not support array.array with unicode type, this is
                # problematic because numpy does not support it on all platforms. Also do not
                # support char as it was removed in Python 3.
                raise TypeError("array.array with char or unicode type is not supported")
            else:
                # N.B., first take a memoryview to make sure that we subsequently create a
                # numpy array from a memory buffer with no copy
>               mem = memoryview(buf)
E               TypeError: memoryview: a bytes-like object is required, not 'Array'

from https://github.com/zarr-developers/numcodecs/commit/bedb8b0981388b1a2292c0d78e044bd36c2aacd8

madsbk commented 3 weeks ago

Sorry no, I am not able to reproduce any of the issues locally :/

If I could reproduce, I would check the type of buf and see why is_ndarray_like(buf) is False. If the type name is 'Array', I would guess is_ndarray_like(buf) should be True