7 tests fail

[yurivict]

Zarr version

2.17.2

Numcodecs version

Numcodecs

Python Version

3.9

Operating System

FreeBSD 14

Installation

port

Description

========================================================================================= FAILURES =========================================================================================
_____________________________________________________________________________ TestNestedFSStore.test_hierarchy _____________________________________________________________________________
[gw2] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestNestedFSStore object at 0x1453af27310>

    def test_hierarchy(self):
        # setup
        store = self.create_store()
        store[self.root + "a"] = b"aaa"
        store[self.root + "b"] = b"bbb"
        store[self.root + "c/d"] = b"ddd"
        store[self.root + "c/e/f"] = b"fff"
        store[self.root + "c/e/g"] = b"ggg"

        # check keys
        assert self.root + "a" in store
        assert self.root + "b" in store
        assert self.root + "c/d" in store
        assert self.root + "c/e/f" in store
        assert self.root + "c/e/g" in store
        assert self.root + "c" not in store
        assert self.root + "c/" not in store
        assert self.root + "c/e" not in store
        assert self.root + "c/e/" not in store
        assert self.root + "c/d/x" not in store

        # check __getitem__
        with pytest.raises(KeyError):
            store[self.root + "c"]
        with pytest.raises(KeyError):
            store[self.root + "c/e"]
        with pytest.raises(KeyError):
            store[self.root + "c/d/x"]

        # test getsize (optional)
        if hasattr(store, "getsize"):
            # TODO: proper behavior of getsize?
            #       v3 returns size of all nested arrays, not just the
            #       size of the arrays in the current folder.
            if self.version == 2:
                assert 6 == store.getsize()
            else:
                assert 15 == store.getsize()
            assert 3 == store.getsize("a")
            assert 3 == store.getsize("b")
            if self.version == 2:
                assert 3 == store.getsize("c")
            else:
                assert 9 == store.getsize("c")
            assert 3 == store.getsize("c/d")
            assert 6 == store.getsize("c/e")
            assert 3 == store.getsize("c/e/f")
            assert 3 == store.getsize("c/e/g")
            # non-existent paths
            assert 0 == store.getsize("x")
            assert 0 == store.getsize("a/x")
            assert 0 == store.getsize("c/x")
            assert 0 == store.getsize("c/x/y")
            assert 0 == store.getsize("c/d/y")
            assert 0 == store.getsize("c/d/y/z")

            # access item via full path
            assert 3 == store.getsize(self.root + "a")

        # test listdir (optional)
        if hasattr(store, "listdir"):
            assert {"a", "b", "c"} == set(store.listdir(self.root))
            assert {"d", "e"} == set(store.listdir(self.root + "c"))
            assert {"f", "g"} == set(store.listdir(self.root + "c/e"))
            # no exception raised if path does not exist or is leaf
            assert [] == store.listdir(self.root + "x")
            assert [] == store.listdir(self.root + "a/x")
            assert [] == store.listdir(self.root + "c/x")
            assert [] == store.listdir(self.root + "c/x/y")
            assert [] == store.listdir(self.root + "c/d/y")
            assert [] == store.listdir(self.root + "c/d/y/z")
>           assert [] == store.listdir(self.root + "c/e/f")
E           AssertionError: assert [] == ['f']
E             Right contains one more item: 'f'
E             Full diff:
E             - ['f']
E             + []

zarr/tests/test_storage.py:409: AssertionError
_________________________________________________________________________________ TestFSStore.test_complex _________________________________________________________________________________
[gw1] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestFSStore object at 0x35cf7d40f8b0>

    def test_complex(self):
        path1 = tempfile.mkdtemp()
        path2 = tempfile.mkdtemp()
        store = self.create_store(
            path="simplecache::file://" + path1,
            simplecache={"same_names": True, "cache_storage": path2},
        )
        assert not store
        assert not os.listdir(path1)
        assert not os.listdir(path2)
        store[self.root + "foo"] = b"hello"
        assert "foo" in os.listdir(str(path1) + "/" + self.root)
        assert self.root + "foo" in store
>       assert not os.listdir(str(path2))
E       AssertionError: assert not ['foo']
E        +  where ['foo'] = <built-in function listdir>('/tmp/tmpglsphvne')
E        +    where <built-in function listdir> = os.listdir
E        +    and   '/tmp/tmpglsphvne' = str('/tmp/tmpglsphvne')

zarr/tests/test_storage.py:1155: AssertionError
________________________________________________________________________________ TestFSStore.test_hierarchy ________________________________________________________________________________
[gw1] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestFSStore object at 0x35cf7bcd6d00>

    def test_hierarchy(self):
        # setup
        store = self.create_store()
        store[self.root + "a"] = b"aaa"
        store[self.root + "b"] = b"bbb"
        store[self.root + "c/d"] = b"ddd"
        store[self.root + "c/e/f"] = b"fff"
        store[self.root + "c/e/g"] = b"ggg"

        # check keys
        assert self.root + "a" in store
        assert self.root + "b" in store
        assert self.root + "c/d" in store
        assert self.root + "c/e/f" in store
        assert self.root + "c/e/g" in store
        assert self.root + "c" not in store
        assert self.root + "c/" not in store
        assert self.root + "c/e" not in store
        assert self.root + "c/e/" not in store
        assert self.root + "c/d/x" not in store

        # check __getitem__
        with pytest.raises(KeyError):
            store[self.root + "c"]
        with pytest.raises(KeyError):
            store[self.root + "c/e"]
        with pytest.raises(KeyError):
            store[self.root + "c/d/x"]

        # test getsize (optional)
        if hasattr(store, "getsize"):
            # TODO: proper behavior of getsize?
            #       v3 returns size of all nested arrays, not just the
            #       size of the arrays in the current folder.
            if self.version == 2:
                assert 6 == store.getsize()
            else:
                assert 15 == store.getsize()
            assert 3 == store.getsize("a")
            assert 3 == store.getsize("b")
            if self.version == 2:
                assert 3 == store.getsize("c")
            else:
                assert 9 == store.getsize("c")
            assert 3 == store.getsize("c/d")
            assert 6 == store.getsize("c/e")
            assert 3 == store.getsize("c/e/f")
            assert 3 == store.getsize("c/e/g")
            # non-existent paths
            assert 0 == store.getsize("x")
            assert 0 == store.getsize("a/x")
            assert 0 == store.getsize("c/x")
            assert 0 == store.getsize("c/x/y")
            assert 0 == store.getsize("c/d/y")
            assert 0 == store.getsize("c/d/y/z")

            # access item via full path
            assert 3 == store.getsize(self.root + "a")

        # test listdir (optional)
        if hasattr(store, "listdir"):
            assert {"a", "b", "c"} == set(store.listdir(self.root))
            assert {"d", "e"} == set(store.listdir(self.root + "c"))
            assert {"f", "g"} == set(store.listdir(self.root + "c/e"))
            # no exception raised if path does not exist or is leaf
            assert [] == store.listdir(self.root + "x")
            assert [] == store.listdir(self.root + "a/x")
            assert [] == store.listdir(self.root + "c/x")
            assert [] == store.listdir(self.root + "c/x/y")
            assert [] == store.listdir(self.root + "c/d/y")
            assert [] == store.listdir(self.root + "c/d/y/z")
>           assert [] == store.listdir(self.root + "c/e/f")
E           AssertionError: assert [] == ['f']
E             Right contains one more item: 'f'
E             Full diff:
E             - ['f']
E             + []

zarr/tests/test_storage.py:409: AssertionError
________________________________________________________________________ TestFSStoreWithKeySeparator.test_hierarchy ________________________________________________________________________
[gw2] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestFSStoreWithKeySeparator object at 0x1453afb09d0>

    def test_hierarchy(self):
        # setup
        store = self.create_store()
        store[self.root + "a"] = b"aaa"
        store[self.root + "b"] = b"bbb"
        store[self.root + "c/d"] = b"ddd"
        store[self.root + "c/e/f"] = b"fff"
        store[self.root + "c/e/g"] = b"ggg"

        # check keys
        assert self.root + "a" in store
        assert self.root + "b" in store
        assert self.root + "c/d" in store
        assert self.root + "c/e/f" in store
        assert self.root + "c/e/g" in store
        assert self.root + "c" not in store
        assert self.root + "c/" not in store
        assert self.root + "c/e" not in store
        assert self.root + "c/e/" not in store
        assert self.root + "c/d/x" not in store

        # check __getitem__
        with pytest.raises(KeyError):
            store[self.root + "c"]
        with pytest.raises(KeyError):
            store[self.root + "c/e"]
        with pytest.raises(KeyError):
            store[self.root + "c/d/x"]

        # test getsize (optional)
        if hasattr(store, "getsize"):
            # TODO: proper behavior of getsize?
            #       v3 returns size of all nested arrays, not just the
            #       size of the arrays in the current folder.
            if self.version == 2:
                assert 6 == store.getsize()
            else:
                assert 15 == store.getsize()
            assert 3 == store.getsize("a")
            assert 3 == store.getsize("b")
            if self.version == 2:
                assert 3 == store.getsize("c")
            else:
                assert 9 == store.getsize("c")
            assert 3 == store.getsize("c/d")
            assert 6 == store.getsize("c/e")
            assert 3 == store.getsize("c/e/f")
            assert 3 == store.getsize("c/e/g")
            # non-existent paths
            assert 0 == store.getsize("x")
            assert 0 == store.getsize("a/x")
            assert 0 == store.getsize("c/x")
            assert 0 == store.getsize("c/x/y")
            assert 0 == store.getsize("c/d/y")
            assert 0 == store.getsize("c/d/y/z")

            # access item via full path
            assert 3 == store.getsize(self.root + "a")

        # test listdir (optional)
        if hasattr(store, "listdir"):
            assert {"a", "b", "c"} == set(store.listdir(self.root))
            assert {"d", "e"} == set(store.listdir(self.root + "c"))
            assert {"f", "g"} == set(store.listdir(self.root + "c/e"))
            # no exception raised if path does not exist or is leaf
            assert [] == store.listdir(self.root + "x")
            assert [] == store.listdir(self.root + "a/x")
            assert [] == store.listdir(self.root + "c/x")
            assert [] == store.listdir(self.root + "c/x/y")
            assert [] == store.listdir(self.root + "c/d/y")
            assert [] == store.listdir(self.root + "c/d/y/z")
>           assert [] == store.listdir(self.root + "c/e/f")
E           AssertionError: assert [] == ['f']
E             Right contains one more item: 'f'
E             Full diff:
E             - ['f']
E             + []

zarr/tests/test_storage.py:409: AssertionError
________________________________________________________________________________ TestN5FSStore.test_complex ________________________________________________________________________________
[gw1] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestN5FSStore object at 0x35cf7cf2a4c0>

    def test_complex(self):
        path1 = tempfile.mkdtemp()
        path2 = tempfile.mkdtemp()
        store = self.create_store(
            path="simplecache::file://" + path1,
            simplecache={"same_names": True, "cache_storage": path2},
        )
        assert not store
        assert not os.listdir(path1)
        assert not os.listdir(path2)
        store[self.root + "foo"] = b"hello"
        assert "foo" in os.listdir(str(path1) + "/" + self.root)
        assert self.root + "foo" in store
>       assert not os.listdir(str(path2))
E       AssertionError: assert not ['foo']
E        +  where ['foo'] = <built-in function listdir>('/tmp/tmp54ti_hui')
E        +    where <built-in function listdir> = os.listdir
E        +    and   '/tmp/tmp54ti_hui' = str('/tmp/tmp54ti_hui')

zarr/tests/test_storage.py:1155: AssertionError
_______________________________________________________________________________ TestN5FSStore.test_hierarchy _______________________________________________________________________________
[gw1] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestN5FSStore object at 0x35cf7c43d550>

    def test_hierarchy(self):
        # setup
        store = self.create_store()
        store[self.root + "a"] = b"aaa"
        store[self.root + "b"] = b"bbb"
        store[self.root + "c/d"] = b"ddd"
        store[self.root + "c/e/f"] = b"fff"
        store[self.root + "c/e/g"] = b"ggg"

        # check keys
        assert self.root + "a" in store
        assert self.root + "b" in store
        assert self.root + "c/d" in store
        assert self.root + "c/e/f" in store
        assert self.root + "c/e/g" in store
        assert self.root + "c" not in store
        assert self.root + "c/" not in store
        assert self.root + "c/e" not in store
        assert self.root + "c/e/" not in store
        assert self.root + "c/d/x" not in store

        # check __getitem__
        with pytest.raises(KeyError):
            store[self.root + "c"]
        with pytest.raises(KeyError):
            store[self.root + "c/e"]
        with pytest.raises(KeyError):
            store[self.root + "c/d/x"]

        # test getsize (optional)
        if hasattr(store, "getsize"):
            # TODO: proper behavior of getsize?
            #       v3 returns size of all nested arrays, not just the
            #       size of the arrays in the current folder.
            if self.version == 2:
                assert 6 == store.getsize()
            else:
                assert 15 == store.getsize()
            assert 3 == store.getsize("a")
            assert 3 == store.getsize("b")
            if self.version == 2:
                assert 3 == store.getsize("c")
            else:
                assert 9 == store.getsize("c")
            assert 3 == store.getsize("c/d")
            assert 6 == store.getsize("c/e")
            assert 3 == store.getsize("c/e/f")
            assert 3 == store.getsize("c/e/g")
            # non-existent paths
            assert 0 == store.getsize("x")
            assert 0 == store.getsize("a/x")
            assert 0 == store.getsize("c/x")
            assert 0 == store.getsize("c/x/y")
            assert 0 == store.getsize("c/d/y")
            assert 0 == store.getsize("c/d/y/z")

            # access item via full path
            assert 3 == store.getsize(self.root + "a")

        # test listdir (optional)
        if hasattr(store, "listdir"):
            assert {"a", "b", "c"} == set(store.listdir(self.root))
            assert {"d", "e"} == set(store.listdir(self.root + "c"))
            assert {"f", "g"} == set(store.listdir(self.root + "c/e"))
            # no exception raised if path does not exist or is leaf
            assert [] == store.listdir(self.root + "x")
            assert [] == store.listdir(self.root + "a/x")
            assert [] == store.listdir(self.root + "c/x")
            assert [] == store.listdir(self.root + "c/x/y")
            assert [] == store.listdir(self.root + "c/d/y")
            assert [] == store.listdir(self.root + "c/d/y/z")
>           assert [] == store.listdir(self.root + "c/e/f")
E           AssertionError: assert [] == ['f']
E             Right contains one more item: 'f'
E             Full diff:
E             - ['f']
E             + []

zarr/tests/test_storage.py:409: AssertionError
_________________________________________________________________________ TestFSStoreFromFilesystem.test_hierarchy _________________________________________________________________________
[gw0] freebsd14 -- Python 3.9.18 /usr/local/bin/python3.9

self = <zarr.tests.test_storage.TestFSStoreFromFilesystem object at 0xfa69af959a0>

    def test_hierarchy(self):
        # setup
        store = self.create_store()
        store[self.root + "a"] = b"aaa"
        store[self.root + "b"] = b"bbb"
        store[self.root + "c/d"] = b"ddd"
        store[self.root + "c/e/f"] = b"fff"
        store[self.root + "c/e/g"] = b"ggg"

        # check keys
        assert self.root + "a" in store
        assert self.root + "b" in store
        assert self.root + "c/d" in store
        assert self.root + "c/e/f" in store
        assert self.root + "c/e/g" in store
        assert self.root + "c" not in store
        assert self.root + "c/" not in store
        assert self.root + "c/e" not in store
        assert self.root + "c/e/" not in store
        assert self.root + "c/d/x" not in store

        # check __getitem__
        with pytest.raises(KeyError):
            store[self.root + "c"]
        with pytest.raises(KeyError):
            store[self.root + "c/e"]
        with pytest.raises(KeyError):
            store[self.root + "c/d/x"]

        # test getsize (optional)
        if hasattr(store, "getsize"):
            # TODO: proper behavior of getsize?
            #       v3 returns size of all nested arrays, not just the
            #       size of the arrays in the current folder.
            if self.version == 2:
                assert 6 == store.getsize()
            else:
                assert 15 == store.getsize()
            assert 3 == store.getsize("a")
            assert 3 == store.getsize("b")
            if self.version == 2:
                assert 3 == store.getsize("c")
            else:
                assert 9 == store.getsize("c")
            assert 3 == store.getsize("c/d")
            assert 6 == store.getsize("c/e")
            assert 3 == store.getsize("c/e/f")
            assert 3 == store.getsize("c/e/g")
            # non-existent paths
            assert 0 == store.getsize("x")
            assert 0 == store.getsize("a/x")
            assert 0 == store.getsize("c/x")
            assert 0 == store.getsize("c/x/y")
            assert 0 == store.getsize("c/d/y")
            assert 0 == store.getsize("c/d/y/z")

            # access item via full path
            assert 3 == store.getsize(self.root + "a")

        # test listdir (optional)
        if hasattr(store, "listdir"):
            assert {"a", "b", "c"} == set(store.listdir(self.root))
            assert {"d", "e"} == set(store.listdir(self.root + "c"))
            assert {"f", "g"} == set(store.listdir(self.root + "c/e"))
            # no exception raised if path does not exist or is leaf
            assert [] == store.listdir(self.root + "x")
            assert [] == store.listdir(self.root + "a/x")
            assert [] == store.listdir(self.root + "c/x")
            assert [] == store.listdir(self.root + "c/x/y")
            assert [] == store.listdir(self.root + "c/d/y")
            assert [] == store.listdir(self.root + "c/d/y/z")
>           assert [] == store.listdir(self.root + "c/e/f")
E           AssertionError: assert [] == ['f']
E             Right contains one more item: 'f'
E             Full diff:
E             - ['f']
E             + []

zarr/tests/test_storage.py:409: AssertionError

Steps to reproduce

run tests

Additional output

No response

[avalentino]

This issue is also reproducible on debian sid

[avalentino]

Apparently the issue is triggered by the update of fsspec from v2024.2.0 to v2024.3.1

[jhamman]

See https://github.com/zarr-developers/zarr-python/pull/1877#issuecomment-2115322894 for a diagnosis of what is going on here. Marking this as help-wanted.

[martindurant]

Damn but I thought I had already commented on these in a different thread. Now to find it...

[jhamman]

You did (https://github.com/zarr-developers/zarr-python/pull/1877#issuecomment-2115322894), but that ticket was closed by dependabot.

[QuLogic]

This is also a dupe of #1679 (or vice versa).

[yurivict]

2.18.3 has 32 test failures: https://freebsd.org/~yuri/py-zarr-2.18.3-tests.log