zarr-developers / zarr-python

An implementation of chunked, compressed, N-dimensional arrays for Python.
https://zarr.readthedocs.io
MIT License
1.47k stars 274 forks source link

ArrayV3Metadata does not declare storage_transformers #2009

Open ghidalgo3 opened 3 months ago

ghidalgo3 commented 3 months ago

Zarr version

3.0.0a0

Numcodecs version

0.12.1

Python Version

3.10.12

Operating System

Linux

Installation

pip install zarr==3.0.0a0

Description

According to the ZarrV3 specification on storage_transformers, array v3 metadata must support a storage_transformers property. If I create a Zarr store with VirtualiZarr which uses the storage_transformers key in the zarr.json, then attempting to read store will produces this exception TypeError: ArrayV3Metadata.__init__() got an unexpected keyword argument 'storage_transformers'.

Steps to reproduce

import zarr
import xarray as xr
import numpy as np
from virtualizarr import ManifestArray, ChunkManifest

arr = ManifestArray(
        chunkmanifest=ChunkManifest(
            entries={"0.0": dict(path="test.nc", offset=6144, length=48)}
        ),
        zarray=dict(
            shape=(2, 3),
            dtype=np.dtype("<i8"),
            chunks=(2, 3),
            compressor=None,
            filters=None,
            fill_value=np.nan,
            order="C",
            zarr_format=3,
        ),
    )
original = xr.Dataset({"a": (["x", "y"], arr)}, attrs={"something": 0})
original.virtualize.to_zarr("store.zarr")
group = zarr.open_group(store="store.zarr")
group.arrays() # This will throw TypeError: ArrayV3Metadata.__init__() got an unexpected keyword argument 'storage_transformers'

Additional output

This is the zarr.json for the a variable in the repro script:

{
    "attributes": {},
    "chunk_grid": {
        "configuration": {
            "chunk_shape": [
                2,
                3
            ]
        },
        "name": "regular"
    },
    "chunk_key_encoding": {
        "configuration": {
            "separator": "/"
        },
        "name": "default"
    },
    "codecs": null,
    "data_type": "int64",
    "dimension_names": [
        "x",
        "y"
    ],
    "fill_value": null,
    "node_type": "array",
    "shape": [
        2,
        3
    ],
    "storage_transformers": [
        {
            "configuration": {
                "manifest": "./manifest.json"
            },
            "name": "chunk-manifest-json"
        }
    ],
    "zarr_format": 3
}
d-v-b commented 3 months ago

thanks for the bug report! I will fix this shortly, and hopefully that fix makes it into the next alpha release