TomNicholas
commented
1 day ago Okay, as the error said something about filenames I tried to run the example in the icechunk docs instead (https://icechunk.io/icechunk-python/virtual/#creating-a-virtual-dataset-with-virtualizarr), which is different files with different names on S3 instead of local and got
```python
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py:451](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py#line=450), in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
450 try:
--> 451 new_vars[k] = decode_cf_variable(
452 k,
453 v,
454 concat_characters=_item_or_default(concat_characters, k, True),
455 mask_and_scale=_item_or_default(mask_and_scale, k, True),
456 decode_times=_item_or_default(decode_times, k, True),
457 stack_char_dim=stack_char_dim,
458 use_cftime=_item_or_default(use_cftime, k, None),
459 decode_timedelta=_item_or_default(decode_timedelta, k, None),
460 )
461 except Exception as e:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py:292](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py#line=291), in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta)
291 if decode_times:
--> 292 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
294 if decode_endianness and not var.dtype.isnative:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py:1001](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py#line=1000), in CFDatetimeCoder.decode(self, variable, name)
1000 calendar = pop_to(attrs, encoding, "calendar")
-> 1001 dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
1002 transform = partial(
1003 decode_cf_datetime,
1004 units=units,
1005 calendar=calendar,
1006 use_cftime=self.use_cftime,
1007 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py:214](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py#line=213), in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
212 values = indexing.ImplicitToExplicitIndexingAdapter(indexing.as_indexable(data))
213 example_value = np.concatenate(
--> 214 [first_n_items(values, 1) or [0], last_item(values) or [0]]
215 )
217 try:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/formatting.py:97](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/formatting.py#line=96), in first_n_items(array, n_desired)
96 array = array._data
---> 97 return np.ravel(to_duck_array(array))[:n_desired]
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/namedarray/pycompat.py:138](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/namedarray/pycompat.py#line=137), in to_duck_array(data, **kwargs)
137 else:
--> 138 return np.asarray(data)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:578](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=577), in ImplicitToExplicitIndexingAdapter.__array__(self, dtype, copy)
577 if Version(np.__version__) >= Version("2.0.0"):
--> 578 return np.asarray(self.get_duck_array(), dtype=dtype, copy=copy)
579 else:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:583](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=582), in ImplicitToExplicitIndexingAdapter.get_duck_array(self)
582 def get_duck_array(self):
--> 583 return self.array.get_duck_array()
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:657](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=656), in LazilyIndexedArray.get_duck_array(self)
654 else:
655 # If the array is not an ExplicitlyIndexedNDArrayMixin,
656 # it may wrap a BackendArray so use its __getitem__
--> 657 array = self.array[self.key]
659 # self.array[self.key] is now a numpy array when
660 # self.array is a BackendArray subclass
661 # and self.key is BasicIndexer((slice(None, None, None),))
662 # so we need the explicit check for ExplicitlyIndexed
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:166](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=165), in ZarrArrayWrapper.__getitem__(self, key)
165 method = self._oindex
--> 166 return indexing.explicit_indexing_adapter(
167 key, array.shape, indexing.IndexingSupport.VECTORIZED, method
168 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:1018](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=1017), in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
1017 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
-> 1018 result = raw_indexing_method(raw_key.tuple)
1019 if numpy_indices.tuple:
1020 # index the loaded np.ndarray
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:156](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=155), in ZarrArrayWrapper._getitem(self, key)
155 def _getitem(self, key):
--> 156 return self._array[key]
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py:1660](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py#line=1659), in Array.__getitem__(self, selection)
1659 elif is_pure_orthogonal_indexing(pure_selection, self.ndim):
-> 1660 return self.get_orthogonal_selection(pure_selection, fields=fields)
1661 else:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/_compat.py:43](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/_compat.py#line=42), in _deprecate_positional_args.._inner_deprecate_positional_args..inner_f(*args, **kwargs)
42 if extra_args <= 0:
---> 43 return f(*args, **kwargs)
45 # extra_args > 0
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py:2102](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py#line=2101), in Array.get_orthogonal_selection(self, selection, out, fields, prototype)
2101 indexer = OrthogonalIndexer(selection, self.shape, self.metadata.chunk_grid)
-> 2102 return sync(
2103 self._async_array._get_selection(
2104 indexer=indexer, out=out, fields=fields, prototype=prototype
2105 )
2106 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py:141](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py#line=140), in sync(coro, loop, timeout)
140 if isinstance(return_result, BaseException):
--> 141 raise return_result
142 else:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py:100](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py#line=99), in _runner(coro)
99 try:
--> 100 return await coro
101 except Exception as ex:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py:961](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py#line=960), in AsyncArray._get_selection(self, indexer, prototype, out, fields)
959 if product(indexer.shape) > 0:
960 # reading chunks and decoding them
--> 961 await self.codec_pipeline.read(
962 [
963 (
964 self.store_path [/](http://localhost:8888/) self.metadata.encode_chunk_key(chunk_coords),
965 self.metadata.get_chunk_spec(chunk_coords, self.order, prototype=prototype),
966 chunk_selection,
967 out_selection,
968 )
969 for chunk_coords, chunk_selection, out_selection in indexer
970 ],
971 out_buffer,
972 drop_axes=indexer.drop_axes,
973 )
974 return out_buffer.as_ndarray_like()
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py:440](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py#line=439), in BatchedCodecPipeline.read(self, batch_info, out, drop_axes)
434 async def read(
435 self,
436 batch_info: Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple]],
437 out: NDBuffer,
438 drop_axes: tuple[int, ...] = (),
439 ) -> None:
--> 440 await concurrent_map(
441 [
442 (single_batch_info, out, drop_axes)
443 for single_batch_info in batched(batch_info, self.batch_size)
444 ],
445 self.read_batch,
446 config.get("async.concurrency"),
447 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:64](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=63), in concurrent_map(items, func, limit)
62 return await func(*item)
---> 64 return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:62](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=61), in concurrent_map..run(item)
61 async with sem:
---> 62 return await func(*item)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py:262](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py#line=261), in BatchedCodecPipeline.read_batch(self, batch_info, out, drop_axes)
261 else:
--> 262 chunk_bytes_batch = await concurrent_map(
263 [
264 (byte_getter, array_spec.prototype)
265 for byte_getter, array_spec, _, _ in batch_info
266 ],
267 lambda byte_getter, prototype: byte_getter.get(prototype),
268 config.get("async.concurrency"),
269 )
270 chunk_array_batch = await self.decode_batch(
271 [
272 (chunk_bytes, chunk_spec)
(...)
276 ],
277 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:64](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=63), in concurrent_map(items, func, limit)
62 return await func(*item)
---> 64 return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:62](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=61), in concurrent_map..run(item)
61 async with sem:
---> 62 return await func(*item)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/storage/common.py:71](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/storage/common.py#line=70), in StorePath.get(self, prototype, byte_range)
70 prototype = default_buffer_prototype()
---> 71 return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/icechunk/__init__.py:469](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/icechunk/__init__.py#line=468), in IcechunkStore.get(self, key, prototype, byte_range)
468 try:
--> 469 result = await self._store.get(key, byte_range)
470 except KeyNotFound as _e:
471 # Zarr python expects None to be returned if the key does not exist
472 # but an IcechunkStore returns an error if the key does not exist
ValueError: store error: unsuccessful repository operation: `error when handling virtual reference error fetching virtual reference dispatch failure`
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
Cell In[43], line 1
----> 1 ds = xr.open_zarr(
2 store,
3 zarr_format=3,
4 consolidated=False,
5 chunks={},
6 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:1337](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=1336), in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, zarr_version, zarr_format, use_zarr_fill_value_as_mask, chunked_array_type, from_array_kwargs, **kwargs)
1322 raise TypeError(
1323 "open_zarr() got unexpected keyword arguments " + ",".join(kwargs.keys())
1324 )
1326 backend_kwargs = {
1327 "synchronizer": synchronizer,
1328 "consolidated": consolidated,
(...)
1334 "zarr_format": zarr_format,
1335 }
-> 1337 ds = open_dataset(
1338 filename_or_obj=store,
1339 group=group,
1340 decode_cf=decode_cf,
1341 mask_and_scale=mask_and_scale,
1342 decode_times=decode_times,
1343 concat_characters=concat_characters,
1344 decode_coords=decode_coords,
1345 engine="zarr",
1346 chunks=chunks,
1347 drop_variables=drop_variables,
1348 chunked_array_type=chunked_array_type,
1349 from_array_kwargs=from_array_kwargs,
1350 backend_kwargs=backend_kwargs,
1351 decode_timedelta=decode_timedelta,
1352 use_cftime=use_cftime,
1353 zarr_version=zarr_version,
1354 use_zarr_fill_value_as_mask=use_zarr_fill_value_as_mask,
1355 )
1356 return ds
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/api.py:671](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/api.py#line=670), in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
659 decoders = _resolve_decoders_kwargs(
660 decode_cf,
661 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...)
667 decode_coords=decode_coords,
668 )
670 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 671 backend_ds = backend.open_dataset(
672 filename_or_obj,
673 drop_variables=drop_variables,
674 **decoders,
675 **kwargs,
676 )
677 ds = _dataset_from_backend_dataset(
678 backend_ds,
679 filename_or_obj,
(...)
689 **kwargs,
690 )
691 return ds
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:1427](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=1426), in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel, zarr_version, zarr_format, store, engine, use_zarr_fill_value_as_mask)
1425 store_entrypoint = StoreBackendEntrypoint()
1426 with close_on_error(store):
-> 1427 ds = store_entrypoint.open_dataset(
1428 store,
1429 mask_and_scale=mask_and_scale,
1430 decode_times=decode_times,
1431 concat_characters=concat_characters,
1432 decode_coords=decode_coords,
1433 drop_variables=drop_variables,
1434 use_cftime=use_cftime,
1435 decode_timedelta=decode_timedelta,
1436 )
1437 return ds
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/store.py:46](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/store.py#line=45), in StoreBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta)
43 vars, attrs = filename_or_obj.load()
44 encoding = filename_or_obj.get_encoding()
---> 46 vars, attrs, coord_names = conventions.decode_cf_variables(
47 vars,
48 attrs,
49 mask_and_scale=mask_and_scale,
50 decode_times=decode_times,
51 concat_characters=concat_characters,
52 decode_coords=decode_coords,
53 drop_variables=drop_variables,
54 use_cftime=use_cftime,
55 decode_timedelta=decode_timedelta,
56 )
58 ds = Dataset(vars, attrs=attrs)
59 ds = ds.set_coords(coord_names.intersection(vars))
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py:462](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py#line=461), in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
451 new_vars[k] = decode_cf_variable(
452 k,
453 v,
(...)
459 decode_timedelta=_item_or_default(decode_timedelta, k, None),
460 )
461 except Exception as e:
--> 462 raise type(e)(f"Failed to decode variable {k!r}: {e}") from e
463 if decode_coords in [True, "coordinates", "all"]:
464 var_attrs = new_vars[k].attrs
ValueError: Failed to decode variable 'time': store error: unsuccessful repository operation: `error when handling virtual reference error fetching virtual reference dispatch failure`
```
Is there something wrong with my environment / zarr version?
mpiannucci
norlandrhagen
Not sure if this is virtualizarr or icechunk's fault but the error was raised inside icechunk so I'm raising it here.
MCVE
Error
Environment