keller-mark / pizzarr

Slice into Zarr arrays in R 🍕
https://keller-mark.github.io/pizzarr/
MIT License
25 stars 2 forks source link

Read and use consolidated metadata when available #77

Closed keller-mark closed 1 month ago

keller-mark commented 1 month ago

User story

Preferred solution

Possible alternatives

dblodgett-usgs commented 1 month ago

I've run into this in spades with a use case I'm implementing over in https://github.com/DOI-USGS/rnz

The pattern has to iterate through all the arrays in a zarr group and read all the metadata -- essentially building consolidated metadata.

My toy example is:

>     zdump(r)
zarr {
dimensions:
t = 5 ;
y = 3 ;
x = 5 ;
variables:
    <f8 data(t, y, x) ;
    <f8 i(y, x) ;
    <f8 j(y, x) ;
    <f8 t(t) ;

// global attributes:
}

But when there are lots of variables, it's painfully slow.

How did you imagine this working? I can start working up an implementation.

dblodgett-usgs commented 1 month ago

Worked up a proposed pattern to pull down and cache consolidated metadata on initialization of an http store. Does this work for you @keller-mark ?

dblodgett-usgs commented 1 month ago

I think with #82 closed, we could call this done?