Support for concat of files with unequal length in concat dim

fsspec / kerchunk

Cloud-friendly access to archival data

https://fsspec.github.io/kerchunk/

MIT License

310 stars 80 forks source link

Support for concat of files with unequal length in concat dim #76

Open cisaacstern opened 3 years ago

cisaacstern commented 3 years ago

@jbusecke and I are wondering if https://github.com/intake/fsspec-reference-maker/blob/89d32322fdd3ba631a4fe846475a8857f10ad872/fsspec_reference_maker/combine.py#L313-L316

should instead check that v / ds0.dims[k] > 1.

We were working with the Pangeo Forge HDFReference recipe class today, and it seemed that this change was needed to concatenate a list of files with varying lengths in the time dimension.

Thoughts, @martindurant ?

cc @rabernat

martindurant commented 3 years ago

True, every example so far has been regular in this sense. I am still missing a fully general way to form the correct zarr key names for a set of input nd-arrays; given that the concat coord(s) may be coordinates of those arrays or not, and you might have multiple variables too. The code in the combine module basically deals with the set of examples that I've seen so far.