Kerchunked dataset unstable for distributed dask arrays

[mpiannucci]

Running this code after loading in the multizarr will give a different result over and over again. When the same code is run against the single zarr, there is no such instability.

EDIT: Nevermind happens with single zarr too :(

There is probably about a 50% hit rate with getting garbage or correct data

[mpiannucci]

This only happens when loading the dataset in as a dask array, aka when setting chunks: {'somthing': 0} on dataset load.

When loaded in as numpy arrays its stable.

[mpiannucci]

This issue also doesnt seem to happen with cfgrib, so likely an encoding or serialization issue somewhere along the line with dask

[mpiannucci]

This happens when the xarray caches from the dask client but the dask client doesnt have the gribberish codec. So need to test after getting the module installable on dask workers

[mpiannucci]

gribberish codec now works as it should, still saw some instability tho, but not sure what is cache and what is not

[mpiannucci]

This is still happening :(

[mpiannucci]

The error here is that one of the chunks for some reason isnt getting written to the kerchunk file, so no error is thrown. When the chunk is manually added back it everything works