This was a subtle bug, introduced by the read -> compute -> write -> read pattern in the interpolation. Parameterized terms do not have gains at the point of the first write but they do have the gain coordinates. This leads to the gain coordinates being chunked as 1D arrays, rather than consistently with the potentially very large gain array. Subsequent writes to the output directory were then seeing the coarsely chunked coordinates and raising errors as those chunks would cause the gains to exceed the zarr maximum chunk size. This PR very simply deletes the temporary interpolation products once the interpolation is complete, ensuring that any subsequent writes are correct.
Edit: Opted for another approach - deletion, while simple, actually introduces other problems. Now, simply create dummy gains on initial write to ensure chunking remains consistent.
Edit 2: Opted for a third approach which is just to write the intermediary products to a sub-directory which gets tidied up at exit.
This was a subtle bug, introduced by the read -> compute -> write -> read pattern in the interpolation. Parameterized terms do not have gains at the point of the first write but they do have the gain coordinates. This leads to the gain coordinates being chunked as 1D arrays, rather than consistently with the potentially very large gain array. Subsequent writes to the output directory were then seeing the coarsely chunked coordinates and raising errors as those chunks would cause the gains to exceed the zarr maximum chunk size. This PR very simply deletes the temporary interpolation products once the interpolation is complete, ensuring that any subsequent writes are correct.
Edit: Opted for another approach - deletion, while simple, actually introduces other problems. Now, simply create dummy gains on initial write to ensure chunking remains consistent.
Edit 2: Opted for a third approach which is just to write the intermediary products to a sub-directory which gets tidied up at exit.