ratt-ru / dask-ms

Implementation of a dask/xarray dataset backed by a CASA MS
https://dask-ms.readthedocs.io
Other
19 stars 7 forks source link

Add experimental code for data fragments. #282

Closed JSKenyon closed 1 year ago

JSKenyon commented 1 year ago
JSKenyon commented 1 year ago

Ok, I think that this is ready for another pair of eyes, if only to sanity check what I have done so far. I think that it is pretty simple. One thing to note is that I elected not to use the current CLI infrastructure. I did make an attempt but ran into issues with nested subparsers.

Currently the CLI is very basic and only provides the option to stat or rebase a fragment. The first of these simply reports the parents of the target fragment. The second allows a user to modify the parent in place. This is useful if you want to exclude bad/irrelevant parents.

The CLI could optionally be extended with the following, more complicated functionality:

JSKenyon commented 1 year ago

This PR doesn't depend on #284 but that PR is likely also required for this functionality to be exploited as it is sometimes necessary to rechunk data being written to a fragment due to zarr chunk size limits.

sjperkins commented 1 year ago

Could you also please rebase this PR on master?

JSKenyon commented 1 year ago

I have rebased to master. I hope I did it correctly - I haven't had much practice with rebase.