ratt-ru / dask-ms

Implementation of a dask/xarray dataset backed by a CASA MS
https://dask-ms.readthedocs.io
Other
19 stars 7 forks source link

Work around casacore getcolslice caching #107

Closed sjperkins closed 4 years ago

sjperkins commented 4 years ago
sjperkins commented 4 years ago

@JSKenyon Could you please review?

With this PR, I now see the following behaviour https://github.com/ska-sa/dask-ms/issues/92#issuecomment-618380352

This change may have impacts on I/O but I believe its an acceptable change for now and I'd be happy to review other proposed solutions in future.

sjperkins commented 4 years ago

It also might be reasonable to set the cache size to 4MB in the case of getcolslice cases

sjperkins commented 4 years ago

Although https://github.com/casacore/casacore/issues/1018#issuecomment-618333792 would mitigate against a 4MB cache.

JSKenyon commented 4 years ago

Going to plumb this in now. Will let you know how it goes.

JSKenyon commented 4 years ago

I have tried this out - seems to work correctly. There is a small impact on performance but well worth the hit to prevent the memory bloat.