ratt-ru / dask-ms

Implementation of a dask/xarray dataset backed by a CASA MS
https://dask-ms.readthedocs.io
Other
19 stars 7 forks source link

Optimise getcol memory usage #9

Closed sjperkins closed 6 years ago

sjperkins commented 6 years ago

Previously, np.concatenate was used to aggregate all row runs in a chunk. This meant that we needed twice us much memory - one set for all row runs and one for the concatenate result. Now, the result is preallocated, and pyrap.tables.table.getcolnp is used to dump data directly into the result array.