search

casangi / xradio

Xarray Radio Astronomy Data IO
https://xradio.readthedocs.io/en/latest/
Other
9 stars 5 forks source link

convert_msv2_to_processing_set excessive memory usage #214

Open maneesh29s opened 1 month ago

maneesh29s commented 1 month ago

We have a simulated MSv2 data which we use for testing purposes on our workstations. The data has following dimensions:

Time: 120 Baseline: 1,30,816 Channels: 150 Polarizations: 1 (XX)

The overall size of the data is around 16 GB.

Till xradio v0.0.31, the convert_msv2_to_processing_set function worked fine.

Since xradio v0.0.33, the converter process asks for too much memory and eventually gets killed. On our workstation with 64 GB RAM, the process reached the peak memory usage of 64 GB and later got killed by the os.

This issue is observed in xradio v0.0.33 and xradio v0.0.34

Jan-Willem commented 1 month ago

@maneesh29s try setting use_table_iter=True in convert_msv2_to_processing_set. Is there a link where I can download the dataset?

maneesh29s commented 1 month ago

Setting use_table_iter=True works with v0.0.34.