In the degridding procedure, we degrid all visibilities. This includes the autos, which imposes a slight memory problem.
Problem: We degrid by region, choosing a square in uv-space and selecting all visibilities which fully fall within this square and degrid them. Then, we move the square and repeat, covering the full uv-space. When we pass over the u=0,v=0 point, we suddenly have all the autos to degrid at once. This results in a memory spike up to roughly 50Gb for that loop.
Solution: Add an option which splits up the degridding process into sub-loops if it surpasses an estimated memory usage. There already exists something similar for source dfts (conserve_memory) so I've co-opted that keyword.
Here is the memory usage for a run with this sublooping. Solid yellow is memory usage, dashed yellow is my memory allocation. Without the sublooping, there are two memory spike at 50Gb (one for each polarization). We now have a relatively flat memory usage profile, and can run a full observation with all the current bells-and-whistles in under 32Gb.
Note to the reviewer:
There are two commits, one with the actual loop content and one with the indentation. I suggest looking at the actual loop content commit for the review.
In the degridding procedure, we degrid all visibilities. This includes the autos, which imposes a slight memory problem.
Problem: We degrid by region, choosing a square in uv-space and selecting all visibilities which fully fall within this square and degrid them. Then, we move the square and repeat, covering the full uv-space. When we pass over the u=0,v=0 point, we suddenly have all the autos to degrid at once. This results in a memory spike up to roughly 50Gb for that loop.
Solution: Add an option which splits up the degridding process into sub-loops if it surpasses an estimated memory usage. There already exists something similar for source dfts (
conserve_memory) so I've co-opted that keyword.Here is the memory usage for a run with this sublooping. Solid yellow is memory usage, dashed yellow is my memory allocation. Without the sublooping, there are two memory spike at 50Gb (one for each polarization). We now have a relatively flat memory usage profile, and can run a full observation with all the current bells-and-whistles in under 32Gb.
Note to the reviewer: There are two commits, one with the actual loop content and one with the indentation. I suggest looking at the actual loop content commit for the review.