When attempting to perform a full-scale run, I encountered OOM errors during the sky region ideal coverage stage in vast_pipeline.pipeline.utils.get_src_skyregion_merged_df.
The current implementation creates a Cartesian product of all sources and sky regions and filters the results by a separation threshold from there. For a full-scale VAST-P1 run with almost 1 million sources and almost 1,000 sky regions, this approach requires a lot of memory.
When attempting to perform a full-scale run, I encountered OOM errors during the sky region ideal coverage stage in
vast_pipeline.pipeline.utils.get_src_skyregion_merged_df.The current implementation creates a Cartesian product of all sources and sky regions and filters the results by a separation threshold from there. For a full-scale VAST-P1 run with almost 1 million sources and almost 1,000 sky regions, this approach requires a lot of memory.