We are attempting to run 3,188 images of a single field through the pipeline. Each image is 10 sec (1 integration) long. The full integration model was subtracted from the visibilities before imaging each integration.
The pipeline is failing at the measurement pairs calculation step. The error is likely memory-related as a spike in memory is observed before the failure. This isn't surprising considering that the work distribution was designed with the regular VAST survey in mind (i.e. several images of many fields, rather than many images of one field).
We could reoptimize the work distribution or add an option to skip measurement pairs. Reoptimizing the work distribution would be great but requires significant development time, so my preference is to implement an option to skip measurement pair calculation.
For the case I mention above, I don't expect that missing the measurement pairs will have a significant impact on the science analysis. The measurement pair metrics are most useful for identifying variable sources among persistent sources. Since the long integration data has been subtracted, theoretically all the remaining sources in the dataset are sources of interest or artefacts.
We are attempting to run 3,188 images of a single field through the pipeline. Each image is 10 sec (1 integration) long. The full integration model was subtracted from the visibilities before imaging each integration.
The pipeline is failing at the measurement pairs calculation step. The error is likely memory-related as a spike in memory is observed before the failure. This isn't surprising considering that the work distribution was designed with the regular VAST survey in mind (i.e. several images of many fields, rather than many images of one field).
We could reoptimize the work distribution or add an option to skip measurement pairs. Reoptimizing the work distribution would be great but requires significant development time, so my preference is to implement an option to skip measurement pair calculation.
For the case I mention above, I don't expect that missing the measurement pairs will have a significant impact on the science analysis. The measurement pair metrics are most useful for identifying variable sources among persistent sources. Since the long integration data has been subtracted, theoretically all the remaining sources in the dataset are sources of interest or artefacts.