Streaming forward modelling will solve the issue of being able to perform high quality simulations of DSA and similar instruments without storing the data.
Current Resource Reqs.
The data volume for a single pol, single channel 10.3min DSA2k (in visibilities) is 60GB. From a time perspective this requires approximately 1 CPU-hour (lower bound as this does not include a sophisticated sky model nor calibration nor accounting for smearing). Memory reqs were not measured however generally there will be multiples of visibilities (1 for residuals, 1 per calibrator). So for a full band pointing this would require 1.92PB storage, and > 32k CPU-hours. For a single mosaic of 10 pointings, this is clearly not feasible.
With streaming resource reqs.
With the streaming forward modelling we will discard the visibilities after they are used to produce the image. The only storage requirement with be a two copies (one data, one residuals) of size solution interval of data 40chan x 6s (16GB per worker if sharded by solution intervals, and 3.2TB in total). The number of CPU-hour will remain constant, however much of this can be accelerated in GPU. Memory requirements will be 1 copy of data + 1 residual + 1 pre calibrator. With 4 calibrators (probably too few) this would be 48GB memory required. Memory reqs for calibration still need to be resolved #92 .
Streaming forward modelling will solve the issue of being able to perform high quality simulations of DSA and similar instruments without storing the data.
Current Resource Reqs.
The data volume for a single pol, single channel 10.3min DSA2k (in visibilities) is 60GB. From a time perspective this requires approximately 1 CPU-hour (lower bound as this does not include a sophisticated sky model nor calibration nor accounting for smearing). Memory reqs were not measured however generally there will be multiples of visibilities (1 for residuals, 1 per calibrator). So for a full band pointing this would require 1.92PB storage, and > 32k CPU-hours. For a single mosaic of 10 pointings, this is clearly not feasible.
With streaming resource reqs.
With the streaming forward modelling we will discard the visibilities after they are used to produce the image. The only storage requirement with be a two copies (one data, one residuals) of size solution interval of data 40chan x 6s (16GB per worker if sharded by solution intervals, and 3.2TB in total). The number of CPU-hour will remain constant, however much of this can be accelerated in GPU. Memory requirements will be 1 copy of data + 1 residual + 1 pre calibrator. With 4 calibrators (probably too few) this would be 48GB memory required. Memory reqs for calibration still need to be resolved #92 .
Tasks