Introduce a new scan activity state called "await_pipeline". This serves two main purposes:
It allows the telescope to continue tracking the calibrator while signalling the end of the solution interval. Previously we slammed on the brakes on the antennas to go to "stop" just to trigger the pipeline in calibrate_delays / bf_phaseup, which is clearly a kludge. It also makes the data during this "stop" scan less valuable. The new scheme ensures extra valid data in the archive.
It provides an opportunity to flush any pending calibration products from the pipeline. This is important for reference pointing, which needs to do the beam fitting after all the offset data has been processed. This is actually the main driver for this functionality.
When the observation script sets the "await_pipeline" activity, it triggers a break in the Accumulator task and additionally sends a PipelineSyncEvent to the Pipeline task. The Pipeline task processes this event by running a flush_pipeline function which currently does nothing, but will do reference pointing beam fitting in future.
The data captured during the "await_pipeline" scan is ignored by the pipeline, since this activity is in the ignore_states list in the _is_break method. The cal pipeline therefore treats this scan as a slew, while katdal will eventually expose this as a track for the benefit of other users.
In the process I also shuffled code around to more logical places and consolidated and reworked test_control to reduce redundancy, so it's probably worth reviewing this commit by commit. The bulk of the PR is actually just moving stuff around and the main action happens in commits 177ffaa and c54e8eb, which only adds about 60 lines of code.
Introduce a new scan activity state called "await_pipeline". This serves two main purposes:
It allows the telescope to continue tracking the calibrator while signalling the end of the solution interval. Previously we slammed on the brakes on the antennas to go to "stop" just to trigger the pipeline in calibrate_delays / bf_phaseup, which is clearly a kludge. It also makes the data during this "stop" scan less valuable. The new scheme ensures extra valid data in the archive.
It provides an opportunity to flush any pending calibration products from the pipeline. This is important for reference pointing, which needs to do the beam fitting after all the offset data has been processed. This is actually the main driver for this functionality.
When the observation script sets the "await_pipeline" activity, it triggers a break in the Accumulator task and additionally sends a
PipelineSyncEventto the Pipeline task. The Pipeline task processes this event by running aflush_pipelinefunction which currently does nothing, but will do reference pointing beam fitting in future.The data captured during the "await_pipeline" scan is ignored by the pipeline, since this activity is in the
ignore_stateslist in the_is_breakmethod. The cal pipeline therefore treats this scan as a slew, while katdal will eventually expose this as a track for the benefit of other users.In the process I also shuffled code around to more logical places and consolidated and reworked
test_controlto reduce redundancy, so it's probably worth reviewing this commit by commit. The bulk of the PR is actually just moving stuff around and the main action happens in commits 177ffaa and c54e8eb, which only adds about 60 lines of code.