transform worker field selection

[AstroRipples]

Hi folks,

Encountered a new (silly) issue when trying to process MGCLS data with CARACal. The continuum calibration has completed successfully and I'm now trying to apply the solutions to the target field. However, in the case of this observing run, the primary calibrator PKS B1934-638 (I'm having none of this J2000 nonsense) was observed as both flux/bandpass and interleaved gain calibrator.

They're in the MS under two different field names, and so CARACal's transform worker is getting confused and can't decide which to use. The full log is at the end of this post, and the excerpt from applycal() is below:

... 
# 2023-03-13 10:48:40   INFO    OldCalibrater::setcallib2(callib)       Arranging to APPLY:
# 2023-03-13 10:48:40   INFO            .   G Jones: table=/stimela_mount/output/caltables/PipelineRun_Continuum-1551621148-1kgb_secondary.F0 (by cal library) calWt=false
# 2023-03-13 10:48:40   SEVERE  OldCalibrater::setcallib2(callib)       Found more than one field at minimum distance, can't decide! Check inputs and try again.
# 2023-03-13 10:48:40   SEVERE  mstransform:::: Error in Calibrater::callib2.
# 2023-03-13 10:48:40   INFO    mstransform:::: ##### End Task: mstransform          #####
# 2023-03-13 10:48:40   INFO    mstransform::::+        ##########################################

Having looked through the documentation of the transform worker, I don't see a way to specify which field the gains are to be applied from. I know I can do this manually in CASA (using the gainfield input to applycal), and will use this route to keep momentum going, but I thought I'd raise this issue.

Log file: log-caracal_split.txt

[KshitijT]

@AstroRipples , off the top of my head, this is happening because of the fldmap being nearest for G-Jones application (then since both the fields in F0 tables are at the same distance, there would be confusion for mstransform). To confirm, could you please share output/caltables/callib-PipelineRun_Continuum-1551621148-1kgb-transform__2.txt ?

[AstroRipples]

Hi @KshitijT 👋 thanks, yeah I suspect that's exactly what's happening. Log file attached: callib-PipelineRun_Continuum-1551621148-1kgb-transform__2.txt

[KshitijT]

I think this is a bit hardcoded at the moment, will try and make options available.

[SpheMakh]

I think this should fix it:

https://github.com/caracal-pipeline/caracal/blob/6e85065d214f0cd133c5c8ba6265559dd485d9da/caracal/workers/crosscal_worker.py#L694-L696

The fix is on branch issue-1474

[AstroRipples]

Hi @SpheMakh 👋 thanks for pushing this fix... I've re-installed CARACal but I think I may be encountering the same issue still, so either it's not fixed or (possibly) I pulled the wrong version, so I wanted to double-check. I've performed the initial calibration and CARACal is at the stage of generating inspection plots, but the inspect worker is failing with a list index out of range error when trying to plot the secondary...

I suspect this means that CARACal is merging the two different PKSB1934-638 fields into one, which may have knock-on problems (#1509). The traceback is below and the full log is appended at the end.

# 2023-07-06 03:37:06: Running ragavi-vis --ms /stimela_mount/msdir/1551621148-cal.MS --xaxis antenna1 --yaxis amp --canvas-height 720 --canvas-width 1080 --corr XX,YY --data-column CORRECTED_DATA --field J1939gaincal --htmlname /stimela_mount/output/1kgb-1551621148-cal-gcal-J1939gaincal-amp_ant.html --iter-axis corr --mem-limit 8GB --num-cores 8
# 06.07.2023@03:37:08 - ragavi.visibilities  - INFO       - Total RAM size: ~503.80 GB
# 06.07.2023@03:37:08 - ragavi.visibilities  - INFO       - Total number of Cores: 128
# 06.07.2023@03:37:08 - ragavi.visibilities  - INFO       - Using 8 cores
# 06.07.2023@03:37:08 - ragavi.visibilities  - INFO       - Memory limit per core: 8GB
# 06.07.2023@03:37:08 - ragavi.visibilities  - INFO       - Available corrs: XX,XY,YX,YY
# 06.07.2023@03:37:08 - ragavi.visibilities  - ERROR      - MS data acquisition failed
# 06.07.2023@03:37:08 - ragavi.visibilities  - ERROR      - list index out of range
# 2023-07-06 03:37:09: ragavi-vis exited with code 255
# INFO:    Cleaning up image...
2023-07-06 03:37:10 CARACal.Stimela.plot-amp_ant-0-gcal ERROR: cd /local/work/riseley/MeerKAT/Abell_3667/.stimela_workdir-1688606140235793 && singularity run --userns --workdir /local/work/riseley/MeerKAT/Abell_3667/.stimela_workdir-1688606140235793 --containall returns error code 1
2023-07-06 03:37:10 CARACal.Stimela.plot-amp_ant-0-gcal ERROR: job failed at 2023-07-06 03:37:10.078630 after 0:00:06.496338
2023-07-06 03:37:10 CARACal ERROR: Job '1kgb' failed: cd /local/work/riseley/MeerKAT/Abell_3667/.stimela_workdir-1688606140235793 && singularity run --userns --workdir /local/work/riseley/MeerKAT/Abell_3667/.stimela_workdir-1688606140235793 --containall returns error code 1 [PipelineException]
2023-07-06 03:37:10 CARACal INFO:   More information can be found in the logfile at OutputData/logs-20230705-111123/log-caracal.txt

Full log: log-caracal-20230705.txt