Gainplot fails to generate (B0.png)

[LeonMtshweni]

• Ragavi running through STIMELA version: 1.7.6
• Python version: 3.9
• Operating System: Ubuntu 22.04.3 LTS

Description

After writing the B0 solutions to the file, caracal failed to generate the B0.png file. K0.png and G0.png files were made successfully

the crosscal worker fails with the following error

2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR: job failed at 2024-02-02 17:50:46.710062 after 0:01:50.928017
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR: Traceback (most recent call last):
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:   File "/idia/users/mtshweni/caracal_venv/lib/python3.9/site-packages/stimela/recipe.py", line 713, in run
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:     job.run_job()
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:   File "/idia/users/mtshweni/caracal_venv/lib/python3.9/site-packages/stimela/recipe.py", line 425, in run_job
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:     self.job.run(output_wrangler=self.apply_output_wranglers)
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:   File "/idia/users/mtshweni/caracal_venv/lib/python3.9/site-packages/stimela/singularity.py", line 123, in run
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:     utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:   File "/idia/users/mtshweni/caracal_venv/lib/python3.9/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR:     raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
2024-02-02 17:50:46 CARACal.Stimela.plotgains-B-0-0 ERROR: stimela.utils.StimelaCabRuntimeError: cd /idia/users/mtshweni/experiments/Feb24_01/.stimela_workdir-17068689100189965 && singularity run --workdir /idia/users/mtshweni/experiments/Feb24_01/.stimela_workdir-17068689100189965 --containall returns error code 1
2024-02-02 17:50:46 CARACal.Stimela.crosscal INFO: Completed jobs : ['save-FullBandWidth32kModeData_crosscal_before-ms0', 'delay_cal-1gc1-0-0-primary', 'plotgains-K-0-0', 'gain_cal-1gc1-0-0-primary', 'plotgains-G-0-0', 'bp_cal-1gc1-0-0-primary']
2024-02-02 17:50:46 CARACal.Stimela.crosscal INFO: Remaining jobs : ['apply_gains-fcal--0', 'auto_flagging-1gc1-0-0-primary', 'delay_cal-1gc1-1-0-primary', 'plotgains-K-0-0', 'gain_cal-1gc1-1-0-primary', 'plotgains-G-0-0', 'bp_cal-1gc1-1-0-primary', 'plotgains-B-0-0', 'apply_gains-bpcal-1gc1-0', 'save-FullBandWidth32kModeData_crosscal_after-ms0', 'summary-1gc1-0']

Bandpass solutions were made successfully, but there was a lof of flagging after

 #  Insufficient unflagged antennas to proceed with this solution.
#    (time=2020/02/29/13:50:41.0 field=0 spw=0 chan=4969) 

What I Did

I routinely ran the same script I'd been running all along, just that for some reason plots failed to generate and caused the entire session to fail

log-caracal (3).txt

[Athanaseus]

Hi @LeonMtshweni, thanks for opening this.

Ragavi running through STIMELA version: 1.7.6 Python version: 3.9 Operating System: Ubuntu 22.04.3 LTS

The setup is ok.

It is not clear what could have gone wrong. Indeed, the ragavi task didn't succeed when plotting .B0 table.

# 2024-02-02 17:48:56: Running ragavi-gains --table /stimela_mount/msdir/FullBandWidth32kModeData-1582940762_sdp_l0-1gc1_primary.B0 --corr  --cmap coolwarm --field 0 --htmlname /stimela_mount/output/FullBandWidth32kModeData-1582940762_sdp_l0-1gc1_primary.B0 --plotname /stimela_mount/output/FullBandWidth32kModeData-1582940762_sdp_l0-1gc1_primary.B0.png
# 02.02.2024@17:48:58 - ragavi.ragavi        - INFO       - Table type: B Jones
# 02.02.2024@17:48:58 - ragavi.ragavi        - INFO       - Spw: 0, Field: J1939-6342, Corr: 0 amplitude
# 02.02.2024@17:48:58 - ragavi.ragavi        - INFO       - Table type: B Jones
# 2024-02-02 17:50:46: ragavi-gains exited with code -11

Is this error persistent when re-running the worker?

[LeonMtshweni]

I tried it again over the weekend and I got the same result.

I should also add that I'm working with 8805 channels from a 32k dataset in the region 1290-1520 MHz.

[Athanaseus]

Alright then it has something to do with the table. Is there a way to plot it using CASA plotcal and see the output? Or if possible share the table I will test it locally. (With CASA or ragavi-gains --table FullBandWidth32kModeData-1582940762_sdp_l0-1gc1_primary.B0 --corr --cmap coolwarm --field 0 --htmlname FullBandWidth32kModeData-1582940762_sdp_l0-1gc1_primary.B0 --plotname FullBandWidth32kModeData-1582940762_sdp_l0-1gc1_primary.B0.png)

[Athanaseus]

Please re-open if you still experiencing this issue.