caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
29 stars 6 forks source link

caracal died with <Signals.SIGKILL: 9>. #1503

Open viralp opened 1 year ago

viralp commented 1 year ago

Hi,

I am getting the following error in the selfcal worker. I am also getting the similar error in stimela (imaging with wsclean), so suspecting something wrong in stimela.

#  == Constructing PSF ==
# Precalculating weights for Briggs'(0) weighting...
# Opening reordered part 3 spw 0 for /stimela_mount/msdir/A1795_raw_vis-ACO1795-corr.ms
# Opening reordered part 3 spw 0 for /stimela_mount/msdir/A1795_raw_vis-ACO1795-corr.ms
# Determining min and max w & theoretical beam size... DONE (w=[0.0638248:23803.1] lambdas, maxuvw=32936.3 lambda)
# Gridding pass 0... Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 69, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['wsclean', '-name', '/stimela_mount/output/continuum/image_1/mypipelinerun_ACO1795_1', '-j', '5', '-mem', '100', '-absmem', '100.0', '-weight', 'briggs', '0.0', '-no-mfs-weighting', '-size', '6000', '6000', '-scale', '1.5asec', '-channels-out', '8', '-nwlayers-factor', '3', '-pol', 'I', '-data-column', 'DATA', '-niter', '1000000', '-auto-threshold', '0.5', '-auto-mask', '20.0', '-gain', '0.1', '-mgain', '0.9', '-join-channels', '-multiscale', '-fit-spectral-pol', '4', '-padding', '1.3', '-save-source-list', '-parallel-deconvolution', '1029', '/stimela_mount/msdir/A1795_raw_vis-ACO1795-corr.ms']' died with <Signals.SIGKILL: 9>.
2023-04-18 16:29:48 CARACal.Stimela.image-field0-iter1 ERROR: cd /export/home/viralp/A1795/meerkat_data/.stimela_workdir-1681855983231505 && singularity run --workdir /export/home/viralp/A1795/meerkat_data/.stimela_workdir-1681855983231505 --containall returns error code 1
2023-04-18 16:29:48 CARACal.Stimela.image-field0-iter1 ERROR: job failed at 2023-04-18 16:29:48.598197 after 0:16:26.311555
2023-04-18 16:29:48 CARACal ERROR: Job 'image-field0-iter1:: Make wsclean image (selfcal iter 1)' failed: cd /export/home/viralp/A1795/meerkat_data/.stimela_workdir-1681855983231505 && singularity run --workdir /export/home/viralp/A1795/meerkat_data/.stimela_workdir-1681855983231505 --containall returns error code 1 [PipelineException]
2023-04-18 16:29:48 CARACal INFO:   More information can be found in the logfile at output/logs-20230418-161259/log-caracal.txt
2023-04-18 16:29:48 CARACal INFO: exiting with error code 1
[log-caracal.txt](https://github.com/caracal-pipeline/caracal/files/11266980/log-caracal.txt)
KshitijT commented 1 year ago

@viralp is this still an issue for you or did updating stimela fix it?

Athanaseus commented 5 months ago

@viralp has this come up again?