Jenkins Singularity run fails

For all the details please see (whoever can): https://jenkins.meqtrees.net/job/PR_meerkathi/43/console
Notice this does not happen on my own laptop
# 2020-05-10 05:54:39   INFO    flagmanager::::
# 2020-05-10 05:54:39   INFO    flagmanager::::+    ##########################################
# 2020-05-10 05:54:39   INFO    flagmanager::::+    ##### Begin Task: flagmanager        #####
# 2020-05-10 05:54:39   INFO    flagmanager:::: flagmanager(vis="/stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms",mode="save",versionname="mypipelinerun_selfcal_before",oldname="",comment="",
# 2020-05-10 05:54:39   INFO    flagmanager::::+            merge="replace")
# 2020-05-10 05:54:39   INFO    flagmanager::AgentFlagger::open Table type is Measurement Set
# 2020-05-10 05:54:39   INFO    flagmanager:::: Save current flagversions to mypipelinerun_selfcal_before
# 2020-05-10 05:54:39   INFO    FlagVersion::saveFlagVersion    Creating new backup flag file called mypipelinerun_selfcal_before
# 2020-05-10 05:54:39   INFO    flagmanager:::: ##### End Task: flagmanager          #####
# 2020-05-10 05:54:39   INFO    flagmanager::::+    ##########################################
2020-05-10 05:54:40 CARACal.Stimela.save-mypipelinerun_selfcal_before-ms0 INFO: job complete at 2020-05-10 05:54:40.427501 after 0:00:07.845252
2020-05-10 05:54:40 CARACal.Stimela.image-field0-iter1 INFO: job started at 2020-05-10 05:54:40.428312

# 
# WSClean version 2.8 (2019-09-16)
# This software package is released under the GPL version 3.
# Author: André Offringa (offringa@gmail.com).
# 
# === IMAGING TABLE ===
#        # Pol Ch JG ²G In Freq(MHz)
# | Independent group:
# +-+-J- 0  I   0  0  0  0  1415-1418 (16)
#   |
#   +-J- 1  I   1  0  1  0  1418-1422 (16)
#   |
#   +-J- 2  I   2  0  2  0  1422-1425 (16)
# 
# Reordering /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms into 3 x 1 parts.

# Reordering: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
# Initializing model visibilities: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%

# Precalculating MF weights for Briggs'(0) weighting...
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Detected 240.2 GB of system memory, usage not limited.
#  == Constructing PSF ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Determining min and max w & theoretical beam size... DONE (w=[0.015092:6429.14] lambdas, maxuvw=7346.42 lambda)
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Suggested number of w-layers: 24
# Will process 24/24 w-layers per pass.
# Gridding pass 0...
# Rows that were required: 107136/0
# Fourier transforms...
# Gridded visibility count: 1.30138e+06
# Fitting beam... major=38.44'', minor=35.15'', PA=2.85 deg, theoretical=28.08''.
# Writing psf image... DONE
#  == Constructing PSF ==
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Determining min and max w & theoretical beam size... DONE (w=[0.0151276:6444.3] lambdas, maxuvw=7363.74 lambda)

# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 770602
# Fitting beam... major=39.44'', minor=35.82'', PA=10.99 deg, theoretical=28.01''.
# Writing psf image... DONE
#  == Constructing PSF ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Determining min and max w & theoretical beam size... DONE (w=[0.0151633:6459.46] lambdas, maxuvw=7381.07 lambda)

# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30246e+06
# Fitting beam... major=38.07'', minor=34.99'', PA=2.72 deg, theoretical=27.95''.
# Writing psf image... DONE
#  == Constructing image ==

# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30138e+06
# Writing dirty image...
#  == Constructing image ==
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 770602
# Writing dirty image...
#  == Constructing image ==

# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...

# Gridded visibility count: 1.30246e+06
# Writing dirty image...
#  == Cleaning (1) ==

# Estimated standard deviation of background noise: 4.07 mJy
# Lowest RMS in image: 3.7 mJy
# Initial peak: 757.85 mJy at 917,896
# Major iteration threshold reached global threshold of 36.97 mJy: final major iteration.
# Performed 24 iterations in total, 24 in this major iteration with sub-minor optimization.
# Stopped on peak 35.86 mJy, because the threshold was reached.
# Auto-masking threshold reached; continuing next major iteration with deeper threshold and mask.
#  == Converting model image to visibilities ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Converting model image to visibilities ==

# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Converting model image to visibilities ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Constructing image ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30138e+06
#  == Constructing image ==

# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 770602
#  == Constructing image ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...

# Gridded visibility count: 1.30246e+06
#  == Cleaning (2) ==
# Estimated standard deviation of background noise: 2.02 mJy
# Initial peak: 265.14 mJy at 917,896
# Next major iteration at: 26.51 mJy
# Performed 65 iterations in total, 41 in this major iteration with sub-minor optimization.
# Stopped on peak 25.8 mJy, because the minor-loop threshold was reached. Continuing cleaning after inversion/prediction round.
#  == Converting model image to visibilities ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Converting model image to visibilities ==

# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Converting model image to visibilities ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Constructing image ==

# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30138e+06
#  == Constructing image ==
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 770602
#  == Constructing image ==

# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...

# Gridded visibility count: 1.30246e+06
#  == Cleaning (3) ==
# Estimated standard deviation of background noise: 1.86 mJy
# Initial peak: 25.31 mJy at 1631,985
# Next major iteration at: 2.53 mJy
# Performed 153 iterations in total, 88 in this major iteration with sub-minor optimization.
# Stopped on peak 2.48 mJy, because the minor-loop threshold was reached. Continuing cleaning after inversion/prediction round.
#  == Converting model image to visibilities ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...

#  == Converting model image to visibilities ==
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Converting model image to visibilities ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Constructing image ==

# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30138e+06
#  == Constructing image ==
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 770602
#  == Constructing image ==

# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...

# Gridded visibility count: 1.30246e+06
#  == Cleaning (4) ==

# Estimated standard deviation of background noise: 1.88 mJy
# Initial peak: 2.19 mJy at 917,896
# Major iteration threshold reached global threshold of 940.09 µJy: final major iteration.
# Performed 185 iterations in total, 32 in this major iteration with sub-minor optimization.
# Stopped on peak 912.85 µJy, because the threshold was reached.
# Writing model image...
#  == Converting model image to visibilities ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Converting model image to visibilities ==
# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...

#  == Converting model image to visibilities ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Fourier transforms for pass 0... Predicting...
#  == Constructing image ==
# Opening reordered part 0 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.08''
# Minimal inversion size: 334 x 334, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30138e+06
#  == Constructing image ==

# Opening reordered part 1 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 28.01''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 770602
#  == Constructing image ==
# Opening reordered part 2 spw 0 for /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms
# Theoretic beam = 27.95''
# Minimal inversion size: 335 x 335, using optimal: 336 x 336
# The theoretically suggested number of w-layers (4) is less than the number of availables
# cores (24). Changing suggested number of w-layers to 24.
# Will process 24/24 w-layers per pass.
# Gridding pass 0... Fourier transforms...
# Gridded visibility count: 1.30246e+06
# 4 major iterations were performed.
# Rendering sources to restored image (beam=35.15''-38.44'', PA=2.85 deg)... DONE
# Writing restored image... DONE
# Rendering sources to restored image (beam=35.82''-39.44'', PA=10.99 deg)... DONE
# Writing restored image... DONE

# Rendering sources to restored image (beam=34.99''-38.07'', PA=2.72 deg)... DONE
# Writing restored image... DONE
# Image buf alloc stats:
#          max alloc'd images = 41 real + 72 complex
#        max allocated chunks = 14
#       current allocated mem = 0.1 GB
# Inversion: 00:00:17.174630, prediction: 00:00:12.297520, deconvolution: 00:00:12.537150
# Fitting beam... major=38.49'', minor=35.24'', PA=4.69 deg, theoretical=27.95''.
# Writing /stimela_mount/output/continuum/image_1/mypipelinerun_circinus_p3_1-MFS-psf.fits...
# Writing /stimela_mount/output/continuum/image_1/mypipelinerun_circinus_p3_1-MFS-dirty.fits...
# Writing /stimela_mount/output/continuum/image_1/mypipelinerun_circinus_p3_1-MFS-residual.fits...
# Writing /stimela_mount/output/continuum/image_1/mypipelinerun_circinus_p3_1-MFS-model.fits...
# Rendering sources to restored image (beam=35.24''-38.49'', PA=4.69 deg)... DONE

# Writing /stimela_mount/output/continuum/image_1/mypipelinerun_circinus_p3_1-MFS-image.fits...
# Adding model data column... DONE
# Writing changed model back to /stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms:
#  0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
# Cleaning up temporary files...
2020-05-10 05:55:31 CARACal.Stimela.image-field0-iter1 INFO: job complete at 2020-05-10 05:55:31.551731 after 0:00:51.123419
2020-05-10 05:55:31 CARACal.Stimela.calibrate-cubical-field0-iter1 INFO: job started at 2020-05-10 05:55:31.552577

# /usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:935: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
#   warn("Attempting to work in a virtualenv. If you encounter problems, please "
# Traceback (most recent call last):
#   File "/usr/local/bin/gocubical", line 5, in <module>
#     from cubical.main import main
#   File "/usr/local/lib/python3.6/dist-packages/cubical/main.py", line 95, in <module>
#     from cubical.data_handler.ms_data_handler import MSDataHandler
#   File "/usr/local/lib/python3.6/dist-packages/cubical/data_handler/ms_data_handler.py", line 24, in <module>
#     from cubical.data_handler.ms_tile import RowChunk, MSTile
#   File "/usr/local/lib/python3.6/dist-packages/cubical/data_handler/ms_tile.py", line 22, in <module>
#     from cubical.degridder.DicoSourceProvider import DicoSourceProvider
#   File "/usr/local/lib/python3.6/dist-packages/cubical/degridder/DicoSourceProvider.py", line 3, in <module>
#     from DDFacet.Imager import ModModelMachine as MMM
#   File "/usr/local/lib/python3.6/dist-packages/DDFacet/__init__.py", line 23, in <module>
#     __version__ = pkg_resources.require("DDFacet")[0].version
#   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/43/test-output/c19030fdf5a781924ebec1a5cbabe490cf345edc/caracal_venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 901, in require
#     needed = self.resolve(parse_requirements(requirements))
#   File "/var/lib/jenkins/jobs/PR_meerkathi/workspace/43/test-output/c19030fdf5a781924ebec1a5cbabe490cf345edc/caracal_venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 792, in resolve
#     raise VersionConflict(dist, req).with_context(dependent_req)
# pkg_resources.ContextualVersionConflict: (scipy 1.4.1 (/var/lib/jenkins/jobs/PR_meerkathi/workspace/43/test-output/c19030fdf5a781924ebec1a5cbabe490cf345edc/caracal_venv/lib/python3.6/site-packages), Requirement.parse('scipy<=1.3.3'), {'DDFacet'})
# 
# If you suspect this is an IPython 7.12.0 bug, please report it at:
#     https://github.com/ipython/ipython/issues
# or send an email to the mailing list at ipython-dev@python.org
# 
# You can print a more detailed traceback right now with "%tb", or use "%debug"
# to interactively debug it.
# 
# Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
#     %config Application.verbose_crash=True
# 
# Traceback (most recent call last):
#   File "/stimela_mount/code/run.py", line 57, in <module>
#     subprocess.check_call(shlex.split(_runc))
#   File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
#     raise CalledProcessError(retcode, cmd)
# subprocess.CalledProcessError: Command '['gocubical', '--sol-jones', 'G', '--data-ms', '/stimela_mount/msdir/1532022061_subset-circinus_p3_corr.ms', '--data-column', 'DATA', '--data-time-chunk', '240', '--data-freq-chunk', '0', '--sel-ddid', '0', '--model-list', 'MODEL_DATA', '--model-ddes', 'never', '--montblanc-dtype', 'float', '--weight-column', 'WEIGHT', '--flags-apply', '-cubical', '--madmax-enable', '1', '--madmax-estimate', 'corr', '--madmax-plot', '1', '--madmax-threshold', '0,10', '--sel-diag', '0', '--sol-term-iters', '50', '--bbc-save-to', '/stimela_mount/output/continuum/selfcal_products/bbc-gains-1-1532022061_subset-circinus_p3_corr.parmdb', '--dist-ncpu', '5', '--dist-max-chunks', '4', '--out-name', '/stimela_mount/output/continuum/selfcal_products/mypipelinerun_1532022061_subset-circinus_p3_corr_1_cubical', '--out-overwrite', '1', '--out-mode', 'sc', '--out-casa-gaintables', '1', '--out-plots', '1', '--dist-nworker', '0', '--log-memory', '1', '--log-boring', '1', '--g-time-int', '120', '--g-freq-int', '0', '--g-clip-low', '0.5', '--g-clip-high', '2.0', '--g-solvable', '1', '--g-type', 'phase-diag', '--g-save-to', '/stimela_mount/output/continuum/selfcal_products/mypipelinerun-g-phase-gains-1-1532022061_subset-circinus_p3_corr.parmdb', '--g-update-type', 'phase-diag', '--g-max-prior-error', '0.3', '--g-max-post-error', '0.3', '--degridding-OverS', '11', '--degridding-Support', '7', '--degridding-Nw', '100', '--degridding-wmax', '0', '--degridding-Padding', '1.7', '--degridding-NDegridBand', '16', '--degridding-MaxFacetSize', '0.25', '--degridding-MinNFacetPerAxis', '1']' returned non-zero exit status 1.
2020-05-10 05:55:36 CARACal.Stimela.calibrate-cubical-field0-iter1 ERROR: cd /var/lib/jenkins/jobs/PR_meerkathi/workspace/43/test-output/c19030fdf5a781924ebec1a5cbabe490cf345edc/minitestConfig-singularity/.stimela_workdir-1589090071253759 && singularity returns error code 1
2020-05-10 05:55:36 CARACal.Stimela.calibrate-cubical-field0-iter1 ERROR: job failed at 2020-05-10 05:55:36.201084 after 0:00:04.648507
2020-05-10 05:55:36 CARACal ERROR: Job 'calibrate-cubical-field0-iter1:: Calibrate step 1 ms=1532022061_subset-circinus_p3_corr.ms' failed: cd /var/lib/jenkins/jobs/PR_meerkathi/workspace/43/test-output/c19030fdf5a781924ebec1a5cbabe490cf345edc/minitestConfig-singularity/.stimela_workdir-1589090071253759 && singularity returns error code 1 [PipelineException]
2020-05-10 05:55:36 CARACal INFO:   More information can be found in the logfile at output/logs-20200510-054611/log-caracal.txt
2020-05-10 05:55:36 CARACal INFO: exiting with error code 1
caracal-pipeline / caracal

Jenkins Singularity run fails #1067
