search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

Cross_cal stimela_casa_3.0.0.img doesn't exist: No such file or directory #596

Closed ianjarog closed 4 years ago

ianjarog commented 5 years ago

After upgrading to python 3 on singularity and using Cassie branch, I got the following errors:

ia/projects/rhodes-ratt/IANJA/N4945PY3/stimela_parameter_files/set_model_cal_0-14061168518289615683748643492968.json:/scratch/configfile:ro --bind /idia/projects/rhodes-ratt/IANJA/meerkathi-venvpy3/lib/python3.7/site-packages/stimela/cargo/cab/casa_setjy/src:/scratch/code:ro --bind /idia/projects/rhodes-ratt/IANJA/meerkathi-venvpy3/lib/python3.7/site-packages/stimela/cargo/cab/singularity_run:/singularity:rw --bind /idia/projects/rhodes-ratt/IANJA/N4945PY3/msdir:/scratch/msdir:rw --bind /idia/projects/rhodes-ratt/IANJA/N4945PY3/input:/scratch/input:ro --bind /idia/projects/rhodes-ratt/IANJA/N4945PY3/output/logs/log-set_model_cal_0.txt:/scratch/logfile:rw --bind /idia/projects/rhodes-ratt/IANJA/N4945PY3/output:/scratch/output:rw -c /idia/projects/rhodes-ratt/IANJA/meerkathipy3/singularityimages/stimela_casa_3.0.0.img 48a meerkathi - 2019-09-13 13:59:50,462 INFO - ^[[91mERROR : Image path /idia/projects/rhodes-ratt/IANJA/meerkathipy3/singularityimages/stimela_casa_3.0.0.img doesn't exist: No such file or directory meerkathi - 2019-09-13 13:59:50,462 INFO - ^[[0m^[[31mABORT : Retval = 255 meerkathi - 2019-09-13 13:59:50,462 INFO - ^[[0m meerkathi - 2019-09-13 13:59:50,462 CRITICAL - INFO:STIMELA-5:Recipe execution failed while running job set_model_cal_0 meerkathi - 2019-09-13 13:59:50,463 CRITICAL - INFO:STIMELA-5:Completed jobs : ['flagset_clear_cross_cal_0', 'clear_cal_0'] meerkathi - 2019-09-13 13:59:50,463 CRITICAL - INFO:STIMELA-5:Remaining jobs : ['delay_cal_0', 'flag_delay_cal_0', 'plot_delay_cal_0', 'bp_cal_0', 'plot_bandpass_0', 'gain_cal_flux_0', 'plot_gain_cal_flux_0', 'gain_cal_gain_0', 'plot_gain_cal_0', 'transfer_fluxscale_0', 'plot_fluxscale_0', 'apply_bpcal_0', 'apply_gcal_0', 'apply_target_0', 'flagset_update_cross_cal_0', 'flagging_summary_crosscal_0'] meerkathi - 2019-09-13 13:59:50,463 CRITICAL - INFO:STIMELA-5:Logging remaining task: delay_cal_0:: Delay calibration ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,463 CRITICAL - INFO:STIMELA-5:Logging remaining task: flag_delay_cal_0:: Flagging gains meerkathi - 2019-09-13 13:59:50,464 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_delay_cal_0:: Plot gaincal phase ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,464 CRITICAL - INFO:STIMELA-5:Logging remaining task: bp_cal_0:: Bandpass calibration ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,464 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_bandpass_0:: Plot bandpass calibration gain caltable=mypipelinerun-full-pol-selected-chunk-1gc1.B0 meerkathi - 2019-09-13 13:59:50,464 CRITICAL - INFO:STIMELA-5:Logging remaining task: gain_cal_flux_0:: Gain calibration fer bandpass ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,464 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_gain_cal_flux_0:: Plot gaincal phase ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,464 CRITICAL - INFO:STIMELA-5:Logging remaining task: gain_cal_gain_0:: Gain calibration ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,465 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_gain_cal_0:: Plot gaincal phase ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,465 CRITICAL - INFO:STIMELA-5:Logging remaining task: transfer_fluxscale_0:: Flux scale transfer ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,465 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_fluxscale_0:: Plot gaincal phase ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,466 CRITICAL - INFO:STIMELA-5:Logging remaining task: apply_bpcal_0:: Apply calibration to field=PKS1934-63, ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,466 CRITICAL - INFO:STIMELA-5:Logging remaining task: apply_gcal_0:: Apply calibration to field=J1318-4620, ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,466 CRITICAL - INFO:STIMELA-5:Logging remaining task: apply_target_0:: Apply calibration to field=NGC4945, ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,466 CRITICAL - INFO:STIMELA-5:Logging remaining task: flagset_update_cross_cal_0 meerkathi - 2019-09-13 13:59:50,466 CRITICAL - INFO:STIMELA-5:Logging remaining task: flagging_summary_crosscal_0:: Flagging summary ms=full-pol-selected-chunk.ms meerkathi - 2019-09-13 13:59:50,466 CRITICAL - INFO:STIMELA-5:Saving pipeline information in .last_cross_calibration.json meerkathi - 2019-09-13 13:59:50,468 ERROR - An unhandled exeption occured. If you think this is a bug please report it. meerkathi - 2019-09-13 13:59:50,468 ERROR - Your logfile is here: /idia/projects/rhodes-ratt/IANJA/N4945PY3/log-meerkathi.txt. meerkathi - 2019-09-13 13:59:50,468 ERROR - You are running version: 0.2.0 meerkathi - 2019-09-13 13:59:50,606 ERROR - Traceback (most recent call last): File "/idia/projects/rhodes-ratt/IANJA/meerkathi-venvpy3/lib/python3.7/site-packages/stimela/recipe.py", line 1025, in run run_job() File "/idia/projects/rhodes-ratt/IANJA/meerkathi-venvpy3/lib/python3.7/site-packages/stimela/recipe.py", line 136, in run_singularity_job self.job.start() File "/idia/projects/rhodes-ratt/IANJA/meerkathi-venvpy3/lib/python3.7/site-packages/stimela/singularity.py", line 86, in start self.image, self.name]) File "/idia/projects/rhodes-ratt/IANJA/meerkathi-venvpy3/lib/python3.7/site-packages/stimela/utils/init.py", line 124, in xrun '%s: returns errr code %d' % (command, process.returncode)) stimela.utils.StimelaCabRuntimeError: singularity instance.start: returns errr code 255

bennahugo commented 5 years ago

I'm relying on an old version of stimela with the correct version of CASA containerized. You will need to manually pull the correct image from dockerhub if you want to use singularity

bennahugo commented 5 years ago

Follow the API instructions here: https://sylabs.io/guides/3.3/user-guide/cli/singularity_pull.html

KshitijT commented 5 years ago

Thanks @bennahugo . The command, I think, is:

singularity pull docker://stimela_casa_3.0.0.img

If run, you'll get a new file in your singularity folder: casa-0.3.0.simg

Rename it to: stimela_casa_3.0.0.img

and rerun.

Let us know if it works, @ianjarog . Thanks !

KshitijT commented 5 years ago

Tagging @bnamumba here, since you are getting the same error?

ianjarog commented 5 years ago

@KshitijT @bennahugo @bnamumba by @KshitijT worked but after re-run another error popped up ImportError: No module named Crasa.Crasa as below

meerkathi - 2019-09-13 18:08:29,759 CRITICAL - INFO:STIMELA-5:Starting container [fae]. Timeout set to -1. The container ID is printed below. meerkathi - 2019-09-13 18:08:29,759 INFO - Running: singularity run instance://fae meerkathi - 2019-09-13 18:08:32,769 INFO - /.singularity.d/runscript: line 8: /etc/init.d/xvfb: No such file or directory meerkathi - 2019-09-13 18:08:32,769 INFO - Virtual frame buffer not installed. You may not be able to plot with this cab meerkathi - 2019-09-13 18:08:32,769 INFO - Traceback (most recent call last): meerkathi - 2019-09-13 18:08:32,770 INFO - File "/scratch/code/run.py", line 4, in meerkathi - 2019-09-13 18:08:32,770 INFO - import Crasa.Crasa as crasa meerkathi - 2019-09-13 18:08:32,770 INFO - ImportError: No module named Crasa.Crasa meerkathi - 2019-09-13 18:08:32,770 INFO - /.singularity.d/runscript: line 11: /etc/init.d/xvfb: No such file or directory meerkathi - 2019-09-13 18:08:32,770 CRITICAL - INFO:STIMELA-5:Recipe execution failed while running job set_model_cal_0 meerkathi - 2019-09-13 18:08:32,770 CRITICAL - INFO:STIMELA-5:Completed jobs : ['flagset_clear_cross_cal_0', 'clear_cal_0']

bennahugo commented 5 years ago

@ianjarog are you sure the casa47_setjy cab is being invoked? The line quoted in the exception is not the line here: https://github.com/SpheMakh/Stimela/blob/master/stimela/cargo/cab/casa47_setjy/src/run.py#L4

bennahugo commented 5 years ago

please check that your stimela HEAD is 8d47698fb92e3cc367fe140ff8a40625f3ecb47e

ianjarog commented 5 years ago

@bennahugo I think CAB = casa_setjy was invoked

bennahugo commented 5 years ago

ok try casa47_setjy

edeblok commented 5 years ago

Running in the same issue here: fresh install of Cassie branch, using pip3 etc, and running the singularity version. Trying to run crosscal I've explicitly defined casa_version: '47' in the config file (but that shouldn't be necessary), and the correct images are added to the recipe (i.e. stimela_casa_0.3.0). Example:

meerkathi - 2019-09-16 15:33:43,647 CRITICAL - INFO:STIMELA-5:Adding cab '/users/blok/SINGULARITY_IMAGES//stimela_casa_0.3.0.img' to recipe. The container will be named 'delay_cal_0'

But when this is executed it still seems to execute the version from the wrong CASA and crashes . "casa_version: '47'" seems to have no effect. (I've tried hardcoding it in the crosscal worker, but to no effect either). How do we get the 47 versions to run?

... meerkathi - 2019-09-16 16:14:38,750 CRITICAL - INFO:STIMELA-5:STEP 3 :: delay_cal_0:: Delay calibration ms=e302_400.ms meerkathi - 2019-09-16 16:14:38,800 CRITICAL - INFO:root:Validating parameters... CAB = casa_gaincal meerkathi - 2019-09-16 16:14:38,861 CRITICAL - INFO:root:Parameters validated and saved. Parameter file is: stimela_parameter_files/delay_cal_0-1400329267501601568640823618732.json ... meerkathi - 2019-09-16 16:14:42,902 INFO - /.singularity.d/runscript: line 8: /etc/init.d/xvfb: No such file or directory meerkathi - 2019-09-16 16:14:42,903 INFO - Virtual frame buffer not installed. You may not be able to plot with this cab meerkathi - 2019-09-16 16:14:42,903 INFO - Traceback (most recent call last): meerkathi - 2019-09-16 16:14:42,903 INFO - File "/scratch/code/run.py", line 4, in meerkathi - 2019-09-16 16:14:42,903 INFO - import Crasa.Crasa as crasa meerkathi - 2019-09-16 16:14:42,904 INFO - ImportError: No module named Crasa.Crasa meerkathi - 2019-09-16 16:14:42,904 INFO - /.singularity.d/runscript: line 11: /etc/init.d/xvfb: No such file or directory

KshitijT commented 5 years ago

@edeblok , could you please check if the casa_47 cabs are present in your virtual environment by doing:

stimela cabs -i casa47_gaincal

?

bennahugo commented 5 years ago

That would indicate the 4.7 cabs are not being called, because there is no such line in the run.py file of casa47_setjy. In my calibration pipeline I've tested that the cabs work (https://github.com/ska-sa/fleeting-pol/blob/master/fleetingpol/calscripts/crosscal.py#L492). This would indicate a problem with CASA 5

edeblok commented 5 years ago

(meerkathi-venv) blok@helo:~$ stimela cabs -i casa47_gaincal Cab casa_gaincal Info Specify Calibration Values of Various Types Base Image stimela/casa:0.3.0

Parameters: Name msname/vis Description Name of input visibility file Type file Default None

Name caltable Description Name of output gain calibration table Type file Default None

Name field Description Select field using field id(s) or field name(s) Type str Default None

Name spw Description Select spectral window/channels Type str Default None

Name selectdata Description Other data selection parameters Type bool Default True

Name timerange Description Select data based on time range Type str Default None

Name uvrange Description Select data within uvrange (default units meters) Type str Default None

Name antenna Description Select data based on antenna/baseline Type str Default None

Name scan Description Scan number range Type str Default None

Name observation Description Select by observation ID(s) Type str Default None

Name msselect Description Optional complex data selection (ignore for now) Type str Default None

Name solint Description Solution interval: egs. 'inf', '60s' (see help) Type str/float/int Default inf

Name combine Description Data axes which to combine for solve (obs, scan, spw, and/or, field) Type str Default None

Name preavg Description Pre-averaging interval (sec) (rarely needed) Type float Default -1.0

Name refant Description Reference antenna name(s) Type str Default None

Name minblperant Description Minimum baselines per antenna required for solve Type int Default 4

Name minsnr Description Reject solutions below this SNR Type float Default 3.0

Name solnorm Description Normalize average solution amplitudes to 1.0 (G, T only) Type bool Default False

Name gaintype Description Type of gain solution (G,T,GSPLINE,K,KCROSS) Type str Default G Choices ['G', 'T', 'GSPLINE', 'K', 'KCROSS']

Name splinetime Description Spline timescale(sec); All spw's are first averaged. Type float Default 3600.0

Name npointaver Description The phase-unwrapping algorithm Type int Default 3

Name phasewrap Description Wrap the phase for jumps greater than this value (degrees) Type float Default 180.0

Name smodel Description Point source Stokes parameters for source model. Type list:str Default None

Name calmode Description Type of solution: ('ap', 'p', 'a') Type str Default ap Choices ['ap', 'p', 'a']

Name append Description Append solutions to the (existing) table Type bool Default False

Name docallib Description Use callib or traditional cal apply parameters Type bool Default False

Name callib Description Cal Library filename Type file Default None

Name gaintable Description Gain calibration table(s) to apply on the fly Type list:file Default None

Name gainfield Description Select a subset of calibrators from gaintable(s) Type list:str/list:int Default None

Name interp Description Temporal interpolation for each gaintable (=linear) Type list Default None

Name spwmap Description Spectral windows combinations to form for gaintables(s) Type list:list Default None

Name parang Description Apply parallactic angle correction on the fly Type bool Default False

(meerkathi-venv) blok@helo:~$ stimela cabs -i casa47_gaincal Cab casa_gaincal Info Specify Calibration Values of Various Types Base Image stimela/casa:0.3.0

Parameters: Name msname/vis Description Name of input visibility file Type file Default None

Name caltable Description Name of output gain calibration table Type file Default None

@edeblok , could you please check if the casa_47 cabs are present in your virtual environment by doing:

stimela cabs -i casa47_gaincal

?

KshitijT commented 5 years ago

Is this tested in singularity, @bennahugo ? I have of course tested it in docker, before making the PR, but most of the people seem to have trouble with using it in singularity.

bennahugo commented 5 years ago

No this is tested in docker. But there is no such line in the run.py file so the meerkathi invocation must be wrong

HaoChen-ast commented 5 years ago

The current master branch does not show this error in cross_cal worker. Python3 and singularity are used. I use the master branch to finish cross_cal and Cassie to do the rest now.

KshitijT commented 5 years ago

The current master branch does not show this error in cross_cal worker. Python3 and singularity are used. I use the master branch to finish cross_cal and Cassie to do the rest now.

Yes, it should not, the CASA 47 cabs are only in Cassie branch. If this works for you please do try this out: In your yml file, after these first two lines :

- cross_cal:
    enable: true
    casa_version: " "

This should take you back to the 'default' behaviour - i.e. that in master, so that there is no need to switch back and forth between branches. Beware of issue #573 though, if you see lot of your data getting flagged with this, you need to use CASA 4.7/docker.

KshitijT commented 5 years ago

I think this is a Stimela/singularity issue, which I am singularly (no excuse for the pun) unqualified to sort out on my own - this works fine with docker. With this in mind I would be extremely glad if @SpheMakh looked at this.

KshitijT commented 5 years ago

In return, I offer my aid in keeping you away from KFC in future, Sphe.

KshitijT commented 4 years ago

@SpheMakh , is this fixed now? I see a casa 47 PR merge on stimela, that was for fixing this issue?

paoloserra commented 4 years ago

@SpheMakh @KshitijT any update on this? We have more users stuck here. Thanks!

KshitijT commented 4 years ago

@SpheMakh mentioned that this is fixed in stimela now, updating stimela should sort this out. Please give another hour to confirm.

SpheMakh commented 4 years ago

I think it's fixed. Could you try again with the master branch of stimela (please re-pull the images before trying again)

KshitijT commented 4 years ago

So I tried this:

  1. Update stimela to the latest master version.
  2. Re-pull singularity images
  3. Re-install the latest version of meerkathi
  4. Run it in default mode on test dataset

It still runs into trouble when using the casa 47 cabs (a different error than the one reported above):

virtual fram  buffer not installed. you may not be able to plot with this cab
meerkathi - 2019-11-06 16:28:16,726 INFO -
meerkathi - 2019-11-06 16:28:16,727 INFO - =========================================
meerkathi - 2019-11-06 16:28:16,727 INFO - The start-up time of CASA may vary
meerkathi - 2019-11-06 16:28:16,727 INFO - depending on whether the shared libraries
meerkathi - 2019-11-06 16:28:16,727 INFO - are cached or not.
meerkathi - 2019-11-06 16:28:16,727 INFO - =========================================
meerkathi - 2019-11-06 16:28:17,729 INFO - setgpid( ) failed: Operation not permitted
meerkathi - 2019-11-06 16:28:17,729 INFO -                    processes may be left dangling...
meerkathi - 2019-11-06 16:28:37,859 INFO - CASA Version 4.7.0-REL (r38335)
meerkathi - 2019-11-06 16:28:37,919 INFO -   Compiled on: Wed 2016/09/28 11:50:32 UTC
meerkathi - 2019-11-06 16:28:41,923 INFO - Traceback (most recent call last):
meerkathi - 2019-11-06 16:28:41,924 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/casapy.py", line 475, in <module>
meerkathi - 2019-11-06 16:28:41,924 INFO -     from taskinit import *
meerkathi - 2019-11-06 16:28:41,924 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/taskinit.py", line 1, in <module>
meerkathi - 2019-11-06 16:28:41,924 INFO -     import pCASA
meerkathi - 2019-11-06 16:28:41,924 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/pCASA.py", line 51, in <module>
meerkathi - 2019-11-06 16:28:41,924 INFO -     import parallel_go
meerkathi - 2019-11-06 16:28:41,925 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/parallel_go.py", line 2, in <module>
meerkathi - 2019-11-06 16:28:41,925 INFO -     if not MPIEnvironment.is_mpi_enabled: from IPython.kernel import client
meerkathi - 2019-11-06 16:28:41,925 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/site-packages/IPython/kernel/client.py", line 38, in <module>
meerkathi - 2019-11-06 16:28:41,925 INFO -     from IPython.kernel.clientconnector import ClientConnector
meerkathi - 2019-11-06 16:28:41,925 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/site-packages/IPython/kernel/clientconnector.py", line 22, in <module>
meerkathi - 2019-11-06 16:28:41,925 INFO -     from IPython.kernel.config import config_manager as kernel_config_manager
meerkathi - 2019-11-06 16:28:41,925 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/site-packages/IPython/kernel/config/__init__.py", line 28, in <module>
meerkathi - 2019-11-06 16:28:41,925 INFO -     security_dir = get_security_dir()
meerkathi - 2019-11-06 16:28:41,926 INFO -   File "/casa-release-4.7.0-el6/lib/python2.7/site-packages/IPython/genutils.py", line 1015, in get_security_dir
meerkathi - 2019-11-06 16:28:41,926 INFO -     os.mkdir(security_dir, 0700)
meerkathi - 2019-11-06 16:28:41,926 INFO - OSError: [Errno 13] Permission denied: '/root/.casa/ipython/security'
meerkathi - 2019-11-06 16:28:50,167 INFO - Reloaded configuration
meerkathi - 2019-11-06 16:28:50,202 INFO - Traceback (most recent call last):
meerkathi - 2019-11-06 16:28:50,202 INFO -   File "/scratch/code/run.py", line 5, in <module>
meerkathi - 2019-11-06 16:28:50,202 INFO -     casa = drivecasa.Casapy(log2term=True, echo_to_stdout=True, timeout=24*3600*10)
meerkathi - 2019-11-06 16:28:50,202 INFO -   File "/usr/local/lib/python2.7/dist-packages/drivecasa/interface.py", line 146, in __init__
meerkathi - 2019-11-06 16:28:50,202 INFO -     self.child.expect(self.prompt, timeout=60)
meerkathi - 2019-11-06 16:28:50,202 INFO -   File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 321, in expect
meerkathi - 2019-11-06 16:28:50,202 INFO -     timeout, searchwindowsize, async)
meerkathi - 2019-11-06 16:28:50,202 INFO -   File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 345, in expect_list
meerkathi - 2019-11-06 16:28:50,202 INFO -     return exp.expect_loop(timeout)
meerkathi - 2019-11-06 16:28:50,202 INFO -   File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 105, in expect_loop
meerkathi - 2019-11-06 16:28:51,363 INFO -     return self.eof(e)
meerkathi - 2019-11-06 16:28:51,363 INFO -   File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 50, in eof
meerkathi - 2019-11-06 16:28:51,363 INFO -     raise EOF(msg)
meerkathi - 2019-11-06 16:28:51,363 INFO - pexpect.exceptions.EOF: End Of File (EOF). Braindead platform.
meerkathi - 2019-11-06 16:28:51,363 INFO - <pexpect.pty_spawn.spawn object at 0x7f3b049ca650>
meerkathi - 2019-11-06 16:28:51,363 INFO - command: /casa-release-4.7.0-el6/bin/casa
meerkathi - 2019-11-06 16:28:51,363 INFO - args: ['/casa-release-4.7.0-el6/bin/casa', '--nologger', '--nogui', '--colors=NoColor', '--log2term']
meerkathi - 2019-11-06 16:28:51,363 INFO - buffer (last 100 chars): ''
meerkathi - 2019-11-06 16:28:51,363 INFO - before (last 100 chars): "00)\r\nOSError: [Errno 13] Permission denied: '/root/.casa/ipython/security'\r\nReloaded configuration\r\n"
meerkathi - 2019-11-06 16:28:51,363 INFO - after: <class 'pexpect.exceptions.EOF'>
meerkathi - 2019-11-06 16:28:51,364 INFO - match: None
meerkathi - 2019-11-06 16:28:51,364 INFO - match_index: None
meerkathi - 2019-11-06 16:28:51,364 INFO - exitstatus: 1
meerkathi - 2019-11-06 16:28:51,364 INFO - flag_eof: True
meerkathi - 2019-11-06 16:28:51,364 INFO - pid: 6
meerkathi - 2019-11-06 16:28:51,364 INFO - child_fd: 5
meerkathi - 2019-11-06 16:28:51,364 INFO - closed: False
meerkathi - 2019-11-06 16:28:51,364 INFO - timeout: 864000
meerkathi - 2019-11-06 16:28:51,364 INFO - delimiter: <class 'pexpect.exceptions.EOF'>
meerkathi - 2019-11-06 16:28:51,364 INFO - logfile: None
meerkathi - 2019-11-06 16:28:51,364 INFO - logfile_read: <open file '<stdout>', mode 'w' at 0x7f3b06d99150>
meerkathi - 2019-11-06 16:28:51,364 INFO - logfile_send: None
meerkathi - 2019-11-06 16:28:51,364 INFO - maxread: 2000
meerkathi - 2019-11-06 16:28:51,364 INFO - ignorecase: False
meerkathi - 2019-11-06 16:28:51,364 INFO - searchwindowsize: None
meerkathi - 2019-11-06 16:28:51,364 INFO - delaybeforesend: 0.05
meerkathi - 2019-11-06 16:28:51,364 INFO - delayafterclose: 0.1
meerkathi - 2019-11-06 16:28:51,364 INFO - delayafterterminate: 0.1
meerkathi - 2019-11-06 16:28:51,365 INFO - searcher: searcher_re:
meerkathi - 2019-11-06 16:28:51,365 INFO -     0: re.compile("CASA <[0-9]+>:")
meerkathi - 2019-11-06 16:28:51,614 CRITICAL - INFO:STIMELA-5:Recipe execution failed while running job set_model_cal_0
meerkathi - 2019-11-06 16:28:51,663 CRITICAL - INFO:STIMELA-5:Completed jobs : ['flagset_clear_cross_cal_0', 'clear_cal_0']
meerkathi - 2019-11-06 16:28:51,663 CRITICAL - INFO:STIMELA-5:Remaining jobs : ['delay_cal_0', 'flag_delay_cal_0', 'plot_delay_cal_0', 'bp_cal_0', 'plot_bandpass_0', 'gain_cal_flux_0', 'plot_gain_cal_flux_0', 'gain_cal_gain_0', 'plot_gain_cal_0', 'transfer_fluxscale_0', 'plot_fluxscale_0', 'applyto_bpcal_0', 'applyto_gcal_0', 'applyto_target_0', 'flagset_update_cross_cal_0', 'flagging_summary_crosscal_0']
meerkathi - 2019-11-06 16:28:51,663 CRITICAL - INFO:STIMELA-5:Logging remaining task: delay_cal_0:: Delay calibration ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,663 CRITICAL - INFO:STIMELA-5:Logging remaining task: flag_delay_cal_0:: Flagging gains
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_delay_cal_0:: Plot gaincal phase ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: bp_cal_0:: Bandpass calibration ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_bandpass_0:: Plot bandpass calibration gain caltable=mypipelinerun-1477074305.subset-1gc1.B0
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: gain_cal_flux_0:: Gain calibration fer bandpass ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_gain_cal_flux_0:: Plot gaincal phase ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: gain_cal_gain_0:: Gain calibration ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_gain_cal_0:: Plot gaincal phase ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,664 CRITICAL - INFO:STIMELA-5:Logging remaining task: transfer_fluxscale_0:: Flux scale transfer ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Logging remaining task: plot_fluxscale_0:: Plot gaincal phase ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Logging remaining task: applyto_bpcal_0:: Apply calibration to field=PKS1934-638, ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Logging remaining task: applyto_gcal_0:: Apply calibration to field=ATCA2259-375, ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Logging remaining task: applyto_target_0:: Apply calibration to field=IC5264, ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Logging remaining task: flagset_update_cross_cal_0
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Logging remaining task: flagging_summary_crosscal_0:: Flagging summary  ms=1477074305.subset.ms
meerkathi - 2019-11-06 16:28:51,665 CRITICAL - INFO:STIMELA-5:Saving pipeline information in .last_cross_calibration.json
meerkathi - 2019-11-06 16:28:51,814 CRITICAL - INFO:STIMELA-5:Stopping container [aae]. The container ID is printed below.
meerkathi - 2019-11-06 16:28:51,815 INFO - Running: singularity instance.stop aae
Stopping aae instance of /home/kshitij/STIMELA_IMAGES_NEW/stimela_casa_0.3.0.img (PID=57126)
meerkathi - 2019-11-06 16:28:56,267 ERROR - An unhandled exeption occured. If you think this is a bug please report it.
meerkathi - 2019-11-06 16:28:56,348 ERROR - Your logfile is here: /home/kshitij/svw_tests/log-meerkathi.txt.
meerkathi - 2019-11-06 16:28:56,348 ERROR - You are running version: 0.1.0-1141-ga9e0a25
meerkathi - 2019-11-06 16:28:57,286 ERROR - Traceback (most recent call last):
  File "/home/kshitij/Stimela/stimela/recipe.py", line 1025, in run
    run_job()
  File "/home/kshitij/Stimela/stimela/recipe.py", line 138, in run_singularity_job
    self.job.run()
  File "/home/kshitij/Stimela/stimela/singularity.py", line 106, in run
    timeout=self.time_out, kill_callback=self.stop)
  File "/home/kshitij/Stimela/stimela/utils/__init__.py", line 144, in xrun
    '%s: returns errr code %d' % (command, process.returncode))
stimela.utils.StimelaCabRuntimeError: singularity run: returns errr code 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kshitij/meerkathi/meerkathi/main.py", line 165, in __run
    pipeline.run_workers()
  File "/home/kshitij/meerkathi/meerkathi/workers/worker_administrator.py", line 253, in run_workers
    recipe.run()
  File "/home/kshitij/Stimela/stimela/recipe.py", line 1054, in run
    raise_(pe, None, sys.exc_info()[2])
  File "/home/kshitij/p3-venv/lib/python3.6/site-packages/future/utils/__init__.py", line 440, in raise_
    raise exc.with_traceback(tb)
  File "/home/kshitij/Stimela/stimela/recipe.py", line 1025, in run
    run_job()

Runs fine when the casa_version parameter is not set to 47. @SpheMakh , could you please test it?

SpheMakh commented 4 years ago

Please do not use the CASA 4.7 images, they were made before singularity. The CASA 5.6 images work perfectly fine

SpheMakh commented 4 years ago

@KshitijT did you find this phantom gaincal issue with your tests?

KshitijT commented 4 years ago

@KshitijT did you find this phantom gaincal issue with your tests?

No, I haven't, yet. If casa 5.6 images work ok then I guess we don't need casa 4.7 at all. But just to clarify, we can't use casa_47 cabs with singularity still?

ratt-priv-ci commented 4 years ago

Same issue with overflagging in casa 5.6. Please don't remove the casa4.7 cabs from stimela. I'm using them for my pipelines

On Wed, Nov 6, 2019 at 9:02 PM Kshitij Thorat notifications@github.com wrote:

@KshitijT https://github.com/KshitijT did you find this phantom gaincal issue with your tests?

No, I haven't, yet. If casa 5.6 images work ok then I guess we don't need casa 4.7 at all. But just to clarify, we can't use casa_47 cabs with singularity still?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ska-sa/meerkathi/issues/596?email_source=notifications&email_token=AEIVPJWCW3GGSKEDOU6HFBTQSMIEJA5CNFSM4IWQLCG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDHUD5Q#issuecomment-550453750, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEIVPJRCGEK2H2ASCCMAFA3QSMIEJANCNFSM4IWQLCGQ .

--

Benjamin Hugo

Junior Software Developer SARAO Black River Park, 2 Fir Street, Observatory, Cape Town, Western Cape, 7925 Contact: [+27] 0716293858 <+27%2071%20629%203858>

PhD. student, Radio Astronomy Techniques and Technologies, Department of Physics and Electronics, Rhodes University

Skype: benna.cn

SpheMakh commented 4 years ago

Same issue with overflagging in casa 5.6. Please don't remove the casa4.7

I have been using both 4.7 and 5.6 on my tests in #632 and there was no difference between the two for any of the gaincal steps. I prefer to not waste any more time on this since there is no evidence that something is wrong with gaincal in 5.6.

SpheMakh commented 4 years ago

No, I haven't, yet. If casa 5.6 images work ok then I guess we don't need casa 4.7 at all. But just to clarify, we can't use casa_47 cabs with singularity still?

The casa47_* cabs were made to only work with docker. They were never tested with singularity, podman or udocker. So @bennahugo should not have propagated those changes to meerkathi and, definitely should not have made them the defaults. In any case, I fixed this a month ago with this commit https://github.com/SpheMakh/Stimela/commit/295b393810cd4c9ba0eb58fc9ac96cc82f4c407c

My point is I don't want to spend more time on these patches because as far as I'm concerened they not needed since I have not seen any compelling evidence that CASA 4.7 does anything better than 5.6.

bennahugo commented 4 years ago

Just FYI I did not add them to meerkathi - I've been working on my own separate pipelines for quite some time now. I'm getting very high flag applycal percentages (90+%) from K tables in v5+, whereas the exact same steps ran with casa 4.7 results in significantly less flags with precisely the same steps and raw data.

On top of this casa 5 broke KCross calibration and plotants. They also removed plotuv.

bennahugo commented 4 years ago

I have no problem with you taking them out of meerkathi btw, but please don't remove them from stimela

SpheMakh commented 4 years ago

This is the issue with casa47 cabs, the home is set to root, which is not a problem for docker but a big problem for CASA+singularity combo

Singularity stimela_casa_0.3.0.img:/home/sphe/work/MISC/test_casa47> echo $HOME
/root
Singularity stimela_casa_1.2.0.img:~/work/MISC/test_casa47> echo $HOME
/home/sphe
SpheMakh commented 4 years ago

I have no problem with you taking them out of meerkathi btw, but please don't remove them from stimela

They need to be compatible with all container technologies. So they will need to be modified

bennahugo commented 4 years ago

Why not pass a environment flag into singularity at runtime to set the home to os.environ["HOME"] ?

bennahugo commented 4 years ago

if that is the only issue

SpheMakh commented 4 years ago

@bennahugo maybe put at least one them of them in the test so we can tell when the base image is broken so this can be avoided in futre.

bennahugo commented 4 years ago

Lets move this ticket to stimela. It is not a pipeline issue.

KshitijT commented 4 years ago

Credit where it is due, @SpheMakh , I added the casa 47 cabs in meerkathi and made them default, not @bennahugo . ;) At the time when this change was made there was a lot of discussion about it and that seemed like the best way. As I said above, I'll confirm if not using casa 4.7 flags a lot of data. Apologies for all the trouble ! @bennahugo , please don't close this issue - this is a pipeline issue, since casa_47 cabs are defaults at the moment. I have made a PR: #650 , which will make singularity users switch to casa 5.6 instead and at least stop the pipeline from crashing. Please let it go through before closing.

paoloserra commented 4 years ago

At the time when this change was made there was a lot of discussion about it and that seemed like the best way.

Indeed, that was also my recollection and now I'm very confused about whether we should use 4.7 or 5.6 ...

If the 4.7 cabs aren't working but 5.6 gives bad results (as was claimed) we should fix those cabs.

KshitijT commented 4 years ago

Ok, this is fixed in stimela (just wait for @SpheMakh to push the changes to master). We checked that casa_47 cabs now work with singularity.