ratt-ru / Stimela-classic

Containerized radio interferometry scripting framework -- NB: Classic version is no longer in active development, use stimela 2! See README for details.
GNU General Public License v2.0
28 stars 16 forks source link

Cubical: 3GC calibration - input parameter parsing error #785

Closed rramij closed 1 year ago

rramij commented 1 year ago

I am trying to make 3GC image with Stimela following the steps and parameters used in CARAcal "ddcal" worker. But I am getting a weird error gocubical: error: no such option: --dd-dd-term. Below is the full log.

job started at 2023-07-18 14:12:05.941105
Starting container [DDcal_round_1-14030384025086416896823180606809]. Timeout set to -1. The container ID is printed below.
running cd /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949 && singularity run --workdir /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949 --containall  --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949/stimela_parameter_files/DDcal_round_1-14030384025086416896823180606809.json:/stimela_mount/configfile:ro --bind /home/rramij/venv/lib/python3.8/site-packages/stimela/cargo/cab/cubical_ddf/src:/stimela_mount/code:ro --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949/passwd:/etc/passwd:rw --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949/group:/etc/group:rw --bind /home/rramij/venv/bin/stimela_runscript:/singularity:ro --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/msdir:/stimela_mount/msdir:rw --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/input:/stimela_mount/input:ro --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/output:/stimela_mount/output:rw --bind /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/output/tmp:/stimela_mount/output/tmp:rw /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/stimela_cubical_1.7.2.sif /singularity
WARNING: Overriding HOME environment variable with SINGULARITYENV_HOME is not permitted
INFO      14:12:10 - main               | Using CubiCal version 1.6.1.
Usage: gocubical [parset file] <options>

gocubical: error: no such option: --dd-dd-term
Traceback (most recent call last):
  File "/stimela_mount/code/run.py", line 57, in <module>
    subprocess.check_call(shlex.split(_runc))
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gocubical', '--sol-jones', 'G,', 'DD', '--data-ms', '/stimela_mount/msdir/A85_700_GWB_CORR.ms', '--data-column', 'DATA', '--data-time-chunk', '384', '--data-freq-chunk', '0', '--model-list', 'MODEL_DATA+-/stimela_mount/output/DD_precal.DicoModel@/stimela_mount/output/dE.reg:/stimela_mount/output/DD_precal.DicoModel@/stimela_mount/output/dE.reg', '--model-ddes', 'auto', '--weight-column', 'WEIGHT', '--madmax-enable', '1', '--madmax-estimate', 'corr', '--madmax-plot', '0', '--madmax-global-threshold', '0.0,12.0', '--madmax-threshold', '0.0,10.0', '--sol-stall-quorum', '0.95', '--sol-term-iters', '50,90,50,90', '--sol-min-bl', '100', '--bbc-save-to', '{out[name]}-BBC-field_{sel[field]}-ddid_{sel[ddid]}.parmdb', '--dist-ncpu', '64', '--dist-max-chunks', '0', '--dist-nthread', '0', '--out-name', '/stimela_mount/output/dE_sub-1', '--out-overwrite', '1', '--out-mode', 'sr', '--out-column', 'CORRECTED_DATA', '--out-model-column', 'MODEL_OUT', '--out-subtract-dirs', '1:', '--out-casa-gaintables', '0', '--out-plots', '1', '--dist-nworker', '0', '--log-memory', '1', '--log-boring', '1', '--g-time-int', '64', '--g-freq-int', '64', '--g-clip-low', '0.5', '--g-clip-high', '1.5', '--g-solvable', '1', '--g-type', 'complex-2x2', '--g-save-to', '/stimela_mount/output/g-phase-gains-1', '--g-update-type', 'phase-diag', '--g-max-prior-error', '0.35', '--g-max-post-error', '0.35', '--dd-dd-term', '1', '--dd-time-int', '64', '--dd-freq-int', '64', '--dd-clip-low', '0.0', '--dd-clip-high', '0.0', '--dd-solvable', '1', '--dd-type', 'complex-2x2', '--dd-save-to', '/stimela_mount/output/dd-cal-1', '--dd-fix-dirs', '0', '--dd-max-prior-error', '0.35', '--dd-max-post-error', '0.35', '--dd-prop-flags', 'always', '--degridding-OverS', '11', '--degridding-Support', '7', '--degridding-Nw', '100', '--degridding-wmax', '0', '--degridding-Padding', '1.7', '--degridding-NDegridBand', '15', '--degridding-MaxFacetSize', '0.15', '--degridding-MinNFacetPerAxis', '1']' returned non-zero exit status 2.
cd /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949 && singularity run --workdir /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949 --containall returns error code 1
job failed at 2023-07-18 14:12:10.585019 after 0:00:04.643914
Traceback (most recent call last):
  File "/home/rramij/venv/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run
    job.run_job()
  File "/home/rramij/venv/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job
    self.job.run(output_wrangler=self.apply_output_wranglers)
  File "/home/rramij/venv/lib/python3.8/site-packages/stimela/singularity.py", line 123, in run
    utils.xrun(f"cd {self.execdir} && singularity run --workdir {self.execdir} --containall",
  File "/home/rramij/venv/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: cd /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949 && singularity run --workdir /home/rramij/Work_Dir/A85_700MHz/GWB/GMRT_DI_pipe_run/Useful/selfcal/1run/msdir/DDcal/.stimela_workdir-16896823177916949 --containall returns error code 1

Why is it working fine in the CARACal pipeline but not in Stimela even with the same parameters? The input parameters are as follows:

dicomod = "DD_precal.DicoModel"
dereg = "dE.reg"
recipe.add("cab/cubical_ddf", "DDcal_round_1", {

# DATA
    "data-ms": msfile,
    "data-column": "DATA",
    "data-time-chunk": data_time_chunk,
    "data-freq-chunk": data_freq_chunk,
# MODEL
    "model-list": spf("MODEL_DATA+-{{}}{}@{{}}{}:{{}}{}@{{}}{}".format(dicomod, dereg, dicomod, dereg), "output", "output", "output", "output"),
# WEIGHT
    "weight-column": "WEIGHT",
# MADMAX
    "madmax-enable": True,
    "madmax-estimate": "corr",
    "madmax-plot": False,
    "madmax-global-threshold": [0.,12.0],
    "madmax-threshold": [0.,10.0],
# SOL
    "sol-jones": "G, DD",   
    "sol-min-bl": 100,
    "sol-stall-quorum": 0.95,
    "sol-term-iters": [50,90,50,90],
# DIST
    "dist-ncpu": 64,
    "dist-max-chunks": 0,
    "dist-nthread": 0,
    "dist-nworker": 0,
# OUT
    "out-name": "dE_sub-1",
    "out-mode": "sr",
    "out-column": "CORRECTED_DATA",
    "out-model-column": "MODEL_OUT",
    "out-plots": True,
    "out-subtract-dirs": "1:",
# LOG
    "log-memory": True,
    "log-boring": True,
# G
    "g-time-int": 64, # sec
    "g-freq-int": 64, # chans
    "g-clip-low": 0.5,
    "g-clip-high": 1.5,
    "g-solvable": True,
    "g-type": "complex-2x2",
    "g-save-to": "g-phase-gains-1",
    "g-update-type": "phase-diag",
    "g-max-prior-error": 0.35,
    "g-max-post-error": 0.35,
# DD
    "dd-max-prior-error": 0.35,
    "dd-max-post-error": 0.35,
    "dd-save-to": "dd-cal-1",
    "dd-type": "complex-2x2",
    "dd-clip-high": 0.0,
    "dd-clip-low": 0.0,
    "dd-solvable": True,
    "dd-time-int": 64,
    "dd-freq-int": 64,
    "dd-dd-term": True,
    "dd-prop-flags": 'always',
    "dd-fix-dirs": "0",
# DEGRIDDING
    "degridding-NDegridBand": 15,
    'degridding-MaxFacetSize': 0.15

    }, input=INPUT, output=OUTPUT, label="Cubical calibration 1")

Please help me with this.

JSKenyon commented 1 year ago

Hey! I think that the problem is in the --sol-jones specification. Omit the space after the comma in G, DD. Hope that works!

rramij commented 1 year ago

Yeah, that was the problem. It is working now. Thanks.