Madmax [?] misbehaviour

[IanHeywood]

Just performed a fresh installation of CubiCal.

The default parset has --madmax-enable = False. Setting to either True or 1 gives me the following error:

 - 17:07:12 - data_handler       [io] [0.8/1.0 1.7/1.9 1.4Gb]   reading MODEL_DATA for model 0 direction 0
 - 17:07:12 - solver             [x03] [0.2/0.6 1.3/1.6 1.5Gb] D0T0F2 G: 18 iters, conv 100.00%, d/fl 21.88%, PGE 0.00397, stall 78.00%, chi^2 1.013 -> 1.009 (1.012), noise 30.5 -> 30.4
 - 17:07:12 - solver             [x03] [0.2/0.6 1.3/1.6 1.5Gb] D0T0F2 Mad Max took out 18 visibilities: 18 (0.00%) new data flags
 - 17:07:12 - main               [io] [0.2/1.1 1.0/2.0 1.5Gb] I/O job(s) complete
 - 17:07:17 - solver             [x01] [1.0/1.0 2.1/2.1 1.6Gb] Solver for tile 0 chunk D0T0F0 failed with exception: list indices must be integers, not str
 - 17:07:17 - solver             [x01] [1.0/1.0 2.1/2.1 1.6Gb] Traceback (most recent call last):
  File "/home/ianh/Software/CubiCal/cubical/solver.py", line 955, in run_solver
    corr_vis, stats = solver(vdm, soldict, label, sol_opts)
  File "/home/ianh/Software/CubiCal/cubical/solver.py", line 727, in solve_and_correct
    _, stats = _solve_gains(vdm.gm, vdm.weighted_obser, vdm.weighted_model, vdm.flags_arr, sol_opts, label=label)
  File "/home/ianh/Software/CubiCal/cubical/solver.py", line 440, in _solve_gains
    beyond_thunderdome("{} iter {}".format(label, num_iter), thr1, thr2)
  File "/home/ianh/Software/CubiCal/cubical/solver.py", line 348, in beyond_thunderdome
    kill_the_bad_guys(baddies, "global Mad Max ({} sigma)".format(med_threshold))
  File "/home/ianh/Software/CubiCal/cubical/solver.py", line 311, in kill_the_bad_guys
    if ['madmax']['plot'] == 'show':
TypeError: list indices must be integers, not str

Leaving it set to False (which I don't want to do) gives me:

 - 17:08:12 - main               [0.1/0.2 1.1/1.2 1.7Gb] finished processing tile #1/31
 - 17:08:12 - main               [0.1/0.2 1.1/1.2 1.7Gb] waiting for I/O on tile #2/31
 - 17:08:12 - main               [0.1/0.2 1.1/1.2 1.7Gb] submitting solver jobs for tile #2/31
 - 17:08:12 - main               [io] [0.2/1.1 1.0/2.0 1.7Gb] saving tile #1/31
 - 17:08:12 - main               [io] [0.2/1.1 1.0/2.0 1.7Gb] I/O handler for load 2 save 0 failed with exception: save() got an unexpected keyword argument 'unlock'
 - 17:08:12 - main               [io] [0.2/1.1 1.0/2.0 1.7Gb] Traceback (most recent call last):
  File "/home/ianh/Software/CubiCal/cubical/workers.py", line 408, in _io_handler
    tile.save(unlock=finalize)
TypeError: save() got an unexpected keyword argument 'unlock'
 - 17:08:13 - solver             [x03] [0.2/0.6 1.2/1.6 1.7Gb] D0T1F2 G: 12 iters, conv 100.00%, d/fl 18.75%, PGE 0.0022, stall 52.12%, chi^2 1.01 -> 1.009 (1.012), noise 30.2 -> 30.2
 - 17:08:20 - solver             [x04] [1.0/1.1 2.0/2.2 1.7Gb] D0T1F0 G: 14 iters, conv 100.00%, d/fl 18.75%, PGE 0.00103, stall 50.79%, chi^2 1.03 -> 1.027 (1.03), noise 33.3 -> 33.2
 - 17:08:20 - solver             [x05] [1.0/1.1 2.0/2.2 1.7Gb] D0T1F1 G: 14 iters, conv 100.00%, d/fl 18.75%, PGE 0.0013, stall 51.37%, chi^2 1.018 -> 1.016 (1.018), noise 29.9 -> 29.9
 - 17:08:21 - main               [0.1/0.2 1.1/1.2 1.8Gb] finished processing tile #2/31
 - 17:08:21 - main               [0.1/0.2 1.1/1.2 1.8Gb] waiting for I/O on tile #3/31
 - 17:08:21 - main               [0.1/0.2 1.1/1.2 1.8Gb] Exiting with exception: TypeError(save() got an unexpected keyword argument 'unlock')
 Traceback (most recent call last):
  File "/home/ianh/Software/CubiCal/cubical/main.py", line 360, in main
    stats_dict = workers.run_process_loop(ms, tile_list, load_model, single_chunk, solver_type, solver_opts, debug_opts)
  File "/home/ianh/Software/CubiCal/cubical/workers.py", line 205, in run_process_loop
    return _run_multi_process_loop(ms, load_model, solver_type, solver_opts, debug_opts)
  File "/home/ianh/Software/CubiCal/cubical/workers.py", line 251, in _run_multi_process_loop
    if not done or not io_futures[itile].result():
  File "/home/ianh/venv/cubipi/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result
    return self.__get_result()
  File "/home/ianh/venv/cubipi/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414, in __get_result
    raise exception_type, self._exception, self._traceback
TypeError: save() got an unexpected keyword argument 'unlock'

[IanHeywood]

Here's the full option set. It's an ASKAP MS, just to make life interesting.

 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  Selected Options:
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [data] Visibility data options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - ms ................................................ = 2017-12-27_163610_0_x32.ms
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - column ............................................ = DATA
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - time-chunk ........................................ = 96
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - freq-chunk ........................................ = 192
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - rebin-time ........................................ = 1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - rebin-freq ........................................ = 1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - chunk-by .......................................... = SCAN_NUMBER
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - chunk-by-jump ..................................... = 1.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - single-chunk ...................................... =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [sel] Data selection options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - field ............................................. = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - ddid .............................................. = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - taql .............................................. =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - chan .............................................. =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [model] Calibration model options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - list .............................................. = MODEL_DATA
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - ddes .............................................. = auto
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - beam-pattern ...................................... = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - beam-l-axis ....................................... = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - beam-m-axis ....................................... = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [montblanc] Montblanc simulation options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - device-type ....................................... = CPU
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - dtype ............................................. = float
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - feed-type ......................................... = linear
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - mem-budget ........................................ = 1024
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - verbosity ......................................... = WARNING
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - threads ........................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [weight] Weighting options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - column ............................................ = WEIGHT_SPECTRUM
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [flags] General flagging options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - apply ............................................. = -cubical
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - auto-init ......................................... = legacy
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - save .............................................. = cubical
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - save-legacy ....................................... = auto
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - reinit-bitflags ................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - warn-thr .......................................... = 0.3
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - see-no-evil ....................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [postmortem] Options for "postmortem" flagging based on solution statistics
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - enable ............................................ = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - tf-chisq-median ................................... = 1.2
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - tf-np-median ...................................... = 0.5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - time-density ...................................... = 0.5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - chan-density ...................................... = 0.5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - ddid-density ...................................... = 0.5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [madmax] Options for the "Mad Max" flagger
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - enable ............................................ = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - estimate .......................................... = corr
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - diag .............................................. = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - offdiag ........................................... = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - threshold ......................................... = [0, 10]
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - global-threshold .................................. = [0, 12]
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - plot .............................................. = 1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - plot-frac-above ................................... = 0.01
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [sol] Solution options which apply at the solver level
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - jones ............................................. = G
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - precision ......................................... = 32
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - delta-g ........................................... = 1e-06
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - delta-chi ......................................... = 1e-06
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - chi-int ........................................... = 5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - last-rites ........................................ = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - stall-quorum ...................................... = 0.99
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - diag-diag ......................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - term-iters ........................................ =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - min-bl ............................................ = 100.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - max-bl ............................................ = 100000000.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - subset ............................................ =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - load-from ......................................... =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - compute-2x2 ....................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - apply-2x2 ......................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - save-to ........................................... = {data[ms]}/BBC-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - per-chan .......................................... = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - plot .............................................. = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [dist] Parallelization and distribution options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - ncpu .............................................. = 6
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - nworker ........................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - nthread ........................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - max-chunks ........................................ = 5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - min-chunks ........................................ = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - pin ............................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - pin-io ............................................ = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - pin-main .......................................... = io
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [out] Options for output products
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - name .............................................. = cube_pcal_2017-12-27_163610_0_x32.ms_2018-07-03-17-07-07
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - mode .............................................. = sc
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - column ............................................ = CORRECTED_DATA
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - model-column ...................................... =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - reinit-column ..................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - subtract-model .................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - subtract-dirs ..................................... = :
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - plots ............................................. = 1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - casa-gaintables ................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [log] Options related to logging
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - memory ............................................ = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - boring ............................................ = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - append ............................................ = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - verbose ........................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - file-verbose ...................................... = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [debug] Debugging options for the discerning masochist
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - pdb ............................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - panic-amplitude ................................... = 0.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - stop-before-solver ................................ = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [misc] Miscellaneous options
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - random-seed ....................................... = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - parset-version .................................... = 0.1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [g] Options for G-Jones term
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - solvable .......................................... = True
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - type .............................................. = phase-diag
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - load-from ......................................... =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - xfer-from ......................................... =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - save-to ........................................... = {data[ms]}/{JONES}-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - dd-term ........................................... = 0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - fix-dirs .......................................... =
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - diag-diag ......................................... = False
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - update-type ....................................... = full
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - time-int .......................................... = 12
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - freq-int .......................................... = 48
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - max-prior-error ................................... = 0.1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - max-post-error .................................... = 0.1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - clip-low .......................................... = 0.1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - clip-high ......................................... = 10.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - clip-after ........................................ = 5
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - max-iter .......................................... = 20
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - conv-quorum ....................................... = 0.99
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - ref-ant ........................................... = None
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - prop-flags ........................................ = default
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] [de] de
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - dd-term ........................................... = 1
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - clip-low .......................................... = 0.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb]  - clip-high ......................................... = 0.0
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] Enabling G-Jones
 - 17:07:08 - main               [0.1/0.1 0.7/0.7 0.7Gb] mode: solve and correct

[o-smirnov]

Sorry. Shocking bloopers. Not sure how this passed testing! I'll fix it tonight or tomorrow early.

Cheers, Oleg

---

Sent from my phone. Quality of spelling inversely proportional to finger size.

On Tue, 03 Jul 2018, 18:13 Ian Heywood, notifications@github.com wrote:

Here's the full option set. It's an ASKAP MS, just to make life interesting.

• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] Selected Options:
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [data] Visibility data options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - ms ................................................ = 2017-12-27_163610_0_x32.ms
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - column ............................................ = DATA
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - time-chunk ........................................ = 96
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - freq-chunk ........................................ = 192
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - rebin-time ........................................ = 1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - rebin-freq ........................................ = 1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - chunk-by .......................................... = SCAN_NUMBER
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - chunk-by-jump ..................................... = 1.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - single-chunk ...................................... =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [sel] Data selection options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - field ............................................. = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - ddid .............................................. = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - taql .............................................. =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - chan .............................................. =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [model] Calibration model options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - list .............................................. = MODEL_DATA
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - ddes .............................................. = auto
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - beam-pattern ...................................... = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - beam-l-axis ....................................... = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - beam-m-axis ....................................... = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [montblanc] Montblanc simulation options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - device-type ....................................... = CPU
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - dtype ............................................. = float
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - feed-type ......................................... = linear
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - mem-budget ........................................ = 1024
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - verbosity ......................................... = WARNING
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - threads ........................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [weight] Weighting options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - column ............................................ = WEIGHT_SPECTRUM
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [flags] General flagging options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - apply ............................................. = -cubical
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - auto-init ......................................... = legacy
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - save .............................................. = cubical
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - save-legacy ....................................... = auto
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - reinit-bitflags ................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - warn-thr .......................................... = 0.3
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - see-no-evil ....................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [postmortem] Options for "postmortem" flagging based on solution statistics
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - enable ............................................ = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - tf-chisq-median ................................... = 1.2
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - tf-np-median ...................................... = 0.5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - time-density ...................................... = 0.5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - chan-density ...................................... = 0.5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - ddid-density ...................................... = 0.5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [madmax] Options for the "Mad Max" flagger
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - enable ............................................ = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - estimate .......................................... = corr
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - diag .............................................. = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - offdiag ........................................... = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - threshold ......................................... = [0, 10]
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - global-threshold .................................. = [0, 12]
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - plot .............................................. = 1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - plot-frac-above ................................... = 0.01
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [sol] Solution options which apply at the solver level
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - jones ............................................. = G
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - precision ......................................... = 32
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - delta-g ........................................... = 1e-06
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - delta-chi ......................................... = 1e-06
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - chi-int ........................................... = 5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - last-rites ........................................ = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - stall-quorum ...................................... = 0.99
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - diag-diag ......................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - term-iters ........................................ =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - min-bl ............................................ = 100.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - max-bl ............................................ = 100000000.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - subset ............................................ =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - load-from ......................................... =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - compute-2x2 ....................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - apply-2x2 ......................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - save-to ........................................... = {data[ms]}/BBC-field{sel[field]}-ddid{sel[ddid]}.parmdb
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - per-chan .......................................... = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - plot .............................................. = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [dist] Parallelization and distribution options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - ncpu .............................................. = 6
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - nworker ........................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - nthread ........................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - max-chunks ........................................ = 5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - min-chunks ........................................ = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - pin ............................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - pin-io ............................................ = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - pin-main .......................................... = io
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [out] Options for output products
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - name .............................................. = cube_pcal_2017-12-27_163610_0_x32.ms_2018-07-03-17-07-07
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - mode .............................................. = sc
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - column ............................................ = CORRECTED_DATA
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - model-column ...................................... =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - reinit-column ..................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - subtract-model .................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - subtract-dirs ..................................... = :
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - plots ............................................. = 1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - casa-gaintables ................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [log] Options related to logging
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - memory ............................................ = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - boring ............................................ = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - append ............................................ = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - verbose ........................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - file-verbose ...................................... = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [debug] Debugging options for the discerning masochist
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - pdb ............................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - panic-amplitude ................................... = 0.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - stop-before-solver ................................ = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [misc] Miscellaneous options
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - random-seed ....................................... = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - parset-version .................................... = 0.1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [g] Options for G-Jones term
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - solvable .......................................... = True
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - type .............................................. = phase-diag
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - load-from ......................................... =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - xfer-from ......................................... =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - save-to ........................................... = {data[ms]}/{JONES}-field{sel[field]}-ddid{sel[ddid]}.parmdb
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - dd-term ........................................... = 0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - fix-dirs .......................................... =
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - diag-diag ......................................... = False
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - update-type ....................................... = full
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - time-int .......................................... = 12
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - freq-int .......................................... = 48
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - max-prior-error ................................... = 0.1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - max-post-error .................................... = 0.1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - clip-low .......................................... = 0.1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - clip-high ......................................... = 10.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - clip-after ........................................ = 5
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - max-iter .......................................... = 20
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - conv-quorum ....................................... = 0.99
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - ref-ant ........................................... = None
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - prop-flags ........................................ = default
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] [de] de
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - dd-term ........................................... = 1
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - clip-low .......................................... = 0.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] - clip-high ......................................... = 0.0
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] Enabling G-Jones
• 17:07:08 - main [0.1/0.1 0.7/0.7 0.7Gb] mode: solve and correct

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/197#issuecomment-402212103, or mute the thread https://github.com/notifications/unsubscribe-auth/AGK5v2OSSZnmoAlkaw-fVBxy2F5iOJYPks5uC5g-gaJpZM4VBPkR .

[IanHeywood]

Thanks chief, no rush.

[o-smirnov]

OK that was a couple of embarrassing bloopers. @IanHeywood can you please test the issue-197 branch? I all works, we'll merge this in.

[IanHeywood]

Can confirm that issue-197 branch is working.

Cheers.

[o-smirnov]

Thanks! Going to issue a PR now.

[IanHeywood]

Hold up! A second run on the same MS after tweaking some solution intervals gave this error:

 - 13:45:06 - data_handler       [io] [2.4/6.6 3.2/8.1 3.5Gb] tile #7/7: reading MS rows 367064~379439
 - 13:45:07 - data_handler       [io] [2.7/6.6 3.5/8.1 3.5Gb]   applying solvable baseline cutoff deselects 2457 rows
 - 13:45:07 - data_handler       [io] [2.7/6.6 3.5/8.1 3.5Gb]   19.85% visibilities deselected via specificed subset and/or baseline cutoffs
 - 13:45:08 - data_handler       [io] [2.8/6.6 3.6/8.1 3.5Gb]   52.97% input visibilities flagged and/or deselected
 - 13:45:09 - data_handler       [io] [3.0/6.6 3.9/8.1 3.8Gb]   reading MODEL_DATA for model 0 direction 0
 - 13:45:10 - main               [io] [2.4/6.6 3.2/8.1 4.0Gb] I/O job(s) complete
 - 13:45:10 - main               [io] [2.4/6.6 3.2/8.1 4.0Gb] saving tile #6/7
 - 13:45:10 - main               [0.2/0.2 1.1/1.2 4.0Gb] submitting solver jobs for tile #7/7
 - 13:45:10 - data_handler       [io] [2.4/6.6 3.2/8.1 4.0Gb] tile #6/7: saving MS rows 303824~367063
 - 13:45:10 - data_handler       [io] [2.4/6.6 4.0/8.1 4.0Gb]   writing CORRECTED_DATA column
 - 13:45:19 - data_handler       [io] [3.5/6.6 4.3/8.1 4.0Gb]   11.78% visibilities flagged by solver: saving to BITFLAG and FLAG columns
 - 13:45:23 - data_handler       [io] [3.6/6.6 4.4/8.1 4.0Gb]   updated BITFLAG column (58.11% visibilities flagged)
 - 13:45:23 - data_handler       [io] [3.6/6.6 4.4/8.1 4.0Gb]   updated BITFLAG_ROW column (58.02% rows flagged)
 - 13:45:23 - data_handler       [io] [3.6/6.6 4.4/8.1 4.0Gb]   updated FLAG column (58.11% visibilities flagged)
 - 13:45:23 - data_handler       [io] [3.6/6.6 4.4/8.1 4.0Gb]   updated FLAG_ROW column (58.02% rows flagged)
 - 13:45:27 - solver             [x04] [2.7/3.3 3.7/5.0 1.1Gb] D0T30F0 G: 11 iters, conv 100.00%, d/fl 18.75%, PGE 0.000507, stall 53.83%, chi^2 1.038 -> 1.031 (1.037), noise 32.7 -> 32.6
 - 13:45:27 - solver             [x04] [2.7/3.3 3.7/5.0 1.1Gb] D0T30F0 Mad Max took out 1632 visibilities: 1632 (0.02%) new data flags
 - 13:45:29 - main               [0.2/0.2 1.1/1.2 1.2Gb] finished processing tile #7/7
 - 13:45:35 - main               [io] [2.4/6.6 3.2/8.1 1.2Gb] I/O job(s) complete
 - 13:45:35 - main               [io] [2.4/6.6 3.2/8.1 1.2Gb] saving tile #7/7
 - 13:45:35 - data_handler       [io] [2.4/6.6 3.2/8.1 1.2Gb] tile #7/7: saving MS rows 367064~379439
 - 13:45:35 - data_handler       [io] [2.4/6.6 3.4/8.1 1.2Gb]   writing CORRECTED_DATA column
 - 13:45:37 - data_handler       [io] [2.6/6.6 3.4/8.1 1.2Gb]   11.78% visibilities flagged by solver: saving to BITFLAG and FLAG columns
 - 13:45:38 - data_handler       [io] [2.7/6.6 3.5/8.1 1.2Gb]   updated BITFLAG column (44.91% visibilities flagged)
 - 13:45:38 - data_handler       [io] [2.7/6.6 3.5/8.1 1.2Gb]   updated BITFLAG_ROW column (44.84% rows flagged)
 - 13:45:38 - data_handler       [io] [2.7/6.6 3.5/8.1 1.2Gb]   updated FLAG column (44.91% visibilities flagged)
 - 13:45:38 - data_handler       [io] [2.7/6.6 3.5/8.1 1.2Gb]   updated FLAG_ROW column (44.84% rows flagged)
 - 13:45:38 - main               [io] [2.7/6.6 3.5/8.1 1.2Gb] I/O handler for load None save -1 failed with exception: Keyword AUTOINIT_IN_PROGRESS in column BITFLAG does not exist
 - 13:45:38 - main               [io] [2.7/6.6 3.5/8.1 1.2Gb] Traceback (most recent call last):
  File "/home/ianh/Software/issue-197/CubiCal/cubical/workers.py", line 408, in _io_handler
    tile.save(final=finalize)
  File "/home/ianh/Software/issue-197/CubiCal/cubical/data_handler/ms_tile.py", line 1100, in save
    self.dh.finalize()
  File "/home/ianh/Software/issue-197/CubiCal/cubical/data_handler/ms_data_handler.py", line 1252, in finalize
    self.ms.removecolkeyword("BITFLAG", "AUTOINIT_IN_PROGRESS")
  File "/home/ianh/venv/issue-197/local/lib/python2.7/site-packages/casacore/tables/table.py", line 1375, in removecolkeyword
    self._removekeyword (columnname, keyword, -1);
RuntimeError: Keyword AUTOINIT_IN_PROGRESS in column BITFLAG does not exist
 - 13:45:40 - main               [0.2/0.2 1.1/1.2 1.2Gb] Exiting with exception: RuntimeError(Keyword AUTOINIT_IN_PROGRESS in column BITFLAG does not exist)
 Traceback (most recent call last):
  File "/home/ianh/Software/issue-197/CubiCal/cubical/main.py", line 360, in main
    stats_dict = workers.run_process_loop(ms, tile_list, load_model, single_chunk, solver_type, solver_opts, debug_opts)
  File "/home/ianh/Software/issue-197/CubiCal/cubical/workers.py", line 205, in run_process_loop
    return _run_multi_process_loop(ms, load_model, solver_type, solver_opts, debug_opts)
  File "/home/ianh/Software/issue-197/CubiCal/cubical/workers.py", line 285, in _run_multi_process_loop
    ms.update_flag_counts(io_futures[-1].result()['flagcounts'])
  File "/home/ianh/venv/issue-197/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result
    return self.__get_result()
  File "/home/ianh/venv/issue-197/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414, in __get_result
    raise exception_type, self._exception, self._traceback
RuntimeError: Keyword AUTOINIT_IN_PROGRESS in column BITFLAG does not exist

[o-smirnov]

Deja oops. Can you try now?

[IanHeywood]

Success!

Cheers.