search

ratt-ru / CubiCal

A fast radio interferometric calibration suite.
GNU General Public License v2.0
18 stars 13 forks source link

Slopecal diverges #321

Open bennahugo opened 4 years ago

bennahugo commented 4 years ago

I've done a calibration test against CASA 4.7 (note 4.7 not 5 - 5's delaycal is also broken) at native resolution of 4s. The cubical gains diverge whereas the 4.7 solutions substantially improves the image. My options are pretty much box standard. The chi^2 diverges in a lot of tiles, however there is ample SNR to solve for the delays:

INFO       - 09:37:37 - cc                 [0.1/0.1 2.5/2.6 0.3Gb] set global console verbosity level 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] started /net/com08/data/bhugo/projects/projects/venv/bin/gocubical slopecal.cfg
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] using output basename slopecal.cc-out/cc
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] slopecal.cc-out/cc.parset already exists, possibly from a previous run
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] backed up existing slopecal.cc-out to slopecal.cc-out.10
INFO       - 09:37:37 - shm_utils          [0.1/0.1 2.5/2.6 0.3Gb] reaping 1 shared memory objects associated with 1 dead cubical processes
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  Selected Options:
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [data] Visibility data options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ms ................................................ = msdir/COMBINED.ms
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - column ............................................ = CORRECTED_DATA
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - time-chunk ........................................ = 8
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - freq-chunk ........................................ = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - rebin-time ........................................ = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - rebin-freq ........................................ = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - chunk-by .......................................... = SCAN_NUMBER
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - chunk-by-jump ..................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - single-chunk ...................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - single-tile ....................................... = -1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [sel] Data selection options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - field ............................................. = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ddid .............................................. = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - taql .............................................. =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - chan .............................................. =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - diag .............................................. = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [out] Options for output products
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - dir ............................................... = slopecal
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - name .............................................. = slopecal.cc-out/cc
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - overwrite ......................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - backup ............................................ = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - mode .............................................. = sc
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - apply-solver-flags ................................ = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - column ............................................ = SC_DATA
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - derotate .......................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - model-column ...................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - weight-column ..................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - reinit-column ..................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - subtract-model .................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - subtract-dirs ..................................... = :
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - plots ............................................. = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - casa-gaintables ................................... = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [model] Calibration model options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - list .............................................. = MODEL_DATA
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ddes .............................................. = auto
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - beam-pattern ...................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - beam-l-axis ....................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - beam-m-axis ....................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - feed-rotate ....................................... = auto
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - pa-rotate ......................................... = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [montblanc] Montblanc simulation options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - device-type ....................................... = CPU
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - dtype ............................................. = float
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - mem-budget ........................................ = 1024
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - verbosity ......................................... = WARNING
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - threads ........................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - pa-rotate ......................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [weight] Weighting options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - column ............................................ = WEIGHT
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - fill-offdiag ...................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - legacy-v1-2 ....................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [flags] General flagging options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - apply ............................................. = -cubical
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - auto-init ......................................... = legacy
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - save .............................................. = cubical
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - save-legacy ....................................... = auto
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - reinit-bitflags ................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - warn-thr .......................................... = 0.3
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - see-no-evil ....................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [degridding] Options for the degridder. Only in use when predicting from DicoModels using DDFacet
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - OverS ............................................. = 11
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - Support ........................................... = 7
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - Nw ................................................ = 100
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - wmax .............................................. = 0.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - Padding ........................................... = 1.7
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - NDegridBand ....................................... = 16
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - MaxFacetSize ...................................... = 0.25
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - MinNFacetPerAxis .................................. = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - NProcess .......................................... = 8
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [postmortem] Options for "postmortem" flagging based on solution statistics
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - enable ............................................ = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - tf-chisq-median ................................... = 1.2
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - tf-np-median ...................................... = 0.5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - time-density ...................................... = 0.5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - chan-density ...................................... = 0.5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ddid-density ...................................... = 0.5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [madmax] Options for the "Mad Max" flagger
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - enable ............................................ = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - residuals ......................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - estimate .......................................... = corr
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - diag .............................................. = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - offdiag ........................................... = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - threshold ......................................... = [0, 35, 25]
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - global-threshold .................................. = [0, 40, 30]
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - plot .............................................. = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - plot-frac-above ................................... = 0.01
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - plot-bl ........................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - flag-ant .......................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - flag-ant-thr ...................................... = 5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [sol] Solution options which apply at the solver level
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - jones ............................................. = slope
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - precision ......................................... = 32
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - delta-g ........................................... = 1e-06
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - delta-chi ......................................... = 1e-06
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - chi-int ........................................... = 5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - last-rites ........................................ = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - stall-quorum ...................................... = 0.99
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - term-iters ........................................ = 50
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - min-bl ............................................ = 150.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-bl ............................................ = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - subset ............................................ =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - load-from ......................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - compute-2x2 ....................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - apply-2x2 ......................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - save-to ........................................... = {out[name]}-BBC-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - per-chan .......................................... = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - plot .............................................. = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [dist] Parallelization and distribution options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ncpu .............................................. = 32
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - nworker ........................................... = 16
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - nthread ........................................... = 8
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-chunks ........................................ = 10
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - min-chunks ........................................ = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - pin ............................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - pin-io ............................................ = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - pin-main .......................................... = io
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [log] Options related to logging
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - memory ............................................ = True
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - stats ............................................. = chi2:.3f
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - stats-warn ........................................ = chi2:10
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - boring ............................................ = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - append ............................................ = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - verbose ........................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - file-verbose ...................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [debug] Debugging options for the discerning masochist
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - pdb ............................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - panic-amplitude ................................... = 0.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - stop-before-solver ................................ = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - escalate-warnings ................................. = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [misc] Miscellaneous options
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - random-seed ....................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - parset-version .................................... = 0.1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [g] g
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - dd-term ........................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - time-int .......................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - freq-int .......................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-low .......................................... = 0.1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-high ......................................... = 10
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-after ........................................ = 5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - conv-quorum ....................................... = 0.99
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ref-ant ........................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [de] de
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - dd-term ........................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-low .......................................... = 0.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-high ......................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - delta-chi ......................................... = 1e-05
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-prior-error ................................... = 0.44
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-post-error .................................... = 0.44
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] [slope] Options for slope-Jones term
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - solvable .......................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - type .............................................. = f-slope
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - load-from ......................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - xfer-from ......................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - save-to ........................................... = {out[name]}-{JONES}-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - dd-term ........................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - fix-dirs .......................................... =
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - update-type ....................................... = diag
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - time-int .......................................... = 8
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - freq-int .......................................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-prior-error ................................... = 2000000.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-post-error .................................... = 2000000.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - low-snr-warn ...................................... = 75
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - high-gain-var-warn ................................ = 30
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-low .......................................... = 0.1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-high ......................................... = 10.0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - clip-after ........................................ = 5
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - max-iter .......................................... = 50
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - epsilon ........................................... = 1e-06
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - delta-chi ......................................... = 1e-06
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - conv-quorum ....................................... = 0.99
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - ref-ant ........................................... = None
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - prop-flags ........................................ = default
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - estimate-pzd ...................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - diag-only ......................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - offdiag-only ...................................... = False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - robust-cov ........................................ = compute
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - robust-scale ...................................... = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - robust-npol ....................................... = 2
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - robust-int ........................................ = 1
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb]  - robust-save-weights ............................... = 0
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] Enabling slope-Jones
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] mode: SolveAndCorrect
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] solver is apply-only type: False
INFO       - 09:37:37 - main               [0.1/0.1 2.5/2.6 0.3Gb] solver requires model: True
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb] reading MS msdir/COMBINED.ms
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb]   detected linear (xy) feeds
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb]   fields are *0: 0408-65, 1: 3C138, 2: 3C286, 3: PKS1934-638
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb]   MS contains 1 spectral windows
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb]   max freq chunk size is -- channels and/or -- MHz
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb]   DDID 0: 4096/4096 channels selected
INFO       - 09:37:37 - data_handler       [0.1/0.1 2.5/2.6 0.3Gb]     found 1 frequency chunks: 0 4096
INFO       - 09:37:39 - data_handler       [0.1/0.2 2.5/2.6 0.3Gb]   applying TAQL query 'FIELD_ID == 0 && DATA_DESC_ID IN [0]' (4796397/14452970 rows selected)
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb]   46 antennas, 4796397 rows, 1/1 DDIDs, 4437 timeslots, 4 corrs
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb]   DDID central frequencies are at 1.28 GHz
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb] Input model feed rotation disabled, PA rotation enabled
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb] Output visibilities derotation enabled
INFO       - 09:37:40 - parallactic_machin [0.2/0.2 2.5/2.6 0.3Gb] Initializing new parallactic angle machine for 46 ECEF positions
INFO       - 09:37:40 - parallactic_machin [0.2/0.2 2.5/2.6 0.3Gb] Conversion epoch is J2000
INFO       - 09:37:40 - parallactic_machin [0.2/0.2 2.5/2.6 0.3Gb] Selecting 'linear' feed bases
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb] Using 1 model(s) for 1 directions(s) (DDEs explicitly disabled)
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb]   model 0 (weight WEIGHT):
INFO       - 09:37:40 - data_handler       [0.2/0.2 2.5/2.6 0.3Gb]     direction 0: MODEL_DATA
INFO       - 09:37:40 - main               [0.2/0.2 2.5/2.6 0.3Gb] subtraction directions set to slice(None, None, None)
INFO       - 09:37:40 - param_db           [0.2/0.2 2.5/2.6 0.3Gb] creating slopecal.cc-out/cc-slope-field_0-ddid_None.parmdb in fragmented mode
INFO       - 09:37:40 - gain_machine       [0.2/0.2 2.5/2.6 0.3Gb] slope solutions will be saved to slopecal.cc-out/cc-slope-field_0-ddid_None.parmdb
INFO       - 09:37:40 - param_db           [0.2/0.2 2.5/2.6 0.3Gb] creating slopecal.cc-out/cc-BBC-field_0-ddid_None.parmdb in fragmented mode
INFO       - 09:37:40 - gain_machine       [0.2/0.2 3.0/3.0 0.3Gb] will compute & save suggested baseline-based corrections (BBCs) to slopecal.cc-out/cc-BBC-field_0-ddid_None.parmdb
INFO       - 09:37:40 - gain_machine       [0.2/0.2 3.0/3.0 0.3Gb]   (these can optionally be applied in a subsequent CubiCal run)
INFO       - 09:37:40 - main               [0.2/0.2 3.0/3.0 0.3Gb] multi-process mode: --dist-nworker 16 (+1), --dist-nthread 8
INFO       - 09:37:40 - main               [0.2/0.2 3.0/3.0 0.3Gb] defining chunks (time 8, freq 0, also when SCAN_NUMBER jumps > 1.0)
INFO       - 09:37:41 - data_handler       [0.5/0.5 3.3/3.3 0.3Gb]   read indexing columns (4796397 total rows)
INFO       - 09:37:42 - data_handler       [0.5/0.5 3.4/3.4 0.3Gb]   built timeslot index (4437 unique timestamps)
INFO       - 09:37:42 - data_handler       [0.5/0.5 3.4/3.4 0.3Gb]   max chunk size is 8 timeslots and/or -- seconds
INFO       - 09:37:53 - data_handler       [0.6/1.7 3.5/4.6 0.3Gb]   found 558 time chunks: 0:0:0 1:8648:8 2:17296:16 3:25944:24 4:34592:32 5:43240:40 6:51888:48 7:60536:56 8:69184:64 9:77832:72 10:86480:80 11:95128:88 12:103776:96 13:112424:104 14:121072:112 15:129720:120 16:138368:128 17:147016:136 18:155664:144 19:164312:152 20:172960:160 21:181608:168 22:190256:176 23:1989
04:184 24:207552:192 25:216200:200 26:224848:208 27:233496:216 28:242144:224 29:250792:232 30:259440:240 31:268088:248 32:276736:256 33:285384:264 34:294032:272 35:302680:280 36:311328:288 37:319976:296 38:328624:304 39:337272:312 40:345920:320 41:354568:328 42:363216:336 43:371864:344 44:380512:352 45:389160:360 46:397808:368 47:406456:376 48:415104:384 49:423752:392 50:43240
0:400 51:441048:408 52:449696:416 53:458344:424 54:466992:432 55:475640:440 56:484288:448 57:492936:456 58:501584:464 59:510232:472 60:518880:480 61:527528:488 62:536176:496 63:544824:504 64:553472:512 65:562120:520 66:570768:528 67:579416:536 68:588064:544 69:596712:552 70:605360:560 71:614008:568 72:622656:576 73:631304:584 74:639952:592 75:647519:599 76:656167:607 77:664815
:615 78:673463:623 79:682111:631 80:690759:639 81:699407:647 82:708055:655 83:716703:663 84:725351:671 85:733999:679 86:742647:687 87:751295:695 88:759943:703 89:768591:711 90:777239:719 91:785887:727 92:794535:735 93:803183:743 94:811831:751 95:820479:759 96:829127:767 97:837775:775 98:846423:783 99:855071:791 100:863719:799 101:872367:807 102:881015:815 103:889663:823 104:89
8311:831 105:906959:839 106:915607:847 107:924255:855 108:932903:863 109:941551:871 110:950199:879 111:958847:887 112:967495:895 113:971819:899 114:980467:907 115:989115:915 116:997763:923 117:1006411:931 118:1015059:939 119:1023707:947 120:1032355:955 121:1041003:963 122:1049651:971 123:1058299:979 124:1066947:987 125:1075595:995 126:1084243:1003 127:1092891:1011 128:1101539:
1019 129:1110187:1027 130:1118835:1035 131:1127483:1043 132:1136131:1051 133:1144779:1059 134:1153427:1067 135:1162075:1075 136:1170723:1083 137:1179371:1091 138:1188019:1099 139:1196667:1107 140:1205315:1115 141:1213963:1123 142:1222611:1131 143:1231259:1139 144:1239907:1147 145:1248555:1155 146:1257203:1163 147:1265851:1171 148:1274499:1179 149:1283147:1187 150:1291795:1195 
151:1296119:1199 152:1304767:1207 153:1313415:1215 154:1322063:1223 155:1330711:1231 156:1339359:1239 157:1348007:1247 158:1356655:1255 159:1365303:1263 160:1373951:1271 161:1382599:1279 162:1391247:1287 163:1399895:1295 164:1408543:1303 165:1417191:1311 166:1425839:1319 167:1434487:1327 168:1443135:1335 169:1451783:1343 170:1460431:1351 171:1469079:1359 172:1477727:1367 173:1
486375:1375 174:1495023:1383 175:1503671:1391 176:1512319:1399 177:1520967:1407 178:1529615:1415 179:1538263:1423 180:1546911:1431 181:1555559:1439 182:1564207:1447 183:1572855:1455 184:1581503:1463 185:1590151:1471 186:1598799:1479 187:1607447:1487 188:1616095:1495 189:1620419:1499 190:1629067:1507 191:1637715:1515 192:1646363:1523 193:1655011:1531 194:1663659:1539 195:167230
7:1547 196:1680955:1555 197:1689603:1563 198:1698251:1571 199:1706899:1579 200:1715547:1587 201:1724195:1595 202:1732843:1603 203:1741491:1611 204:1750139:1619 205:1758787:1627 206:1767435:1635 207:1776083:1643 208:1784731:1651 209:1793379:1659 210:1802027:1667 211:1810675:1675 212:1819323:1683 213:1827971:1691 214:1836619:1699 215:1845267:1707 216:1853915:1715 217:1862563:172
3 218:1871211:1731 219:1879859:1739 220:1888507:1747 221:1897155:1755 222:1905803:1763 223:1914451:1771 224:1923099:1779 225:1931747:1787 226:1940395:1795 227:1944719:1799 228:1953367:1807 229:1962015:1815 230:1970663:1823 231:1979311:1831 232:1987959:1839 233:1996607:1847 234:2005255:1855 235:2013903:1863 236:2022551:1871 237:2031199:1879 238:2039847:1887 239:2048495:1895 240
:2057143:1903 241:2065791:1911 242:2074439:1919 243:2083087:1927 244:2091735:1935 245:2100383:1943 246:2109031:1951 247:2117679:1959 248:2126327:1967 249:2134975:1975 250:2143623:1983 251:2152271:1991 252:2160919:1999 253:2169567:2007 254:2178215:2015 255:2186863:2023 256:2195511:2031 257:2204159:2039 258:2212807:2047 259:2221455:2055 260:2230103:2063 261:2238751:2071 262:2247
399:2079 263:2256047:2087 264:2264695:2095 265:2270100:2100 266:2278748:2108 267:2287396:2116 268:2296044:2124 269:2304692:2132 270:2313340:2140 271:2321988:2148 272:2330636:2156 273:2339284:2164 274:2347932:2172 275:2356580:2180 276:2365228:2188 277:2373876:2196 278:2382524:2204 279:2391172:2212 280:2399820:2220 281:2408468:2228 282:2417116:2236 283:2425764:2244 284:2434412:2
252 285:2443060:2260 286:2451708:2268 287:2460356:2276 288:2469004:2284 289:2477652:2292 290:2486300:2300 291:2494948:2308 292:2503596:2316 293:2512244:2324 294:2520892:2332 295:2529540:2340 296:2538188:2348 297:2546836:2356 298:2555484:2364 299:2564132:2372 300:2572780:2380 301:2581428:2388 302:2590076:2396 303:2594400:2400 304:2603048:2408 305:2611696:2416 306:2620344:2424 3
07:2628992:2432 308:2637640:2440 309:2646288:2448 310:2654936:2456 311:2663584:2464 312:2672232:2472 313:2680880:2480 314:2689528:2488 315:2698176:2496 316:2706824:2504 317:2715472:2512 318:2724120:2520 319:2732768:2528 320:2741416:2536 321:2750064:2544 322:2758712:2552 323:2767360:2560 324:2776008:2568 325:2784656:2576 326:2793304:2584 327:2801952:2592 328:2810600:2600 329:28
19248:2608 330:2827896:2616 331:2836544:2624 332:2845192:2632 333:2853840:2640 334:2862488:2648 335:2871136:2656 336:2879784:2664 337:2888432:2672 338:2897080:2680 339:2905728:2688 340:2914376:2696 341:2919781:2701 342:2928429:2709 343:2937077:2717 344:2945725:2725 345:2954373:2733 346:2963021:2741 347:2971669:2749 348:2980317:2757 349:2988965:2765 350:2997613:2773 351:3006261
:2781 352:3014909:2789 353:3023557:2797 354:3032205:2805 355:3040853:2813 356:3049501:2821 357:3058149:2829 358:3066797:2837 359:3075445:2845 360:3084093:2853 361:3092741:2861 362:3101389:2869 363:3110037:2877 364:3118685:2885 365:3127333:2893 366:3135981:2901 367:3144629:2909 368:3153277:2917 369:3161925:2925 370:3170573:2933 371:3179221:2941 372:3187869:2949 373:3196517:2957
 374:3205165:2965 375:3213813:2973 376:3222461:2981 377:3231109:2989 378:3239757:2997 379:3248405:3005 380:3257053:3013 381:3265701:3021 382:3274349:3029 383:3282997:3037 384:3291645:3045 385:3300293:3053 386:3308941:3061 387:3317589:3069 388:3326237:3077 389:3334885:3085 390:3343533:3093 391:3352181:3101 392:3360829:3109 393:3369477:3117 394:3378125:3125 395:3386773:3133 396:
3395421:3141 397:3404069:3149 398:3412717:3157 399:3421365:3165 400:3430013:3173 401:3438661:3181 402:3447309:3189 403:3455957:3197 404:3464605:3205 405:3473253:3213 406:3481901:3221 407:3490549:3229 408:3499197:3237 409:3507845:3245 410:3516493:3253 411:3525141:3261 412:3533789:3269 413:3542437:3277 414:3551085:3285 415:3559733:3293 416:3568381:3301 417:3577029:3309 418:35856
77:3317 419:3594325:3325 420:3602973:3333 421:3611621:3341 422:3620269:3349 423:3628917:3357 424:3637565:3365 425:3646213:3373 426:3654861:3381 427:3663509:3389 428:3672157:3397 429:3680805:3405 430:3689453:3413 431:3698101:3421 432:3706749:3429 433:3715397:3437 434:3724045:3445 435:3732693:3453 436:3741341:3461 437:3749989:3469 438:3758637:3477 439:3767285:3485 440:3775933:34
93 441:3784581:3501 442:3793229:3509 443:3801877:3517 444:3810525:3525 445:3819173:3533 446:3827821:3541 447:3836469:3549 448:3845117:3557 449:3853765:3565 450:3862413:3573 451:3871061:3581 452:3879709:3589 453:3888357:3597 454:3897005:3605 455:3905653:3613 456:3914301:3621 457:3922949:3629 458:3931597:3637 459:3940245:3645 460:3948893:3653 461:3957541:3661 462:3966189:3669 46
3:3974837:3677 464:3983485:3685 465:3992133:3693 466:4000781:3701 467:4009429:3709 468:4018077:3717 469:4026725:3725 470:4035373:3733 471:4044021:3741 472:4052669:3749 473:4061317:3757 474:4069965:3765 475:4078613:3773 476:4087261:3781 477:4095909:3789 478:4104557:3797 479:4113205:3805 480:4121853:3813 481:4130501:3821 482:4139149:3829 483:4147797:3837 484:4156445:3845 485:416
5093:3853 486:4173741:3861 487:4182389:3869 488:4191037:3877 489:4199685:3885 490:4208333:3893 491:4216981:3901 492:4225629:3909 493:4234277:3917 494:4242925:3925 495:4251573:3933 496:4260221:3941 497:4268869:3949 498:4277517:3957 499:4286165:3965 500:4294813:3973 501:4303461:3981 502:4312109:3989 503:4320757:3997 504:4329405:4005 505:4338053:4013 506:4346701:4021 507:4355349:
4029 508:4363997:4037 509:4372645:4045 510:4381293:4053 511:4389941:4061 512:4398589:4069 513:4407237:4077 514:4415885:4085 515:4424533:4093 516:4433181:4101 517:4441829:4109 518:4450477:4117 519:4459125:4125 520:4467773:4133 521:4476421:4141 522:4485069:4149 523:4493717:4157 524:4502365:4165 525:4511013:4173 526:4519661:4181 527:4528309:4189 528:4536957:4197 529:4545605:4205 
530:4554253:4213 531:4562901:4221 532:4571549:4229 533:4580197:4237 534:4588845:4245 535:4597493:4253 536:4606141:4261 537:4614789:4269 538:4623437:4277 539:4632085:4285 540:4640733:4293 541:4649381:4301 542:4658029:4309 543:4666677:4317 544:4675325:4325 545:4683973:4333 546:4692621:4341 547:4701269:4349 548:4709917:4357 549:4718565:4365 550:4727213:4373 551:4735861:4381 552:4
744509:4389 553:4753157:4397 554:4761805:4405 555:4770453:4413 556:4779101:4421 557:4787749:4429 4437
INFO       - 09:37:55 - data_handler       [0.8/1.7 8.5/8.5 0.3Gb]   generated 558 row chunks based on time and DDID
INFO       - 09:37:55 - data_handler       [0.8/1.7 8.5/8.5 0.3Gb]   row chunks yield 558 potential tiles
INFO       - 09:37:56 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb]   coarsening this to 56 tiles (max 10 chunks per tile, based on 16/10 requested)
INFO       - 09:38:01 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb] the MS appears to have a properly formed BITFLAG column
INFO       - 09:38:01 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb]   bitflag 'legacy' already exists, will not auto-fill
INFO       - 09:38:01 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb]   BITFLAG column defines the following flagsets: legacy:1 cubical:2
INFO       - 09:38:01 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb]   will exclude flagset cubical
INFO       - 09:38:01 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb]   applying BITFLAG mask 1 to input data
INFO       - 09:38:01 - data_handler       [1.0/1.7 8.6/8.6 0.3Gb]   will save output flags into BITFLAG 'cubical' (2), and into FLAG/FLAG_ROW
INFO       - 09:38:01 - main               [0.9/1.7 8.8/8.8 0.3Gb] waiting for I/O on tile 0/56
INFO       - 09:38:01 - main               [io] [0.9/0.9 8.8/8.8 0.3Gb] loading tile 0/56
INFO       - 09:38:01 - ms_tile            [io] [0.9/0.9 8.8/8.8 0.3Gb] tile 0/56: reading MS rows 0~86479
INFO       - 09:38:01 - data_handler       [io] [0.9/0.9 8.8/8.8 0.3Gb] reading CORRECTED_DATA
INFO       - 09:38:42 - ms_tile            [io] [24.5/24.5 30.0/30.0 0.3Gb]   applying solvable baseline cutoff deselects 7600 rows
INFO       - 09:38:42 - ms_tile            [io] [24.5/24.5 30.0/30.0 0.3Gb]   8.79% visibilities deselected via specificed subset and/or baseline cutoffs
INFO       - 09:38:43 - data_handler       [io] [24.8/24.8 30.0/30.0 0.3Gb] reading BITFLAG
WARNING    - 09:39:32 - ms_tile            [io] [33.1/39.0 36.5/42.5 0.3Gb]   0.06% input visibilities flagged as invalid (0/inf/nan)
INFO       - 09:39:43 - ms_tile            [io] [34.4/39.0 37.9/42.5 0.3Gb]   89.05% input visibilities flagged and/or deselected
INFO       - 09:40:01 - data_handler       [io] [47.6/52.8 61.7/61.7 18.8Gb] reading MODEL_DATA
INFO       - 09:46:56 - main               [io] [10.7/76.1 14.1/82.8 29.4Gb] I/O job(s) complete
INFO       - 09:46:56 - main               [0.9/1.7 8.8/8.8 29.4Gb] submitting solver jobs for tile 0/56
INFO       - 09:46:56 - main               [io] [10.7/76.1 14.1/82.8 29.4Gb] loading tile 1/56
INFO       - 09:46:56 - ms_tile            [io] [10.7/76.1 14.1/82.8 29.4Gb] tile 1/56: reading MS rows 86480~172959
INFO       - 09:46:56 - data_handler       [io] [10.7/76.1 14.1/82.8 29.4Gb] reading CORRECTED_DATA
WARNING    - 09:48:05 - solver             [x09] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T8F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:48:14 - solver             [x05] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T4F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:48:14 - solver             [x04] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T3F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:48:16 - solver             [x08] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T7F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:48:59 - solver             [x10] [13.0/14.9 16.3/40.4 29.4Gb] 'slope' D0T9F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:50:17 - solver             [x02] [13.0/14.9 16.2/40.9 29.4Gb] 'slope' D0T1F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:50:20 - solver             [x03] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T2F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
INFO       - 09:50:24 - madmax             [x10] [16.0/16.0 20.0/40.4 29.4Gb] D0T9F0 iter 1 (slope): antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 09:50:26 - madmax             [x10] [16.1/16.1 20.0/40.4 29.4Gb] 133 baselines flagged on mad residuals (--madmax-flag-ant 1)
WARNING    - 09:50:26 - solver             [x01] [13.0/14.9 16.2/40.9 29.4Gb] 'slope' D0T0F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:50:31 - solver             [x06] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T5F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 09:50:37 - solver             [x07] [13.0/14.9 16.3/41.0 29.4Gb] 'slope' D0T6F0 Stations 0, 1, 2, 5, 6, 8, 10, 13, 18, 19, 21, 22, 26, 29, 31, 34, 35, 36, 40 (19/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
INFO       - 09:51:41 - madmax             [x10] [16.0/16.5 20.0/40.4 29.4Gb] D0T9F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 09:51:44 - madmax             [x10] [16.1/16.5 20.0/40.4 29.4Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 09:51:58 - ms_tile            [io] [31.9/76.1 35.2/82.8 29.4Gb]   applying solvable baseline cutoff deselects 7600 rows
INFO       - 09:51:58 - ms_tile            [io] [31.9/76.1 35.2/82.8 29.4Gb]   8.79% visibilities deselected via specificed subset and/or baseline cutoffs
INFO       - 09:51:59 - data_handler       [io] [31.9/76.1 35.2/82.8 29.4Gb] reading BITFLAG
INFO       - 09:55:46 - solver             [x09] [15.5/19.4 18.9/41.0 30.5Gb] D0T8F0 has 624204 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 1612 visibilities
INFO       - 09:56:07 - solver             [x08] [15.6/19.4 18.9/41.0 31.5Gb] D0T7F0 has 625286 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 2694 visibilities
INFO       - 09:56:40 - solver             [x05] [15.6/19.4 18.9/41.0 33.7Gb] D0T4F0 has 625970 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 3378 visibilities
INFO       - 09:56:40 - solver             [x04] [15.6/19.4 18.9/41.0 33.7Gb] D0T3F0 has 624654 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 2062 visibilities
WARNING    - 09:57:49 - ms_tile            [io] [38.5/76.1 41.8/82.8 36.4Gb]   1.90% input visibilities flagged as invalid (0/inf/nan)
INFO       - 09:58:06 - ms_tile            [io] [39.8/76.1 43.1/82.8 36.4Gb]   10.68% input visibilities flagged and/or deselected
INFO       - 09:58:28 - data_handler       [io] [53.0/76.1 67.0/82.8 55.7Gb] reading MODEL_DATA
INFO       - 09:58:48 - solver             [x02] [15.5/19.4 18.9/40.9 56.2Gb] D0T1F0 has 625344 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 2752 visibilities
INFO       - 09:59:06 - solver             [x01] [15.5/19.4 18.9/40.9 57.6Gb] D0T0F0 has 625478 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 2886 visibilities
INFO       - 09:59:17 - solver             [x07] [15.5/19.4 18.9/41.0 57.8Gb] D0T6F0 has 626658 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 4066 visibilities
INFO       - 09:59:34 - solver             [x10] [15.6/19.7 19.5/40.4 60.8Gb] D0T9F0 has 4822804 (6.96%) new data flags: gain flags LOWSNR:32768(2.17%) (2.17% total), MadMax took out 12900116 visibilities
INFO       - 09:59:39 - solver             [x03] [15.5/19.4 18.9/41.0 62.1Gb] D0T2F0 has 624812 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 2220 visibilities
INFO       - 09:59:41 - solver             [x06] [15.5/19.4 18.9/41.0 62.1Gb] D0T5F0 has 627240 (0.90%) new data flags: gain flags LOWSNR:622592(41.30%) (41.30% total), MadMax took out 4648 visibilities
INFO       - 10:00:10 - main               [1.0/1.7 9.1/9.2 64.8Gb] finished processing tile 0/56
INFO       - 10:00:10 - main               [1.0/1.7 9.1/9.2 64.8Gb] waiting for I/O on tile 1/56
INFO       - 10:03:08 - main               [io] [16.1/81.5 19.4/88.1 75.3Gb] I/O job(s) complete
INFO       - 10:03:08 - main               [1.0/1.7 9.1/9.2 75.3Gb] submitting solver jobs for tile 1/56
INFO       - 10:03:08 - main               [io] [16.1/81.5 19.4/88.1 75.3Gb] saving tile 0/56
INFO       - 10:03:08 - ms_tile            [io] [16.1/81.5 19.4/88.1 75.3Gb] tile 0/56: saving MS rows 0~86479
INFO       - 10:03:08 - ms_tile            [io] [16.1/81.5 30.0/88.1 75.3Gb]   writing SC_DATA column
INFO       - 10:03:37 - ms_tile            [io] [29.3/81.5 32.6/88.1 75.3Gb]   47.96% visibilities flagged by solver: saving to BITFLAG and FLAG columns
INFO       - 10:03:55 - ms_tile            [io] [29.3/81.5 32.6/88.1 75.3Gb]   updated BITFLAG column (89.51% visibilities flagged)
INFO       - 10:03:56 - ms_tile            [io] [29.3/81.5 32.6/88.1 75.3Gb]   updated BITFLAG_ROW column (66.99% rows flagged)
INFO       - 10:04:02 - ms_tile            [io] [24.0/81.5 27.3/88.1 75.3Gb]   updated FLAG column (89.51% visibilities flagged)
INFO       - 10:04:03 - ms_tile            [io] [24.0/81.5 27.3/88.1 75.3Gb]   updated FLAG_ROW column (66.99% rows flagged)
INFO       - 10:04:21 - main               [io] [11.8/81.5 14.7/88.1 30.0Gb] loading tile 2/56
INFO       - 10:04:21 - ms_tile            [io] [11.8/81.5 14.7/88.1 30.0Gb] tile 2/56: reading MS rows 172960~259439
INFO       - 10:04:21 - data_handler       [io] [11.8/81.5 14.7/88.1 30.0Gb] reading CORRECTED_DATA
WARNING    - 10:04:44 - solver             [x12] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T11F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:44 - solver             [x15] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T14F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:44 - solver             [x13] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T12F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:45 - solver             [x16] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T15F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:46 - solver             [x11] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T10F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:47 - solver             [x14] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T13F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:44 - solver             [x12] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T11F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:44 - solver             [x15] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T14F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:44 - solver             [x13] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T12F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:45 - solver             [x16] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T15F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:46 - solver             [x11] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T10F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:04:47 - solver             [x14] [13.0/14.9 16.3/40.1 30.0Gb] 'slope' D0T13F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
INFO       - 10:05:53 - madmax             [x16] [16.1/16.1 20.0/40.1 30.0Gb] D0T15F0 iter 1 (slope): antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:05:53 - madmax             [x13] [16.1/16.1 20.0/40.1 30.0Gb] D0T12F0 iter 1 (slope): antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:05:55 - madmax             [x15] [16.1/16.1 20.0/40.1 30.0Gb] D0T14F0 iter 1 (slope): antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:05:55 - madmax             [x16] [16.1/16.1 20.0/40.1 30.0Gb] D0T15F0 iter 1 (slope): antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:05:55 - madmax             [x16] [16.1/16.1 20.0/40.1 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:05:55 - madmax             [x13] [16.1/16.1 20.0/40.1 30.0Gb] D0T12F0 iter 1 (slope): antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:05:55 - madmax             [x13] [16.1/16.1 20.0/40.1 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:05:55 - madmax             [x12] [16.1/16.1 20.0/40.1 30.0Gb] D0T11F0 iter 1 (slope): antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:05:57 - madmax             [x15] [16.1/16.1 20.0/40.1 30.0Gb] D0T14F0 iter 1 (slope): antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:05:57 - madmax             [x15] [16.1/16.1 20.0/40.1 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:05:58 - madmax             [x12] [16.1/16.1 20.0/40.1 30.0Gb] D0T11F0 iter 1 (slope): antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:05:58 - madmax             [x12] [16.1/16.1 20.0/40.1 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:05:58 - madmax             [x11] [16.0/16.0 20.0/40.1 30.0Gb] D0T10F0 iter 1 (slope): antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:05:58 - madmax             [x14] [16.0/16.0 20.0/40.1 30.0Gb] D0T13F0 iter 1 (slope): antennas m001,m002,m003,m029 have mad residuals, refer to Mad Max plots
INFO       - 10:06:00 - madmax             [x04] [15.1/19.4 19.0/42.9 30.0Gb] D0T19F0 initial: antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:06:01 - madmax             [x11] [16.1/16.1 20.0/40.1 30.0Gb] D0T10F0 iter 1 (slope): antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:06:01 - madmax             [x11] [16.1/16.1 20.0/40.1 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:06:01 - madmax             [x14] [16.1/16.1 20.0/40.1 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:06:02 - madmax             [x04] [15.1/19.4 19.0/42.9 30.0Gb] D0T19F0 initial: antennas m026,m029 have mad residuals, refer to Mad Max plots
INFO       - 10:06:02 - madmax             [x04] [15.1/19.4 19.0/42.9 30.0Gb] 217 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:06:06 - madmax             [x09] [15.1/19.4 19.0/42.9 30.0Gb] D0T16F0 initial: antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:06:06 - madmax             [x05] [15.1/19.4 19.0/42.9 30.0Gb] D0T18F0 initial: antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:06:08 - madmax             [x05] [15.1/19.4 19.0/42.9 30.0Gb] D0T18F0 initial: antennas m026,m029 have mad residuals, refer to Mad Max plots
INFO       - 10:06:08 - madmax             [x05] [15.1/19.4 19.0/42.9 30.0Gb] 217 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:06:09 - madmax             [x09] [15.1/19.4 19.0/42.9 30.0Gb] D0T16F0 initial: antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:06:09 - madmax             [x09] [15.1/19.4 19.0/42.9 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
WARNING    - 10:06:14 - solver             [x04] [14.1/19.4 18.0/42.9 30.0Gb] 'slope' D0T19F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:06:21 - solver             [x05] [14.1/19.4 18.0/42.9 30.0Gb] 'slope' D0T18F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
WARNING    - 10:06:21 - solver             [x09] [14.1/19.4 18.0/42.9 30.0Gb] 'slope' D0T16F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
INFO       - 10:06:23 - madmax             [x08] [15.1/19.4 19.0/42.9 30.0Gb] D0T17F0 initial: antennas m001,m002,m003 have mad residuals, refer to Mad Max plots
INFO       - 10:06:26 - madmax             [x08] [15.1/19.4 19.0/42.9 30.0Gb] D0T17F0 initial: antennas m029 have mad residuals, refer to Mad Max plots
INFO       - 10:06:26 - madmax             [x08] [15.1/19.4 19.0/42.9 30.0Gb] 176 baselines flagged on mad residuals (--madmax-flag-ant 1)
WARNING    - 10:06:38 - solver             [x08] [14.1/19.4 18.0/42.9 30.0Gb] 'slope' D0T17F0 Stations 8 (1/46) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data
INFO       - 10:07:18 - madmax             [x16] [16.1/16.1 20.0/40.1 30.0Gb] D0T15F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:07:18 - madmax             [x13] [16.1/16.1 20.0/40.1 30.0Gb] D0T12F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:07:19 - madmax             [x12] [16.1/16.1 20.0/40.1 30.0Gb] D0T11F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:07:20 - madmax             [x15] [16.1/16.1 20.0/40.1 30.0Gb] D0T14F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:07:20 - madmax             [x16] [16.1/16.1 20.0/40.1 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:07:21 - madmax             [x13] [16.1/16.1 20.0/40.1 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:07:21 - madmax             [x12] [16.1/16.1 20.0/40.1 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:07:22 - madmax             [x11] [16.0/16.1 20.0/40.1 30.0Gb] D0T10F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:07:22 - madmax             [x14] [16.0/16.1 20.0/40.1 30.0Gb] D0T13F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:07:23 - madmax             [x15] [16.1/16.1 20.0/40.1 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:07:25 - madmax             [x11] [16.1/16.1 20.0/40.1 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:07:25 - madmax             [x14] [16.1/16.1 20.0/40.1 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:08:17 - madmax             [x04] [16.1/19.4 20.0/42.9 30.0Gb] D0T19F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:08:19 - madmax             [x04] [16.1/19.4 20.1/42.9 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:08:23 - madmax             [x09] [16.2/19.4 20.1/42.9 30.0Gb] D0T16F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:08:25 - madmax             [x09] [16.2/19.4 20.1/42.9 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:08:27 - madmax             [x05] [16.1/19.4 20.0/42.9 30.0Gb] D0T18F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:08:30 - madmax             [x05] [16.1/19.4 20.1/42.9 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:08:35 - madmax             [x08] [16.2/19.4 20.1/42.9 30.0Gb] D0T17F0 iter 5 (slope): antennas m006 have mad residuals, refer to Mad Max plots
INFO       - 10:08:37 - madmax             [x08] [16.2/19.4 20.1/42.9 30.0Gb] 45 baselines flagged on mad residuals (--madmax-flag-ant 1)
INFO       - 10:08:55 - ms_tile            [io] [32.9/81.5 35.9/88.1 30.0Gb]   applying solvable baseline cutoff deselects 7586 rows
INFO       - 10:08:55 - ms_tile            [io] [32.9/81.5 35.9/88.1 30.0Gb]   8.77% visibilities deselected via specificed subset and/or baseline cutoffs
INFO       - 10:08:56 - data_handler       [io] [32.9/81.5 35.9/88.1 30.0Gb] reading BITFLAG
WARNING    - 10:13:59 - solver             [x15] [15.1/18.2 19.0/40.1 30.0Gb] D0T14F0 (end solve) slope: 30 iters, conv 0.00%, g/fl 2.17% [LOWSNR:32768(2.17%) MISSING:32768(2.17%)], d/fl 2.17%, PGE 0.00374, stall 99.13%, chi^2 296.2 -> 505.4 (524.8), noise 1.02e+02 -> 0.735 Shows signs of divergence. If you see this message often you may have significant RFI present in your data or your solution intervals are too short.
WARNING    - 10:14:08 - solver             [x11] [15.0/18.1 19.0/40.1 30.0Gb] D0T10F0 (end solve) slope: 30 iters, conv 0.00%, g/fl 2.17% [LOWSNR:32768(2.17%) MISSING:32768(2.17%)], d/fl 2.17%, PGE 0.00374, stall 99.88%, chi^2 295.6 -> 512.8 (533.4), noise 1.02e+02 -> 0.728 Shows signs of divergence. If you see this message often you may have significant RFI present in your data or your solution intervals are too short.
WARNING    - 10:14:32 - solver             [x13] [15.1/18.2 19.0/40.1 30.0Gb] D0T12F0 (end solve) slope: 35 iters, conv 0.00%, g/fl 2.17% [LOWSNR:32768(2.17%) MISSING:32768(2.17%)], d/fl 2.17%, PGE 0.00374, stall 99.28%, chi^2 297.1 -> 508.7 (528.3), noise 1.03e+02 -> 0.741 Shows signs of divergence. If you see this message often you may have significant RFI present in your data or your solution intervals are too short.
WARNING    - 10:14:44 - solver             [x16] [15.1/18.2 19.0/40.1 30.0Gb] D0T15F0 (end solve) slope: 35 iters, conv 0.00%, g/fl 2.17% [LOWSNR:32768(2.17%) MISSING:32768(2.17%)], d/fl 2.17%, PGE 0.00374, stall 99.35%, chi^2 295.8 -> 506.0 (525.7), noise 1e+02 -> 0.722 Shows signs of divergence. If you see this message often you may have significant RFI present in your data or your solution intervals are too short.
WARNING    - 10:15:07 - solver             [x12] [15.1/18.2 19.0/40.1 30.0Gb] D0T11F0 (end solve) slope: 40 iters, conv 10.99%, g/fl 2.17% [LOWSNR:32768(2.17%) MISSING:32768(2.17%)], d/fl 2.17%, PGE 0.00374, stall 99.91%, chi^2 294.8 -> 506.5 (527.0), noise 1.03e+02 -> 0.736 Shows signs of divergence. If you see this message often you may have significant RFI present in your data or your solution intervals are too short.
WARNING    - 10:15:25 - solver             [x14] [15.0/18.1 19.0/40.1 30.0Gb] D0T13F0 (end solve) slope: 40 iters, conv 7.37%, g/fl 2.17% [LOWSNR:32768(2.17%) MISSING:32768(2.17%)], d/fl 2.17%, PGE 0.00374, stall 99.47%, chi^2 296.1 -> 506.7 (526.6), noise 1.04e+02 -> 0.742 Shows signs of divergence. If you see this message often you may have significant RFI present in your data or your solution intervals are too short.
INFO       - 10:16:56 - solver             [x15] [15.6/19.7 19.5/40.1 32.1Gb] D0T14F0 has 18428460 (26.58%) new data flags: gain flags LOWSNR:32768(2.17%) (2.17% total), MadMax took out 18526764 visibilities
...
INFO       - 08:18:33 - main               [8.8/8.8 14.8/14.8 0.3Gb]   summary stats for {chi2:.3f}:
            D0T0F0    D0T1F0    D0T2F0    D0T3F0    D0T4F0    D0T5F0    D0T6F0    D0T7F0    
  D0T0F0    0.217     0.242     0.211     0.192     0.190     0.191     0.238     0.214     
  D0T8F0    0.230     78.703    533.444   527.027   528.336   526.610   524.760   525.749   
  D0T16F0   524.000   515.901   452.874   460.638   453.048   525.303   627.855   455.312   
  D0T24F0   638.663   627.180   455.152   453.836   450.389   446.339   445.720   541.811   
  D0T32F0   599.330   612.013   614.444   624.563   543.390   530.873   575.875   478.741   
  D0T40F0   0.213     0.038     2.874     0.077     0.152     0.397     0.186     1.665     
  D0T48F0   157.110   479.859   420.056   477.816   478.764   476.771   475.890   475.770   
  D0T56F0   411.853   480.962   474.414   476.261   566.834   569.129   472.108   415.505   
  D0T64F0   473.101   487.280   469.874   466.421   463.135   463.682   463.143   42.220    
  D0T72F0   25.518    319.447   523.779   0.381     0.097     0.000     0.000     0.000     
  D0T80F0   0.311     0.323     1.911     5.513     9.931     22.809    27.778    137.119   
  D0T88F0   885.950   471.719   571.032   580.470   238.663   78.947    629.564   584.918   
  D0T96F0   581.753   579.752   582.809   582.540   575.650   585.408   581.049   421.592   
  D0T104F0  578.361   572.195   571.327   567.573   578.758   571.878   574.483   570.965   
  D0T112F0  489.292   0.400     0.392     0.397     0.404     0.409     0.390     0.390     
  D0T120F0  2.232     5.641     2.512     3.900     284.764   568.888   244.377   274.388   
  D0T128F0  829.129   648.372   529.880   860.358   867.767   872.620   412.414   70.881    
  D0T136F0  457.157   867.054   868.917   714.776   549.593   728.080   720.008   710.434   
  D0T144F0  724.639   526.577   456.814   533.570   537.613   533.157   462.576   0.278     
  D0T152F0  0.268     0.278     0.284     0.282     0.294     0.278     0.059     0.019     
  D0T160F0  1.340     4.752     382.232   625.367   626.542   626.468   619.591   525.053   
  D0T168F0  525.615   524.826   622.445   620.664   621.751   618.977   612.553   613.715   
  D0T176F0  617.955   617.955   615.138   620.239   403.472   11.688    617.350   607.605   
  D0T184F0  453.417   459.332   429.077   603.996   690.162   0.000     2.305     0.078     
  D0T192F0  0.194     0.191     0.200     0.232     0.044     0.000     0.000     0.000     
  D0T200F0  49.113    294.446   728.193   723.352   434.711   126.477   98.010    732.049   
  D0T208F0  438.414   164.659   163.434   362.164   724.304   699.105   686.349   706.172   
  D0T216F0  682.343   669.393   666.526   661.046   675.030   700.645   776.392   955.580   
  D0T224F0  970.267   777.431   972.947   0.000     0.000     0.000     0.000     0.000     
  D0T232F0  0.000     0.000     37.232    60.777    60.365    60.679    672.356   1703.782  
  D0T240F0  1726.190  1732.681  225.341   210.104   831.957   84.769    752.981   1682.021  
  D0T248F0  1663.406  1665.169  981.749   1208.133  1677.456  1687.007  1663.957  1611.467  
  D0T256F0  1642.577  1641.602  1664.099  1629.837  1629.641  663.216   642.477   646.210   
  D0T264F0  640.774   20.897    5.048     0.000     0.000     0.000     0.249     0.236     
  D0T272F0  1.537     21.318    21.419    17.730    716.247   1368.433  1292.137  1347.055  
  D0T280F0  1065.679  1070.194  1074.235  1473.278  323.947   886.912   865.997   858.107   
  D0T288F0  859.737   860.173   866.691   1075.281  1061.981  1059.142  1060.283  1062.133  
  D0T296F0  1366.262  1490.014  1467.363  1481.633  232.644   220.210   66.458    0.177     
  D0T304F0  0.362     0.373     0.367     0.366     0.359     0.349     0.923     6.285     
  D0T312F0  6.241     5.706     286.413   503.985   503.225   622.404   1021.631  404.399   
  D0T320F0  43.488    79.201    1206.929  1027.894  1023.643  985.975   77.490    24.073    
  D0T328F0  24.351    874.409   1298.992  990.489   1286.027  581.183   1111.576  1122.861  
  D0T336F0  1073.322  1170.406  1167.740  1130.422  1140.266  0.202     0.202     0.189     
  D0T344F0  0.170     0.173     0.173     0.177     0.177     0.176     0.171     0.165     
  D0T352F0  0.140     0.145     108.668   1314.528  1300.383  1295.495  1297.747  1303.062  
  D0T360F0  788.761   966.567   1288.266  1289.970  1295.167  1286.816  1280.934  1270.571  
  D0T368F0  1251.333  1243.224  1248.159  1247.151  1250.645  1251.151  1216.851  1239.879  
  D0T376F0  1256.861  1263.626  1260.381  1223.125  1239.269  455.450   291.740   1236.799  
  D0T384F0  1227.993  0.217     0.214     0.208     0.216     0.211     0.211     0.214     
  D0T392F0  0.213     0.211     98.146    1245.507  1254.113  1246.672  1259.816  1250.974  
  D0T400F0  1260.564  1247.043  1249.568  1258.566  1250.206  1229.893  1212.598  1181.057  
  D0T408F0  1269.972  1272.680  1279.462  1099.914  512.805   116.416   449.371   1102.630  
  D0T416F0  1085.035  1245.264  1250.719  1083.555  890.498   1420.256  1412.828  1238.433  
  D0T424F0  1258.078  0.196     0.175     0.159     0.158     0.022     0.144     0.169     
  D0T432F0  0.166     0.166     96.246    1278.138  1258.893  1420.303  1455.356  1249.779  
  D0T440F0  1244.324  1234.036  1227.035  1205.366  1197.014  1212.087  1179.252  1176.032  
  D0T448F0  1157.755  1155.215  1166.857  1154.748  1163.686  1153.767  1168.120  1162.912  
  D0T456F0  1165.232  1166.960  1173.006  1165.160  1081.959  1150.676  1157.002  1155.190  
  D0T464F0  1684.472  0.000     0.000     0.000     0.000     0.000     0.000     0.000     
  D0T472F0  0.000     0.000     164.296   1403.284  290.802   848.998   1863.084  1850.462  
  D0T480F0  1823.281  1817.963  1614.626  1128.882  1106.348  1097.507  1104.781  1088.438  
  D0T488F0  1086.522  1090.387  1081.407  1072.592  1065.465  1060.641  1066.768  1080.274  
  D0T496F0  1069.751  1055.078  1065.918  1088.745  1091.006  1072.611  1062.042  1127.514  
  D0T504F0  1143.597  0.113     0.112     0.117     0.119     0.111     0.115     0.109     
  D0T512F0  0.112     0.113     97.755    1116.356  1120.214  1124.664  1126.707  1173.769  
  D0T520F0  1169.247  1169.911  1165.627  1004.251  269.987   446.243   614.380   71.446    
  D0T528F0  71.415    238.221   501.644   586.850   1203.632  1213.678  1197.285  1145.350  
  D0T536F0  1155.413  1131.432  1164.536  1170.426  1161.629  823.031   815.802   795.086   
  D0T544F0  793.918   0.207     0.188     0.067     0.008     0.008     0.005     0.167     
  D0T552F0  0.217     0.223     0.214     0.212     0.209     0.197
bennahugo commented 4 years ago

Here is delaycal+gaincal at 30s intervals (B transferred from another calibrator): image after solving the residuals at 4s intervals with CASA K+G solutions the following -30dB (abs peak positive/abs peak negative) map is obtained image

JSKenyon commented 4 years ago

If you are feeling brave, you can give the issue-326-jsk branch a shot @bennahugo . It is still a WIP but should help in the case of bad delay errors. Produces initial guesses by taking the FFT of the data along the frequency axis.