search

ratt-ru / CubiCal

A fast radio interferometric calibration suite.
GNU General Public License v2.0
18 stars 13 forks source link

cubical dumping core midway through a run due to plotting errors #320

Open viralp opened 5 years ago

viralp commented 5 years ago

Hi

I ran crystalball for one direction and then used model list with cubical as follows. There was no error in cubical, but then wsclean is failing for a given model data column. 

gocubical  --data-ms ../../../msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms --data-column DATA  --out-column DD_DATA --weight-column WEIGHT --model-ddes auto  --g-solvable True --g-type complex-2x2 --flags-reinit-bitflags True --g-save-to ampphase-1.parmdb --flags-auto-init 'legacy' --sol-jones G,dE  --sol-min-bl 50 --g-clip-high 1.5 --g-clip-low 0.5 --g-time-int 1 --g-freq-int 1 --model-list "MODEL_DATA+-bright_source_column:bright_source_column" --out-name ampphase_1 --data-freq-chunk 100 --data-time-chunk 100 --dist-ncpu 30 --out-mode sc --madmax-enable 1 --madmax-plot 'show'  --madmax-threshold [0,10] --madmax-estimate corr --out-casa-gaintables True --sol-term-iters 50,50 --dist-max-chunks 1 --de-type complex-2x2 --de-clip-high 1.5 --de-clip-low 0.5 --de-solvable True --de-time-int 100 --de-freq-int 100 --de-save-to ddjone_cal_1.paramdb

wsclean -name A2384_wsclean_cubical -mem 100 -weight briggs 0.5 -super-weight 1.0 -weighting-rank-filter-size 16 -taper-gaussian 0 -size 6000 6000 -scale 1.5asec -channels-out 3 -grid-mode kb -kernel-size 7 -oversampling 63 -pol I -intervals-out 1 -data-column DD_DATA -niter 1000000 -auto-threshold 0.5 -auto-mask 7.0 -gain 0.1 -mgain 0.99 -join-channels -multiscale-scale-bias 0.6 -multiscale-scales 10,20,30 -fit-spectral-pol 2 -fit-beam -elliptical-beam -padding 1.3 ../../../msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms

Getting the following error in wsclean

=== IMAGING TABLE ===
       # Pol Ch JG ²G In Freq(MHz)
| Independent group:
+-+-J- 0  I   0  0  0  0  856-1141 (273)
  |
  +-J- 1  I   1  0  1  0  1141-1426 (273)
  |
  +-J- 2  I   2  0  2  0  1426-1712 (273)

Reordering ../../../msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms into 3 x 1 parts.
Reordering: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
+ + + + + + + + + + + + + + + + + + +
+ An exception occured:
+ >>> Table DataManager error: Invalid operation: TSM: no array in row 409980 of column DD_DATA in /vault-ike/parekh/A2384_MeerKAT/stimela_pipeline/msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms/table.f44
+ + + + + + + + + + + + + + + + + + +
IanHeywood commented 5 years ago

Can you please post the full cubical log?

viralp commented 5 years ago

Hi Ian,

Here is the full log of cubical - INFO - 10:51:29 - cc [0.2/0.2 2.4/2.4 115.7Gb] set global console verbosity level 0 INFO - 10:51:29 - main [0.2/0.2 2.4/2.4 115.7Gb] started /home/parekh/stimela/bin/gocubical --data-ms ../../../msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms --data-column DATA --out-column DD_DATA --weight-column WEIGHT --model-ddes auto --g-solvable True --g-type complex-2x2 --flags-reinit-bitflags True --g-save-to ampphase-1.parmdb --flags-auto-init legacy --sol-jones G,dE --sol-min-bl 50 --g-clip-high 1.5 --g-clip-low 0.5 --g-time-int 1 --g-freq-int 1 --model-list MODEL_DATA+-bright_source_column:bright_source_column --out-name ampphase_1 --data-freq-chunk 100 --data-time-chunk 100 --dist-ncpu 30 --out-mode sc --madmax-enable 1 --madmax-plot show --madmax-threshold [0,10] --madmax-estimate corr --out-casa-gaintables True --sol-term-iters 50,50 --dist-max-chunks 1 --de-type complex-2x2 --de-clip-high 1.5 --de-clip-low 0.5 --de-solvable True --de-time-int 100 --de-freq-int 100 --de-save-to ddjone_cal_1.paramdb INFO - 10:51:29 - main [0.2/0.2 2.4/2.4 115.7Gb] using output basename cubical.cc-out/ampphase_1 INFO - 10:51:29 - main [0.2/0.2 2.4/2.4 115.7Gb] creating new output directory cubical.cc-out INFO - 10:51:29 - shm_utils [0.2/0.2 2.4/2.4 115.7Gb] reaping 1 shared memory objects associated with 1 dead cubical processes INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] Selected Options: INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [data] Visibility data options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - ms ................................................ = ../../../msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - column ............................................ = DATA INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - time-chunk ........................................ = 100 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - freq-chunk ........................................ = 100 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - rebin-time ........................................ = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - rebin-freq ........................................ = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - chunk-by .......................................... = SCAN_NUMBER INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - chunk-by-jump ..................................... = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - single-chunk ...................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - single-tile ....................................... = -1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [sel] Data selection options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - field ............................................. = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - ddid .............................................. = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - taql .............................................. = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - chan .............................................. = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - diag .............................................. = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [out] Options for output products INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - dir ............................................... = cubical INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - name .............................................. = cubical.cc-out/ampphase_1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - overwrite ......................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - backup ............................................ = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - mode .............................................. = sc INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - apply-solver-flags ................................ = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - column ............................................ = DD_DATA INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - derotate .......................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - model-column ...................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - weight-column ..................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - reinit-column ..................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - subtract-model .................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - subtract-dirs ..................................... = : INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - plots ............................................. = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - casa-gaintables ................................... = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [model] Calibration model options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - list .............................................. = MODEL_DATA+-bright_source_column:bright_source_column INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - ddes .............................................. = auto INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - beam-pattern ...................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - beam-l-axis ....................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - beam-m-axis ....................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - feed-rotate ....................................... = auto INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - pa-rotate ......................................... = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [montblanc] Montblanc simulation options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - device-type ....................................... = CPU INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - dtype ............................................. = float INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - mem-budget ........................................ = 1024 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - verbosity ......................................... = WARNING INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - threads ........................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - pa-rotate ......................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [weight] Weighting options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - column ............................................ = WEIGHT INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - fill-offdiag ...................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - legacy-v1-2 ....................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [flags] General flagging options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - apply ............................................. = -cubical INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - auto-init ......................................... = legacy INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - save .............................................. = cubical INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - save-legacy ....................................... = auto INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - reinit-bitflags ................................... = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - warn-thr .......................................... = 0.3 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - see-no-evil ....................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [degridding] Options for the degridder. Only in use when predicting from DicoModels using DDFacet INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - OverS ............................................. = 11 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - Support ........................................... = 7 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - Nw ................................................ = 100 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - wmax .............................................. = 0.0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - Padding ........................................... = 1.7 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - NDegridBand ....................................... = 16 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - MaxFacetSize ...................................... = 0.25 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - MinNFacetPerAxis .................................. = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - NProcess .......................................... = 8 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [postmortem] Options for "postmortem" flagging based on solution statistics INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - enable ............................................ = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - tf-chisq-median ................................... = 1.2 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - tf-np-median ...................................... = 0.5 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - time-density ...................................... = 0.5 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - chan-density ...................................... = 0.5 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - ddid-density ...................................... = 0.5 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [madmax] Options for the "Mad Max" flagger INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - enable ............................................ = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - residuals ......................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - estimate .......................................... = corr INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - diag .............................................. = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - offdiag ........................................... = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - threshold ......................................... = [0, 10] INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - global-threshold .................................. = [0, 12] INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - plot .............................................. = show INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - plot-frac-above ................................... = 0.01 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - plot-bl ........................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - flag-ant .......................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - flag-ant-thr ...................................... = 5 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] [sol] Solution options which apply at the solver level INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - jones ............................................. = ['G', 'dE'] INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - precision ......................................... = 32 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - delta-g ........................................... = 1e-06 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - delta-chi ......................................... = 1e-06 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - chi-int ........................................... = 5 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - last-rites ........................................ = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - stall-quorum ...................................... = 0.99 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - term-iters ........................................ = [50, 50] INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - min-bl ............................................ = 50.0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.8Gb] - max-bl ............................................ = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - subset ............................................ = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] [bbc] Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains). INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - load-from ......................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - compute-2x2 ....................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - apply-2x2 ......................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - save-to ........................................... = {out[name]}-BBC-field{sel[field]}-ddid{sel[ddid]}.parmdb INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - per-chan .......................................... = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - plot .............................................. = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] [dist] Parallelization and distribution options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - ncpu .............................................. = 30 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - nworker ........................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - nthread ........................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-chunks ........................................ = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - min-chunks ........................................ = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - pin ............................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - pin-io ............................................ = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - pin-main .......................................... = io INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] [log] Options related to logging INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - memory ............................................ = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - stats ............................................. = chi2:.3f INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - stats-warn ........................................ = chi2:10 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - boring ............................................ = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - append ............................................ = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - verbose ........................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - file-verbose ...................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] [debug] Debugging options for the discerning masochist INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - pdb ............................................... = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - panic-amplitude ................................... = 0.0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - stop-before-solver ................................ = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - escalate-warnings ................................. = False INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] [misc] Miscellaneous options INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - random-seed ....................................... = None INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - parset-version .................................... = 0.1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] [g] Options for G-Jones term INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - solvable .......................................... = True INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - type .............................................. = complex-2x2 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - load-from ......................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - xfer-from ......................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - save-to ........................................... = ampphase-1.parmdb INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - dd-term ........................................... = 0 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - fix-dirs .......................................... = INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - update-type ....................................... = full INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - time-int .......................................... = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - freq-int .......................................... = 1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-prior-error ................................... = 0.1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-post-error .................................... = 0.1 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - low-snr-warn ...................................... = 75 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - high-gain-var-warn ................................ = 30 INFO - 10:51:53 - main [0.2/0.2 2.5/2.5 103.9Gb] - clip-low .......................................... = 0.5 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - clip-high ......................................... = 1.5 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - clip-after ........................................ = 5 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-iter .......................................... = 20 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - epsilon ........................................... = 1e-06 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - delta-chi ......................................... = 1e-06 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - conv-quorum ....................................... = 0.99 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - ref-ant ........................................... = None INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - prop-flags ........................................ = default INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - estimate-pzd ...................................... = False INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - diag-only ......................................... = 0 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - offdiag-only ...................................... = False INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-cov ........................................ = compute INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-scale ...................................... = 1 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-npol ....................................... = 2 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-int ........................................ = 1 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-save-weights ............................... = 0 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] [de] Options for dE-Jones term INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - solvable .......................................... = True INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - type .............................................. = complex-2x2 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - load-from ......................................... = INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - xfer-from ......................................... = INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - save-to ........................................... = ddjone_cal_1.paramdb INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - dd-term ........................................... = 1 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - fix-dirs .......................................... = INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - update-type ....................................... = full INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - time-int .......................................... = 100 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - freq-int .......................................... = 100 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-prior-error ................................... = 0.44 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-post-error .................................... = 0.44 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - low-snr-warn ...................................... = 75 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - high-gain-var-warn ................................ = 30 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - clip-low .......................................... = 0.5 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - clip-high ......................................... = 1.5 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - clip-after ........................................ = 5 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - max-iter .......................................... = 20 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - epsilon ........................................... = 1e-06 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - delta-chi ......................................... = 1e-05 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - conv-quorum ....................................... = 0.99 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - ref-ant ........................................... = None INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - prop-flags ........................................ = default INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - estimate-pzd ...................................... = False INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - diag-only ......................................... = 0 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - offdiag-only ...................................... = False INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-cov ........................................ = compute INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-scale ...................................... = 1 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-npol ....................................... = 2 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-int ........................................ = 1 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] - robust-save-weights ............................... = 0 INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] Enabling G,dE-Jones INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] mode: SolveAndCorrect INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] solver is apply-only type: False INFO - 10:51:54 - main [0.2/0.2 2.5/2.5 103.9Gb] solver requires model: True INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 103.9Gb] reading MS ../../../msdir/A2384_MeerKAT_raw-ABELL2384_corr.ms INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 104.0Gb] detected linear (xy) feeds INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 104.0Gb] fields are *0: ABELL2384 INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 104.0Gb] MS contains 1 spectral windows INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 104.0Gb] max freq chunk size is 100 channels and/or -- MHz INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 104.0Gb] DDID 0: 819/819 channels selected INFO - 10:51:54 - data_handler [0.2/0.2 2.5/2.5 104.0Gb] found 9 frequency chunks: 0 100 200 300 400 500 600 700 800 819 INFO - 10:51:55 - data_handler [0.2/0.3 2.6/2.6 104.5Gb] applying TAQL query 'FIELD_ID == 0 && DATA_DESC_ID IN [0]' (2455860/2455860 rows selected) INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] 60 antennas, 2455860 rows, 1/1 DDIDs, 1342 timeslots, 2 corrs INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] DDID central frequencies are at 1.28 GHz INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] Input model feed rotation enabled, PA rotation enabled INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] Output visibilities derotation enabled INFO - 10:51:55 - parallactic_machin [0.3/0.3 2.6/2.6 104.6Gb] Initializing new parallactic angle machine for 60 ECEF positions INFO - 10:51:55 - parallactic_machin [0.3/0.3 2.6/2.6 104.6Gb] Conversion epoch is J2000 INFO - 10:51:55 - parallactic_machin [0.3/0.3 2.6/2.6 104.6Gb] Selecting 'linear' feed bases INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] Using 1 model(s) for 2 directions(s) INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] model 0 (weight WEIGHT): INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] direction 0: MODEL_DATA-bright_source_column INFO - 10:51:55 - data_handler [0.3/0.3 2.6/2.6 104.6Gb] direction 1: bright_source_column INFO - 10:51:55 - main [0.3/0.3 2.6/2.6 104.6Gb] subtraction directions set to slice(None, None, None) INFO - 10:51:55 - param_db [0.3/0.3 2.6/2.6 104.6Gb] creating ampphase-1.parmdb in fragmented mode INFO - 10:51:55 - gain_machine [0.3/0.3 2.6/2.6 104.6Gb] G solutions will be saved to ampphase-1.parmdb INFO - 10:51:55 - param_db [0.3/0.3 2.6/2.6 104.6Gb] creating ddjone_cal_1.paramdb in fragmented mode INFO - 10:51:55 - gain_machine [0.3/0.3 2.6/2.6 104.6Gb] dE solutions will be saved to ddjone_cal_1.paramdb INFO - 10:51:55 - param_db [0.3/0.3 2.6/2.6 104.6Gb] creating cubical.cc-out/ampphase_1-BBC-field_0-ddid_None.parmdb in fragmented mode INFO - 10:51:55 - gain_machine [0.3/0.3 2.8/2.8 104.6Gb] will compute & save suggested baseline-based corrections (BBCs) to cubical.cc-out/ampphase_1-BBC-field_0-ddid_None.parmdb INFO - 10:51:55 - gain_machine [0.3/0.3 2.8/2.8 104.6Gb] (these can optionally be applied in a subsequent CubiCal run) INFO - 10:51:55 - main [0.3/0.3 2.8/2.8 104.6Gb] multi-process mode: 29+1 workers, single thread INFO - 10:51:55 - main [0.3/0.3 2.8/2.8 104.6Gb] defining chunks (time 100, freq 100, also when SCAN_NUMBER jumps > 1.0) INFO - 10:51:55 - data_handler [0.4/0.4 2.9/2.9 104.9Gb] read indexing columns (2455860 total rows) INFO - 10:51:57 - data_handler [0.4/0.4 2.9/2.9 105.6Gb] built timeslot index (1342 unique timestamps) INFO - 10:51:57 - data_handler [0.4/0.4 2.9/2.9 105.6Gb] max chunk size is 100 timeslots and/or -- seconds INFO - 10:52:07 - data_handler [0.5/1.0 3.0/3.5 111.4Gb] found 24 time chunks: 0:0:0 1:183000:100 2:204960:112 3:387960:212 4:409920:224 5:592920:324 6:613050:335 7:796050:435 8:818010:447 9:1001010:547 10:1022970:559 11:1205970:659 12:1227930:671 13:1410930:771 14:1432890:783 15:1615890:883 16:1637850:895 17:1820850:995 18:1842810:1007 19:2025810:1107 20:2047770:1119 21:2230770:1219 22:2252730:1231 23:2435730:1331 1342 INFO - 10:52:07 - data_handler [0.5/1.0 3.1/3.5 111.6Gb] generated 24 row chunks based on time and DDID INFO - 10:52:07 - data_handler [0.5/1.0 3.1/3.5 111.6Gb] row chunks yield 24 potential tiles INFO - 10:52:07 - data_handler [0.6/1.0 3.2/3.5 111.8Gb] coarsening this to 24 tiles (max 9 chunks per tile, based on 29/1 requested) INFO - 10:52:07 - data_handler [0.6/1.0 3.2/3.5 111.8Gb] will re-initialize BITFLAG column, since --flags-reinit-bitflags is set. INFO - 10:52:07 - data_handler [0.6/1.0 3.2/3.5 111.8Gb] WARNING: current state of FLAG column will be used to init bitflags! INFO - 10:52:09 - data_handler [0.6/1.0 3.2/3.5 112.3Gb] removing current BITFLAG/BITFLAG_ROW columns INFO - 10:52:09 - data_handler [0.6/1.0 3.2/3.5 112.8Gb] inserting new column BITFLAG INFO - 10:52:09 - data_handler [0.6/1.0 3.2/3.5 112.8Gb] inserting new column BITFLAG_ROW INFO - 10:52:10 - data_handler [0.6/1.0 3.2/3.5 113.2Gb] auto-filling bitflag 'legacy' from FLAG/FLAG_ROW column. Please do not interrupt this process! INFO - 10:52:10 - data_handler [0.6/1.0 3.2/3.5 113.2Gb] note that all other bitflags will be cleared by this INFO - 10:53:11 - data_handler [0.7/2.1 3.2/4.6 138.2Gb] auto-fill complete INFO - 10:53:11 - data_handler [0.7/2.1 3.2/4.6 138.2Gb] BITFLAG column defines the following flagsets: legacy:1 INFO - 10:53:11 - data_handler [0.7/2.1 3.2/4.6 138.2Gb] will exclude flagset cubical INFO - 10:53:11 - data_handler [0.7/2.1 3.2/4.6 138.2Gb] flagset 'cubical' not found -- ignoring INFO - 10:53:11 - data_handler [0.7/2.1 3.2/4.6 138.2Gb] applying BITFLAG mask 1 to input data INFO - 10:53:11 - data_handler [0.7/2.1 3.2/4.6 138.2Gb] will save output flags into BITFLAG 'cubical' (2), and into FLAG/FLAG_ROW INFO - 10:53:11 - main [0.7/2.1 3.4/4.6 138.2Gb] waiting for I/O on tile 0/24 INFO - 10:53:11 - main [io] [0.6/0.6 3.4/3.4 138.2Gb] loading tile 0/24 INFO - 10:53:11 - ms_tile [io] [0.6/0.6 3.4/3.4 138.2Gb] tile 0/24: reading MS rows 0~182999 INFO - 10:53:11 - data_handler [io] [0.6/0.6 3.4/3.4 138.2Gb] reading DATA INFO - 10:53:28 - ms_tile [io] [5.1/5.1 8.4/8.4 90.0Gb] applying solvable baseline cutoff deselects 6900 rows INFO - 10:53:28 - ms_tile [io] [5.1/5.1 8.4/8.4 90.1Gb] 3.77% visibilities deselected via specificed subset and/or baseline cutoffs INFO - 10:53:28 - data_handler [io] [5.1/5.1 8.4/8.4 90.1Gb] reading BITFLAG INFO - 10:53:51 - ms_tile [io] [7.6/10.1 10.3/12.8 99.8Gb] 60.00% input visibilities flagged and/or deselected INFO - 10:53:59 - data_handler [io] [10.7/11.5 17.9/17.9 108.1Gb] reading MODEL_DATA INFO - 10:58:10 - data_handler [io] [15.6/17.8 20.6/25.1 130.6Gb] reading bright_source_column INFO - 10:59:02 - ms_tile [io] [17.8/20.6 22.8/25.6 94.6Gb] 119893822 (40.00%) model visibilities flagged as 0/inf/nan INFO - 11:02:19 - main [io] [1.9/25.4 4.6/28.1 111.2Gb] I/O job(s) complete INFO - 11:02:19 - main [0.7/2.1 3.4/4.6 111.3Gb] submitting solver jobs for tile 0/24 INFO - 11:02:19 - main [io] [1.9/25.4 4.6/28.1 111.3Gb] loading tile 1/24 INFO - 11:02:19 - ms_tile [io] [1.9/25.4 4.6/28.1 111.3Gb] tile 1/24: reading MS rows 183000~204959 INFO - 11:02:19 - data_handler [io] [1.9/25.4 4.6/28.1 111.3Gb] reading DATA INFO - 11:02:20 - ms_tile [io] [2.4/25.4 5.2/28.1 111.9Gb] applying solvable baseline cutoff deselects 828 rows INFO - 11:02:20 - ms_tile [io] [2.4/25.4 5.2/28.1 111.9Gb] 3.77% visibilities deselected via specificed subset and/or baseline cutoffs INFO - 11:02:20 - data_handler [io] [2.4/25.4 5.2/28.1 111.9Gb] reading BITFLAG INFO - 11:02:22 - ms_tile [io] [2.6/25.4 5.4/28.1 113.0Gb] 60.62% input visibilities flagged and/or deselected INFO - 11:02:23 - data_handler [io] [3.0/25.4 6.3/28.1 113.9Gb] reading MODEL_DATA INFO - 11:02:46 - data_handler [io] [3.6/25.4 6.6/28.1 125.1Gb] reading bright_source_column INFO - 11:03:07 - ms_tile [io] [3.9/25.4 6.9/28.1 132.9Gb] 14166740 (39.38%) model visibilities flagged as 0/inf/nan ERROR - 11:03:26 - solver [x09] [2.2/9.9 5.1/12.9 133.4Gb] D0T0F8 no solutions: G: 0/1900 ints; dE: 0/1 ints; flags PRIOR:6726000(98.33%) LOWSNR:6840000(100.00%) MISSING:114000(1.67%) INFO - 11:03:35 - main [io] [2.0/25.4 4.8/28.1 102.5Gb] I/O job(s) complete INFO - 11:03:50 - solver [x09] [2.2/9.9 5.1/12.9 86.1Gb] D0T0F8 has 114000 (1.67%) new data flags: gain flags LOWSNR:114000(100.00%) (100.00% total) CRITICAL - 11:05:29 - solver [x03] [8.7/13.4 11.6/16.4 133.6Gb] 'dE' D0T0F2 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:05:29 - solver [x03] [8.7/13.4 11.6/16.4 133.6Gb] 'dE' D0T0F2 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:05:30 - solver [x05] [8.7/13.5 11.6/16.5 133.7Gb] 'dE' D0T0F4 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:05:30 - solver [x05] [8.7/13.5 11.6/16.5 133.7Gb] 'dE' D0T0F4 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:05:41 - solver [x07] [8.7/13.8 11.6/16.8 134.4Gb] 'dE' D0T0F6 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:05:41 - solver [x07] [8.7/13.8 11.6/16.8 134.4Gb] 'dE' D0T0F6 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:05:42 - solver [x01] [8.7/13.9 11.6/16.9 134.4Gb] 'dE' D0T0F0 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:05:42 - solver [x01] [8.7/13.9 11.6/16.9 134.4Gb] 'dE' D0T0F0 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:07:31 - solver [x06] [8.7/13.4 11.6/16.4 127.4Gb] 'dE' D0T0F5 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:07:31 - solver [x06] [8.7/13.4 11.6/16.4 127.4Gb] 'dE' D0T0F5 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:07:57 - solver [x02] [8.7/13.4 11.6/16.4 134.4Gb] 'dE' D0T0F1 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:07:57 - solver [x02] [8.7/13.4 11.6/16.4 134.4Gb] 'dE' D0T0F1 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:08:12 - solver [x08] [8.7/13.9 11.6/16.9 92.2Gb] 'dE' D0T0F7 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:08:12 - solver [x08] [8.7/13.9 11.6/16.9 92.2Gb] 'dE' D0T0F7 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:08:16 - solver [x04] [8.7/14.0 11.6/17.0 86.3Gb] 'dE' D0T0F3 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:08:16 - solver [x04] [8.7/14.0 11.6/17.0 86.3Gb] 'dE' D0T0F3 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 11:17:13 - solver [x02] [14.2/16.4 17.2/19.3 130.6Gb] D0T0F1 has 136707 (0.38%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2 visibilities INFO - 11:17:28 - solver [x05] [14.2/16.4 17.2/19.3 126.2Gb] D0T0F4 has 225338 (0.63%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2 visibilities INFO - 11:17:43 - solver [x06] [14.2/16.4 17.2/19.3 80.8Gb] D0T0F5 has 142945 (0.40%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2 visibilities INFO - 11:18:52 - solver [x01] [14.2/16.4 17.2/19.3 130.6Gb] D0T0F0 has 488455 (1.36%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2 visibilities INFO - 11:22:21 - solver [x03] [14.2/16.4 17.2/19.3 104.6Gb] D0T0F2 has 226992 (0.63%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 130 visibilities INFO - 11:27:40 - solver [x07] [14.2/16.4 17.2/19.3 138.0Gb] D0T0F6 has 770286 (2.14%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 358 visibilities INFO - 11:36:20 - solver [x08] [14.2/16.4 17.2/19.3 119.0Gb] D0T0F7 has 723167 (2.01%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 160 visibilities INFO - 11:46:50 - solver [x04] [14.2/16.4 17.2/19.3 119.3Gb] D0T0F3 has 686679 (1.91%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2456 visibilities INFO - 11:48:12 - main [0.7/2.1 3.6/4.6 119.3Gb] finished processing tile 0/24 INFO - 11:48:12 - main [0.7/2.1 3.6/4.6 119.3Gb] waiting for I/O on tile 1/24 INFO - 11:48:12 - main [0.7/2.1 3.6/4.6 119.3Gb] submitting solver jobs for tile 1/24 INFO - 11:48:12 - main [io] [2.0/25.4 4.8/28.1 119.3Gb] saving tile 0/24 INFO - 11:48:12 - ms_tile [io] [2.0/25.4 4.8/28.1 119.3Gb] tile 0/24: saving MS rows 0~182999 INFO - 11:48:12 - ms_tile [io] [2.0/25.4 7.0/28.1 119.3Gb] writing DD_DATA column INFO - 11:48:19 - ms_tile [io] [5.0/25.4 7.7/28.1 119.3Gb] 56.40% visibilities flagged by solver: saving to BITFLAG and FLAG columns INFO - 11:48:23 - ms_tile [io] [5.2/25.4 8.0/28.1 119.3Gb] updated BITFLAG column (60.25% visibilities flagged) INFO - 11:48:23 - ms_tile [io] [5.2/25.4 8.0/28.1 119.3Gb] updated BITFLAG_ROW column (12.24% rows flagged) ERROR - 11:48:24 - solver [x18] [0.7/1.6 3.7/4.6 119.3Gb] D0T1F8 no solutions: G: 0/228 ints; dE: 0/1 ints; flags PRIOR:807120(98.33%) LOWSNR:820800(100.00%) MISSING:13680(1.67%) INFO - 11:48:24 - ms_tile [io] [4.1/25.4 6.9/28.1 119.3Gb] updated FLAG column (60.25% visibilities flagged) INFO - 11:48:24 - ms_tile [io] [4.1/25.4 6.9/28.1 119.3Gb] updated FLAG_ROW column (12.25% rows flagged) INFO - 11:48:27 - main [io] [1.0/25.4 3.5/28.1 108.1Gb] loading tile 2/24 INFO - 11:48:27 - ms_tile [io] [1.0/25.4 3.5/28.1 108.1Gb] tile 2/24: reading MS rows 204960~387959 INFO - 11:48:28 - data_handler [io] [1.0/25.4 3.5/28.1 108.1Gb] reading DATA CRITICAL - 11:48:31 - solver [x15] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F5 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:31 - solver [x15] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F5 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:48:34 - solver [x17] [1.6/2.1 4.5/5.1 108.2Gb] 'dE' D0T1F7 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:34 - solver [x17] [1.6/2.1 4.5/5.1 108.2Gb] 'dE' D0T1F7 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:48:34 - solver [x13] [1.6/2.1 4.5/5.1 108.2Gb] 'dE' D0T1F3 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:34 - solver [x13] [1.6/2.1 4.5/5.1 108.2Gb] 'dE' D0T1F3 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 11:48:37 - ms_tile [io] [5.4/25.4 8.6/28.1 108.2Gb] applying solvable baseline cutoff deselects 6900 rows INFO - 11:48:37 - ms_tile [io] [5.4/25.4 8.6/28.1 108.2Gb] 3.77% visibilities deselected via specificed subset and/or baseline cutoffs CRITICAL - 11:48:37 - solver [x16] [1.6/2.1 4.5/5.1 108.2Gb] 'dE' D0T1F6 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:37 - solver [x16] [1.6/2.1 4.5/5.1 108.2Gb] 'dE' D0T1F6 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 11:48:37 - data_handler [io] [5.4/25.4 8.6/28.1 108.2Gb] reading BITFLAG INFO - 11:48:38 - solver [x18] [0.7/1.6 3.7/4.6 108.2Gb] D0T1F8 has 13680 (1.67%) new data flags: gain flags LOWSNR:13680(100.00%) (100.00% total) CRITICAL - 11:48:40 - solver [x14] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F4 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:40 - solver [x14] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F4 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:48:41 - solver [x11] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F1 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:41 - solver [x11] [1.6/2.0 4.5/5.0 108.2Gb] 'G' D0T1F1 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data WARNING - 11:48:41 - solver [x11] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F1 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:48:41 - solver [x12] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F2 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:41 - solver [x12] [1.6/2.0 4.5/5.0 108.2Gb] 'dE' D0T1F2 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:48:42 - solver [x10] [1.6/2.1 4.6/5.1 108.2Gb] 'dE' D0T1F0 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:48:42 - solver [x10] [1.6/2.1 4.6/5.1 108.2Gb] 'dE' D0T1F0 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 11:48:58 - ms_tile [io] [7.9/25.4 10.5/28.1 108.3Gb] 59.84% input visibilities flagged and/or deselected INFO - 11:49:06 - data_handler [io] [11.1/25.4 18.1/28.1 112.3Gb] reading MODEL_DATA INFO - 11:50:03 - solver [x15] [2.3/2.5 5.2/5.5 112.4Gb] D0T1F5 has 25148 (0.58%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total) INFO - 11:50:27 - solver [x14] [2.3/2.5 5.2/5.5 112.5Gb] D0T1F4 has 32510 (0.75%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 4 visibilities INFO - 11:50:51 - solver [x11] [2.3/2.5 5.2/5.5 112.5Gb] D0T1F1 has 16550 (0.38%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2 visibilities INFO - 11:51:04 - solver [x17] [2.3/2.5 5.2/5.5 112.6Gb] D0T1F7 has 84284 (1.95%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 30 visibilities INFO - 11:51:11 - solver [x12] [2.3/2.5 5.2/5.5 112.6Gb] D0T1F2 has 29321 (0.68%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 38 visibilities INFO - 11:51:21 - solver [x10] [2.3/2.5 5.2/5.5 112.7Gb] D0T1F0 has 58924 (1.36%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total) INFO - 11:51:35 - solver [x13] [2.3/2.5 5.2/5.5 112.7Gb] D0T1F3 has 87679 (2.03%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 180 visibilities INFO - 11:51:45 - data_handler [io] [16.0/25.4 20.8/28.1 105.4Gb] reading bright_source_column INFO - 11:52:02 - ms_tile [io] [18.2/25.4 23.0/28.1 98.5Gb] 120380988 (40.16%) model visibilities flagged as 0/inf/nan INFO - 11:52:23 - solver [x16] [2.3/2.5 5.2/5.5 100.9Gb] D0T1F6 has 101440 (2.35%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 34 visibilities INFO - 11:52:33 - main [0.7/2.1 3.6/4.6 103.1Gb] finished processing tile 1/24 INFO - 11:52:33 - main [0.7/2.1 3.6/4.6 103.1Gb] waiting for I/O on tile 2/24 INFO - 11:55:02 - main [io] [2.2/25.7 4.8/28.3 117.2Gb] I/O job(s) complete INFO - 11:55:02 - main [0.7/2.1 3.6/4.6 117.2Gb] submitting solver jobs for tile 2/24 INFO - 11:55:02 - main [io] [2.2/25.7 4.8/28.3 117.2Gb] saving tile 1/24 INFO - 11:55:02 - ms_tile [io] [2.2/25.7 4.8/28.3 117.2Gb] tile 1/24: saving MS rows 183000~204959 INFO - 11:55:02 - ms_tile [io] [2.2/25.7 5.1/28.3 117.2Gb] writing DD_DATA column INFO - 11:55:03 - ms_tile [io] [2.6/25.7 5.1/28.3 117.2Gb] 56.41% visibilities flagged by solver: saving to BITFLAG and FLAG columns INFO - 11:55:04 - ms_tile [io] [2.6/25.7 5.2/28.3 117.2Gb] updated BITFLAG column (60.95% visibilities flagged) INFO - 11:55:04 - ms_tile [io] [2.6/25.7 5.2/28.3 117.2Gb] updated BITFLAG_ROW column (10.43% rows flagged) INFO - 11:55:04 - ms_tile [io] [2.5/25.7 5.0/28.3 117.2Gb] updated FLAG column (60.95% visibilities flagged) INFO - 11:55:04 - ms_tile [io] [2.5/25.7 5.0/28.3 117.2Gb] updated FLAG_ROW column (10.58% rows flagged) INFO - 11:55:05 - main [io] [2.1/25.7 4.7/28.3 115.7Gb] loading tile 3/24 INFO - 11:55:05 - ms_tile [io] [2.1/25.7 4.7/28.3 115.7Gb] tile 3/24: reading MS rows 387960~409919 INFO - 11:55:05 - data_handler [io] [2.1/25.7 4.7/28.3 115.7Gb] reading DATA INFO - 11:55:06 - ms_tile [io] [2.6/25.7 5.3/28.3 115.7Gb] applying solvable baseline cutoff deselects 828 rows INFO - 11:55:06 - ms_tile [io] [2.6/25.7 5.3/28.3 115.7Gb] 3.77% visibilities deselected via specificed subset and/or baseline cutoffs INFO - 11:55:06 - data_handler [io] [2.6/25.7 5.3/28.3 115.7Gb] reading BITFLAG INFO - 11:55:09 - ms_tile [io] [2.9/25.7 5.5/28.3 115.7Gb] 62.28% input visibilities flagged and/or deselected INFO - 11:55:10 - data_handler [io] [3.3/25.7 6.4/28.3 116.2Gb] reading MODEL_DATA INFO - 11:55:31 - data_handler [io] [3.8/25.7 6.7/28.3 116.5Gb] reading bright_source_column INFO - 11:55:33 - ms_tile [io] [4.1/25.7 7.0/28.3 116.5Gb] 13568940 (37.72%) model visibilities flagged as 0/inf/nan INFO - 11:56:09 - main [io] [2.3/25.7 4.8/28.3 116.7Gb] I/O job(s) complete ERROR - 11:56:16 - solver [x27] [2.2/9.9 5.1/12.9 116.7Gb] D0T2F8 no solutions: G: 0/1900 ints; dE: 0/1 ints; flags PRIOR:6726000(98.33%) LOWSNR:6840000(100.00%) MISSING:114000(1.67%) INFO - 11:56:41 - solver [x27] [2.2/9.9 5.1/12.9 116.7Gb] D0T2F8 has 114000 (1.67%) new data flags: gain flags LOWSNR:114000(100.00%) (100.00% total) CRITICAL - 11:58:08 - solver [x21] [8.7/13.5 11.6/16.4 117.5Gb] 'dE' D0T2F2 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:58:08 - solver [x21] [8.7/13.5 11.6/16.4 117.5Gb] 'dE' D0T2F2 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:58:10 - solver [x23] [8.7/13.5 11.6/16.5 117.5Gb] 'dE' D0T2F4 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:58:10 - solver [x23] [8.7/13.5 11.6/16.5 117.5Gb] 'dE' D0T2F4 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:58:15 - solver [x25] [8.7/13.8 11.6/16.8 117.5Gb] 'dE' D0T2F6 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:58:15 - solver [x25] [8.7/13.8 11.6/16.8 117.5Gb] 'dE' D0T2F6 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:58:20 - solver [x19] [8.7/14.0 11.6/16.9 117.5Gb] 'dE' D0T2F0 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:58:20 - solver [x19] [8.7/14.0 11.6/16.9 117.5Gb] 'dE' D0T2F0 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 11:59:38 - solver [x20] [8.7/13.4 11.6/16.4 117.5Gb] 'dE' D0T2F1 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 11:59:38 - solver [x20] [8.7/13.4 11.6/16.4 117.5Gb] 'dE' D0T2F1 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:00:06 - solver [x26] [8.7/13.9 11.6/16.9 117.5Gb] 'dE' D0T2F7 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:00:06 - solver [x26] [8.7/13.9 11.6/16.9 117.5Gb] 'dE' D0T2F7 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:00:26 - solver [x22] [8.7/14.0 11.6/17.0 117.5Gb] 'dE' D0T2F3 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:00:26 - solver [x22] [8.7/14.0 11.6/17.0 117.5Gb] 'dE' D0T2F3 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:00:49 - solver [x24] [8.7/13.4 11.6/16.4 117.5Gb] 'dE' D0T2F5 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:00:49 - solver [x24] [8.7/13.4 11.6/16.4 117.5Gb] 'dE' D0T2F5 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 12:11:48 - solver [x23] [14.2/16.4 17.2/19.3 107.7Gb] D0T2F4 has 229421 (0.64%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 72 visibilities INFO - 12:12:25 - solver [x21] [14.2/16.4 17.2/19.3 139.2Gb] D0T2F2 has 234137 (0.65%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 134 visibilities INFO - 12:12:43 - solver [x20] [14.2/16.4 17.2/19.3 152.8Gb] D0T2F1 has 141749 (0.39%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total) INFO - 12:12:45 - solver [x24] [14.2/16.4 17.2/19.3 153.2Gb] D0T2F5 has 134551 (0.37%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total) INFO - 12:17:36 - solver [x19] [14.3/16.4 17.2/19.3 139.2Gb] D0T2F0 has 492594 (1.37%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 416 visibilities INFO - 12:18:15 - solver [x25] [14.2/16.4 17.2/19.3 121.9Gb] D0T2F6 has 772321 (2.15%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 392 visibilities INFO - 12:24:42 - solver [x26] [14.2/16.4 17.2/19.3 118.7Gb] D0T2F7 has 693041 (1.93%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 64 visibilities INFO - 12:35:54 - solver [x22] [14.2/16.4 17.2/19.3 136.8Gb] D0T2F3 has 703678 (1.95%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 7730 visibilities INFO - 12:36:33 - main [0.7/2.1 3.6/4.6 139.6Gb] finished processing tile 2/24 INFO - 12:36:33 - main [0.7/2.1 3.6/4.6 139.6Gb] waiting for I/O on tile 3/24 INFO - 12:36:33 - main [0.7/2.1 3.6/4.6 139.6Gb] submitting solver jobs for tile 3/24 INFO - 12:36:33 - main [io] [2.2/25.7 4.8/28.3 139.6Gb] saving tile 2/24 INFO - 12:36:33 - ms_tile [io] [2.2/25.7 4.8/28.3 139.6Gb] tile 2/24: saving MS rows 204960~387959 INFO - 12:36:33 - ms_tile [io] [2.2/25.7 7.1/28.3 139.6Gb] writing DD_DATA column ERROR - 12:36:36 - solver [x07] [0.9/16.5 3.8/20.7 139.6Gb] D0T3F8 no solutions: G: 0/228 ints; dE: 0/1 ints; flags PRIOR:807120(98.33%) LOWSNR:820800(100.00%) MISSING:13680(1.67%) INFO - 12:36:37 - solver [x07] [0.8/16.5 3.8/20.7 136.6Gb] D0T3F8 has 13680 (1.67%) new data flags: gain flags LOWSNR:13680(100.00%) (100.00% total) INFO - 12:36:40 - ms_tile [io] [5.2/25.7 7.7/28.3 118.2Gb] 56.74% visibilities flagged by solver: saving to BITFLAG and FLAG columns INFO - 12:36:44 - ms_tile [io] [5.4/25.7 8.0/28.3 99.5Gb] updated BITFLAG column (60.08% visibilities flagged) CRITICAL - 12:36:44 - solver [x06] [1.7/16.5 4.7/20.7 98.6Gb] 'dE' D0T3F5 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:44 - solver [x06] [1.7/16.5 4.7/20.7 98.6Gb] 'dE' D0T3F5 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 12:36:44 - ms_tile [io] [5.4/25.7 8.0/28.3 98.4Gb] updated BITFLAG_ROW column (11.59% rows flagged) CRITICAL - 12:36:44 - solver [x05] [1.7/16.5 4.7/20.7 97.8Gb] 'dE' D0T3F4 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:44 - solver [x05] [1.7/16.5 4.7/20.7 97.8Gb] 'dE' D0T3F4 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 12:36:45 - ms_tile [io] [4.3/25.7 6.9/28.3 95.4Gb] updated FLAG column (60.08% visibilities flagged) INFO - 12:36:45 - ms_tile [io] [4.3/25.7 6.9/28.3 95.1Gb] updated FLAG_ROW column (11.59% rows flagged) CRITICAL - 12:36:45 - solver [x02] [1.7/16.5 4.7/20.7 93.3Gb] 'dE' D0T3F3 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:45 - solver [x02] [1.7/16.5 4.7/20.7 93.3Gb] 'dE' D0T3F3 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:36:45 - solver [x03] [1.7/16.5 4.6/20.7 92.9Gb] 'dE' D0T3F7 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:45 - solver [x03] [1.7/16.5 4.6/20.7 92.9Gb] 'dE' D0T3F7 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:36:46 - solver [x01] [1.7/16.5 4.7/20.7 91.5Gb] 'dE' D0T3F6 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:46 - solver [x01] [1.7/16.5 4.7/20.7 91.5Gb] 'dE' D0T3F6 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:36:46 - solver [x28] [1.6/2.1 4.6/5.1 89.7Gb] 'dE' D0T3F0 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:46 - solver [x28] [1.6/2.1 4.6/5.1 89.7Gb] 'dE' D0T3F0 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 12:36:48 - main [io] [1.0/25.7 3.6/28.3 80.4Gb] loading tile 4/24 INFO - 12:36:48 - ms_tile [io] [1.0/25.7 3.6/28.3 80.4Gb] tile 4/24: reading MS rows 409920~592919 INFO - 12:36:48 - data_handler [io] [1.0/25.7 3.6/28.3 80.4Gb] reading DATA CRITICAL - 12:36:53 - solver [x29] [1.6/2.0 4.5/5.0 80.5Gb] 'dE' D0T3F1 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:53 - solver [x29] [1.6/2.0 4.5/5.0 80.5Gb] 'dE' D0T3F1 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data CRITICAL - 12:36:58 - solver [x09] [1.6/9.9 4.5/12.9 82.5Gb] 'dE' D0T3F2 Low SNR in directions 1. Increase solution intervals or raise max-prior-error! WARNING - 12:36:58 - solver [x09] [1.6/9.9 4.5/12.9 82.6Gb] 'dE' D0T3F2 Stations 1, 2 (2/60) fully flagged due to low SNR. These stations may be faulty or your SNR requirements (max-prior-error) are not met. New flags will be raised for this chunk of data INFO - 12:36:59 - ms_tile [io] [5.4/25.7 8.6/28.3 83.1Gb] applying solvable baseline cutoff deselects 6900 rows INFO - 12:36:59 - ms_tile [io] [5.4/25.7 8.6/28.3 83.1Gb] 3.77% visibilities deselected via specificed subset and/or baseline cutoffs INFO - 12:36:59 - data_handler [io] [5.5/25.7 8.6/28.3 83.1Gb] reading BITFLAG INFO - 12:37:21 - ms_tile [io] [8.0/25.7 10.5/28.3 94.7Gb] 60.00% input visibilities flagged and/or deselected INFO - 12:37:29 - data_handler [io] [11.1/25.7 18.1/28.3 103.4Gb] reading MODEL_DATA INFO - 12:38:37 - solver [x28] [2.3/2.5 5.2/5.5 132.6Gb] D0T3F0 has 59934 (1.39%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 2 visibilities INFO - 12:38:37 - solver [x06] [2.4/16.5 5.3/20.7 132.7Gb] D0T3F5 has 33691 (0.78%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total) INFO - 12:38:46 - solver [x05] [2.4/16.5 5.3/20.7 108.8Gb] D0T3F4 has 40228 (0.93%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 4 visibilities INFO - 12:39:28 - solver [x29] [2.3/2.5 5.2/5.5 100.4Gb] D0T3F1 has 28001 (0.65%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total) INFO - 12:39:45 - solver [x03] [2.4/16.5 5.3/20.7 108.9Gb] D0T3F7 has 87233 (2.02%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 18 visibilities INFO - 12:40:14 - solver [x01] [2.4/16.5 5.3/20.7 125.0Gb] D0T3F6 has 105993 (2.45%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 148 visibilities INFO - 12:40:20 - solver [x09] [2.3/9.9 5.2/12.9 128.8Gb] D0T3F2 has 41096 (0.95%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 64 visibilities INFO - 12:40:32 - data_handler [io] [16.0/25.7 20.8/28.3 135.1Gb] reading bright_source_column INFO - 12:40:49 - ms_tile [io] [18.2/25.7 23.0/28.3 135.2Gb] 119891104 (40.00%) model visibilities flagged as 0/inf/nan INFO - 12:41:29 - solver [x02] [2.4/16.5 5.3/20.7 109.0Gb] D0T3F3 has 87328 (2.02%) new data flags: gain flags LOWSNR:62(51.67%) (51.67% total), MadMax took out 470 visibilities INFO - 12:43:46 - main [io] [2.3/25.8 4.8/28.3 96.5Gb] I/O job(s) complete

Thanks.

IanHeywood commented 5 years ago

It doesn't look like it's finished here, unless I'm missing something. At the start it identifies 24 data tiles but the log only goes as far a tile 4. Usually there's a "time take for solve and save" message at the end of a successful run.

Some other observations:

viralp commented 5 years ago

Ok we solved this issue. Some of the processors were failing and messaging 'core dump' error. I turned off all the plotting options and its working.

ratt-priv-ci commented 5 years ago

Watch your logs please - I even made these flashing red

CRITICAL - 12:00:49 - solver [x24] [8.7/13.4 11.6/16.4 117.5Gb] 'dE' D0T2F5
Low SNR in directions 1. Increase solution intervals or raise
max-prior-error!

These dE solutions will not be solved and will therefore will not subtract. Since you only have one dE direction it is the reason why your sources are not subtracted

On Fri, Oct 4, 2019 at 10:31 AM Viral Parekh notifications@github.com wrote:

Ok we solved this issue. Some of the processors were failing and messaging 'core dump' error. I turned off all the plotting options and its working.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ratt-ru/CubiCal/issues/320?email_source=notifications&email_token=AEIVPJWVM7BO4PXRKGRZCV3QM35OHA5CNFSM4I4YP4PKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAK5QLI#issuecomment-538302509, or mute the thread https://github.com/notifications/unsubscribe-auth/AEIVPJVAWW22AOUXINGHNKTQM35OHANCNFSM4I4YP4PA .

--

Benjamin Hugo

Junior Software Developer SARAO Black River Park, 2 Fir Street, Observatory, Cape Town, Western Cape, 7925 Contact: [+27] 0716293858 <+27%2071%20629%203858>

PhD. student, Radio Astronomy Techniques and Technologies, Department of Physics and Electronics, Rhodes University

Skype: benna.cn

o-smirnov commented 5 years ago

@viralp how did you solve it?

I think I'm seeing the same in the pipeline here...