Applying gains in Cubical results in Error

[PeterKamphuis]

When I try to apply previously derived gains cubical throws the following error several time.

meerkathi - 2019-09-04 16:54:27,520 INFO - ERROR - 14:54:24 - solver [x03] [0.8/0.9 1.8/2.2 0.9Gb] Solver for tile 0 chunk D0T2F0 failed with exception: Failed in nop ython mode pipeline (step: nopython frontend) meerkathi - 2019-09-04 16:54:27,520 INFO - Unknown attribute 'shape' of type none meerkathi - 2019-09-04 16:54:27,520 INFO - meerkathi - 2019-09-04 16:54:27,520 INFO - File "../../usr/local/lib/python2.7/dist-packages/cubical/kernels/diag_phase_only.py", line 69: meerkathi - 2019-09-04 16:54:27,520 INFO - def compute_jhj(m, jhj, t_int=1, f_int=1): meerkathi - 2019-09-04 16:54:27,520 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - n_dir = m.shape[0] meerkathi - 2019-09-04 16:54:27,521 INFO - ^ meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - [1] During: typing of get attribute at /usr/local/lib/python2.7/dist-packages/cubical/kernels/diag_phase_only.py (69) meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - File "../../usr/local/lib/python2.7/dist-packages/cubical/kernels/diag_phase_only.py", line 69: meerkathi - 2019-09-04 16:54:27,521 INFO - def compute_jhj(m, jhj, t_int=1, f_int=1): meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - n_dir = m.shape[0] meerkathi - 2019-09-04 16:54:27,521 INFO - ^ meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - This is not usually a problem with Numba itself but instead often caused by meerkathi - 2019-09-04 16:54:27,521 INFO - the use of unsupported features or an issue in resolving types. meerkathi - 2019-09-04 16:54:27,521 INFO - meerkathi - 2019-09-04 16:54:27,521 INFO - To see Python/NumPy features supported by the latest release of Numba visit: meerkathi - 2019-09-04 16:54:27,521 INFO - http://numba.pydata.org/numba-doc/latest/reference/pysupported.html meerkathi - 2019-09-04 16:54:27,522 INFO - and meerkathi - 2019-09-04 16:54:27,522 INFO - http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html meerkathi - 2019-09-04 16:54:27,522 INFO - meerkathi - 2019-09-04 16:54:27,522 INFO - For more information about typing errors and how to debug them visit: meerkathi - 2019-09-04 16:54:27,522 INFO - http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile meerkathi - 2019-09-04 16:54:27,522 INFO - meerkathi - 2019-09-04 16:54:27,522 INFO - If you think your code should work with Numba, please report the error message meerkathi - 2019-09-04 16:54:27,522 INFO - and traceback, along with a minimal reproducer at: meerkathi - 2019-09-04 16:54:27,522 INFO - https://github.com/numba/numba/issues/new

Cubical seems to continue while throwing that error several time more and then stimela exits with a critical error:

utils.StimelaCabRuntimeError: gocubical: returns errr code 1

I used the following parset:

[data] _Help = Visibility data options ms = /scratch/msdir/Quick-GRB111005A_corr.ms column = DATA time-chunk = 128 freq-chunk = 0 rebin-time = 1 rebin-freq = 1 chunk-by = SCAN_NUMBER chunk-by-jump = 1 single-chunk = single-tile = -1

[sel] _Help = Data selection options field = 0 ddid = None taql = chan = diag = True

[out] _Help = Options for output products dir = cubical name = /scratch/output/continuum/selfcal_products/Quick-Quick-GRB111005A_corr.ms_2_restored_cubical overwrite = False backup = 1 mode = ac apply-solver-flags = True column = CORRECTED_DATA derotate = None model-column = weight-column = reinit-column = False subtract-model = 0 subtract-dirs = : plots = 1 casa-gaintables = False

[model] _Help = Calibration model options list = ddes = never beam-pattern = None beam-l-axis = None beam-m-axis = None feed-rotate = auto pa-rotate = True

[montblanc] _Help = Montblanc simulation options device-type = CPU dtype = float mem-budget = 1024 verbosity = WARNING threads = 0 pa-rotate = None

[weight] _Help = Weighting options column = WEIGHT fill-offdiag = False legacy-v1-2 = False

[flags] _Help = General flagging options apply = -cubical auto-init = legacy save = cubical save-legacy = auto reinit-bitflags = False warn-thr = 0.3 see-no-evil = 0

[degridding] _Help = Options for the degridder. Only in use when predicting from DicoModels using DDFacet OverS = 11 Support = 7 Nw = 100 wmax = 0.0 Padding = 1.7 NDegridBand = 16 MaxFacetSize = 0.25 MinNFacetPerAxis = 1 NProcess = 8

[postmortem] _Help = Options for "postmortem" flagging based on solution statistics enable = False tf-chisq-median = 1.2 tf-np-median = 0.5 time-density = 0.5 chan-density = 0.5 ddid-density = 0.5

[madmax] _Help = Options for the "Mad Max" flagger enable = 1 residuals = 0 estimate = corr diag = True offdiag = False threshold = [0, 5] global-threshold = [0, 12] plot = 0 plot-frac-above = 0.01 plot-bl = flag-ant = 0 flag-ant-thr = 5

[sol] _Help = Solution options which apply at the solver level jones = G precision = 32 delta-g = 1e-06 delta-chi = 1e-06 chi-int = 5 last-rites = True stall-quorum = 0.99 term-iters = 0 min-bl = 0.0 max-bl = 0 subset =

[bbc] _Help = Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains). load-from = compute-2x2 = False apply-2x2 = False save-to = file per-chan = True plot = True

[dist] _Help = Parallelization and distribution options ncpu = 6 nworker = 0 nthread = 0 max-chunks = 0 min-chunks = 0 pin = 0 pin-io = False pin-main = io

[log] _Help = Options related to logging memory = True stats = chi2:.3f stats-warn = chi2:10 boring = False append = False verbose = 0 file-verbose = None

[debug] _Help = Debugging options for the discerning masochist pdb = False panic-amplitude = 0.0 stop-before-solver = False escalate-warnings = False

[misc] _Help = Miscellaneous options random-seed = None parset-version = 0.1

[JONES-TEMPLATE] _Help = Options for {LABEL}-Jones term _NameTemplate = {LABEL} _ExpandedFrom = --sol-jones _OtherTemplates = Help:label label = {LABEL} solvable = 1 type = complex-2x2 load-from = xfer-from = save-to = {out[name]}-{JONES}-field{sel[field]}-ddid_{sel[ddid]}.parmdb dd-term = False fix-dirs = update-type = full time-int = 1 freq-int = 1 max-prior-error = 0.1 max-post-error = 0.1 low-snr-warn = 75 high-gain-var-warn = 30 clip-low = 0.1 clip-high = 10.0 clip-after = 5 max-iter = 20 epsilon = 1e-06 delta-chi = 1e-06 conv-quorum = 0.99 ref-ant = None prop-flags = default estimate-pzd = False diag-only = 0 offdiag-only = False robust-cov = compute robust-scale = 1 robust-npol = 2 robust-int = 1 robust-save-weights = 0

[g] _Help = Options for G-Jones term label = G solvable = 0 type = phase-diag load-from = xfer-from = /scratch/output/continuum/selfcal_products/g-gains-2-Quick-GRB111005Acorr.parmdb save-to = {out[name]}-{JONES}-field{sel[field]}-ddid_{sel[ddid]}.parmdb dd-term = 0 fix-dirs = update-type = phase-diag time-int = 5 freq-int = 0 max-prior-error = 0.1 max-post-error = 0.1 low-snr-warn = 75 high-gain-var-warn = 30 clip-low = 0.1 clip-high = 10 clip-after = 5 max-iter = 20 epsilon = 1e-06 delta-chi = 1e-06 conv-quorum = 0.99 ref-ant = None prop-flags = default estimate-pzd = False diag-only = 0 offdiag-only = False robust-cov = compute robust-scale = 1 robust-npol = 2 robust-int = 1 robust-save-weights = 0 _Templated = True

[de] _Templated = 1 dd-term = 1 clip-low = 0.0 clip-high = 0 delta-chi = 1e-05 max-prior-error = 0.44 max-post-error = 0.44

No idea why this is going wrong as it seems to work correctly when generating the gains with similar settings.

[KshitijT]

I think load-from is the appropriate option to use here, not xfer-from ?

[PeterKamphuis]

that results in the error:

Solver for tile 0 chunk D0T2F0 failed with exception: G:phase does not define /scratch/output/continuum/selfcal_products/g-gains-2-Quick-GRB111005A_corr.parmdb on the correct grid. Consider using -xfer-from rather than -load-from

Probably because I have changed the freq-chunk from 128 to 0 between deriving and applying. In anycase xfer would be necessary for interpolating to the high res data. It used to work with the old Cubical.

[PeterKamphuis]

And load-from with the correct chunks results in the same error.

[JSKenyon]

@PeterKamphuis This looks like an empty model array is somehow being passed into the kernel. The crash seems to be caused by an attempt to access the shape of the model, which is None type for some reason. I would suggest re-raising this on the CubiCal repository. Out of interest, how large is your MS?

[JSKenyon]

Actually, I am very confused - the error is coming from code which should not be called. If this is a solution transfer, I find it odd that a kernel which is used for computing gains is being called. All the more reason to raise this on the CubiCal repo.

[PeterKamphuis]

@JSKenyon I also thought that this module shouldn't be called at all but I'm not so experienced with cubical so maybe I have a wrong setting in the parset. That is one of the reasons for raising it here, because the only thing relating to the solver that I could see that was set wrong is apply-solver-flags = True but this is a fixed parameter in Stimela as far as I can see. The ms is a test set so only 4Gb

I'll raise the issue on the Cubical github as well.

[PeterKamphuis]

I raised an issue on the Cubical github here.

[bennahugo]

Closing this as it is not a framework issue