Unexpected keyword argument tdim_ind (solving for G whilst applying K)

[IanHeywood]

I've generated f-slope K solutions and now I'm trying to apply them whilst I solve for complex-2x2 G solutions in a second run, but it crashes with the following:

INFO      17:52:19 - gain_machine       [x01] [26.3/102.0 20.4/49.0 13.0Gb] D0T0F0: loading K:delay from selfcal.cc-out/se
lfcal-K-field_0-ddid_None.parmdb
ERROR     17:52:53 - solver             [x01] [32.0/107.7 26.2/54.8 13.0Gb] Solver for tile 0 chunk D0T0F0 failed with exc
eption: copy_or_identity() got an unexpected keyword argument 'tdim_ind'
INFO      17:52:54 - solver             [x01] [32.0/107.7 26.2/54.8 13.0Gb] Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 923, in run_solver
    corr_vis = solver_machine.run()
  File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 746, in run
    SolveOnly.run(self)
  File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 728, in run
    self.sol_opts, label=self.label)
  File "/usr/local/lib/python3.6/dist-packages/cubical/solver.py", line 147, in _solve_gains
    gm.compute_residual(obser_arr, model_arr, resid_arr, require_full=True)
  File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 362, in compute_residual
    g, gh = self.accumulate_gains()
  File "/usr/local/lib/python3.6/dist-packages/cubical/machines/jones_chain_machine.py", line 290, in accumulate_gains
    g0 = self.jones_terms[0]._gainres_to_fullres(self.jones_terms[0].gains, tdim_ind=1)
TypeError: copy_or_identity() got an unexpected keyword argument 'tdim_ind'

I'm hoping this is possible to save having to write out intermediate visibilities and play musical columns with the MS, but maybe it isn't...

Here is the parset:

[data]
_Help = Visibility data options
ms = 
column = DATA
time-chunk = 48
freq-chunk = 1024
rebin-time = 1
rebin-freq = 1
chunk-by = SCAN_NUMBER
chunk-by-jump = 0
single-chunk = 
single-tile = -1
normalize = 0

[sel]
_Help = Data selection options
field = 0
ddid = None
taql = 
chan = 
diag = False

[out]
_Help = Options for output products
dir = selfcal
name = selfcal2.cc-out/selfcal
overwrite = False
backup = 1
mode = sc
apply-solver-flags = True
column = CORRECTED_DATA
derotate = None
model-column = 
weight-column = 
reinit-column = False
subtract-model = 0
subtract-dirs = 0
correct-dir = 0
plots = 1
casa-gaintables = True

[model]
_Help = Calibration model options
list = MODEL_DATA
ddes = auto
beam-pattern = None
beam-l-axis = None
beam-m-axis = None
feed-rotate = auto
pa-rotate = False

[montblanc]
_Help = Montblanc simulation options
device-type = CPU
dtype = double
mem-budget = 1024
verbosity = WARNING
threads = 0
pa-rotate = None

[weight]
_Help = Weighting options
column = WEIGHT_SPECTRUM
fill-offdiag = False
legacy-v1-2 = False

[flags]
_Help = General flagging options
apply = -cubical
auto-init = legacy
save = cubical
save-legacy = auto
reinit-bitflags = False
warn-thr = 0.3
see-no-evil = 0

[degridding]
_Help = Options for the degridder. Only in use when predicting from DicoModels using DDFacet
OverS = 11
Support = 7
Nw = 100
wmax = 0.0
Padding = 1.7
NDegridBand = 16
MaxFacetSize = 0.25
MinNFacetPerAxis = 1
NProcess = 8
BeamModel = None
NBand = 0
FITSFile = beam_$(corr)_$(reim).fits
FITSFeed = None
FITSFeedSwap = False
DtBeamMin = 5.0
FITSParAngleIncDeg = 5.0
FITSLAxis = -X
FITSMAxis = Y
FITSVerbosity = 0
FeedAngle = 0.0
FlipVisibilityHands = 0

[postmortem]
_Help = Options for "postmortem" flagging based on solution statistics
enable = False
tf-chisq-median = 1.2
tf-np-median = 0.5
time-density = 0.5
chan-density = 0.5
ddid-density = 0.5

[madmax]
_Help = Options for the "Mad Max" flagger
enable = 0
residuals = 0
estimate = corr
diag = True
offdiag = True
threshold = 10
global-threshold = 12
plot = 1
plot-frac-above = 0.01
plot-bl = 
flag-ant = 0
flag-ant-thr = 5

[sol]
_Help = Solution options which apply at the solver level
jones = K,G
precision = 32
delta-g = 1e-06
delta-chi = 1e-06
chi-int = 5
last-rites = True
stall-quorum = 0.99
term-iters = 50,50
flag-divergence = 0
min-bl = 100.0
max-bl = 0
subset = 

[bbc]
_Help = Options for baseline-based corrections (a.k.a. BBCs, a.k.a. interferometer gains).
load-from = 
compute-2x2 = False
apply-2x2 = False
save-to = {out[name]}-BBC-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
per-chan = True
plot = True

[dist]
_Help = Parallelization and distribution options
ncpu = 8
nworker = 0
nthread = 0
max-chunks = 2
min-chunks = 0
pin = 0
pin-io = False
pin-main = io
safe = True

[log]
_Help = Options related to logging
memory = True
stats = chi2:.3f
stats-warn = chi2:10
boring = True
append = False
verbose = 0
file-verbose = None

[debug]
_Help = Debugging options for the discerning masochist
pdb = False
panic-amplitude = 0.0
stop-before-solver = False
escalate-warnings = 0

[misc]
_Help = Miscellaneous options
random-seed = None
parset-version = 0.1

[k]
_Help = Options for K-Jones term
label = K
solvable = 0
type = f-slope
load-from = selfcal.cc-out/selfcal-K-field_0-ddid_None.parmdb
xfer-from = 
save-to = 
dd-term = 0
fix-dirs = 
update-type = full
time-int = 4
freq-int = 1024
max-prior-error = 0.0
max-post-error = 0.0
clip-low = 0.1
clip-high = 10
clip-after = 5
max-iter = 50
epsilon = 1e-06
delta-chi = 1e-06
conv-quorum = 0.99
ref-ant = None
prop-flags = default
estimate-pzd = False
diag-only = 1
offdiag-only = 0
robust-cov = compute
robust-npol = 2
robust-int = 1
robust-save-weights = 0
_Templated = True

[g]
_Help = Options for G-Jones term
label = G
solvable = 1
type = complex-2x2
delay-estimate-pad-factor = 8
load-from = 
xfer-from = 
save-to = {out[name]}-{JONES}-field_{sel[field]}-ddid_{sel[ddid]}.parmdb
dd-term = 0
fix-dirs = 
update-type = full
estimate-pzd = False
time-int = 48
freq-int = 1024
max-prior-error = 0.3
max-post-error = 0.3
low-snr-warn = 75
high-gain-var-warn = 30
clip-low = 0.1
clip-high = 10000
clip-after = 5
max-iter = 50
pin-slope-iters = 0
epsilon = 1e-06
delta-chi = 1e-06
conv-quorum = 0.99
ref-ant = None
prop-flags = default
diag-only = 0
offdiag-only = False
robust-cov = compute
robust-scale = 1
robust-npol = 2
robust-int = 1
robust-save-weights = 0
estimate-delays = False
_Templated = True

[o-smirnov]

It should b a simple fix, I'll do it tomorrow...

[o-smirnov]

@IanHeywood could you check the issue-428 branch please?

I know how you got into uncharted waters here: my own workflow has been to use a standard complex-2x2 gain with --load-from and intervals of 1,1. It is a little-documented but cool feature that the gain tables generated by f-slope can also be loaded by a standard gain machine... whereas you took the far more obvious (to the non-developer) route of using the same kind of f-slope machine to load the gains into, and that's where the bug lurked.

Anyway please carry on with --k-type f-slope --k-load-from, as this really needs to be tested properly, but know that you have a working fallback.

[IanHeywood]

Just to clarify: Will using 1,1 and --load-from apply the nearest solution in time/freq?

Interpolation requires --xfer-from?

[o-smirnov]

Not exactly. -load-from will throw an error if the grids (by which I mean the time/frequency intervals derived from the underlying data grid) don't match the table exactly. The idea was that you would use -load-from when you're sure that you intend to be using solutions from exactly the same field, as is, no interpolation required. (And so you presumably do want to see an error message if this is not the case, because that means something went screwy. At least that was the idea).

-xfer-from will happily interpolate all over the place, no matching grids needed.

One wrinkle here is that slope solutions with timeint=N, freqint=0 will result in a gain solutions table with a grid of timeint=N, freqint=1 (since it unrolls the frequency slope implicitly.) Hence, intervals of 1,1 are used above for loading (assuming 1,0 was used for solving.)