chain machine: ValueError: could not broadcast input array from shape foo into shape bar

[KshitijT]

I am trying to reduce a dataset with the same options (more or less - added --out-casa-gaintables=0) since I was getting an error when Cubical tried to export to CASA gaintables.) Now get a new error:

Traceback (most recent call last):
  File "/home/kshitij/meerkathi-for-gmrt/meerkathi/meerkathi-for-gmrt-venv/local/lib/python2.7/site-packages/cubical/main.py", line 353, in main
    stats_dict = workers.run_process_loop(ms, load_model, single_chunk, solver_type, solver_opts, debug_opts)
  File "/home/kshitij/meerkathi-for-gmrt/meerkathi/meerkathi-for-gmrt-venv/local/lib/python2.7/site-packages/cubical/workers.py", line 201, in run_process_loop
    return _run_multi_process_loop(ms, load_model, solver_type, solver_opts, debug_opts)
  File "/home/kshitij/meerkathi-for-gmrt/meerkathi/meerkathi-for-gmrt-venv/local/lib/python2.7/site-packages/cubical/workers.py", line 268, in _run_multi_process_loop
    stats = future.result()
  File "/home/kshitij/meerkathi-for-gmrt/meerkathi/meerkathi-for-gmrt-venv/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result
    return self.__get_result()
  File "/home/kshitij/meerkathi-for-gmrt/meerkathi/meerkathi-for-gmrt-venv/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 414, in __get_result
    raise exception_type, self._exception, self._traceback
ValueError: could not broadcast input array from shape (1,2,1,61,2,2) into shape (4,22,256,61,2,2)

[JSKenyon]

This is probably related to #187. Poking @o-smirnov and @bennahugo for thoughts. @KshitijT, could you please post your parset or make it available to us. If the data is a manageable size, it might be helpful to give us access.

[o-smirnov]

@KshitijT, this is on a Rhodes machine, correct? Please tell me where, I'd like to run it as you and poke around.

[KshitijT]

@JSKenyon , the cubical options I used are here:

ocubical --data-ms msdir/1524947605_cont_sm-corr.ms\
          --out-column SC_NEW_DATA\
          --data-column DATA\
          --sol-min-bl  1100.0\
          --model-list 'input/lsm-for-de-test.lsm.html@dE+MODEL_DATA'\
          --model-ddes auto\
          --weight-column WEIGHT_SPECTRUM\
          --sol-jones g,dE\
          --out-subtract-dirs 1:\
          --data-freq-chunk 256\
          --data-time-chunk 90\
          --de-time-int 90\
          --de-freq-int 256\
          --de-type complex-2x2\
          --de-fix-dirs 0\
          --de-save-to '{data[ms]}/{JONES}-field:{sel[field]}-ddid:{sel[ddid]}-rnd1.parmdb'\
          --g-save-to '{data[ms]}/{JONES}-field:{sel[field]}-ddid:{sel[ddid]}-rnd1.parmdb'\
          --g-solvable 1\
          --g-update-type phase-diag\
          --g-time-int 15\
          --g-freq-int 1000000\
          --de-dd-term 1\
          --g-dd-term 0\
          --sel-field 0\
          --g-max-iter 20\
          --flags-auto-init legacy\
          --out-mode sr\
          --log-verbose solver=2\
          --dist-min-chunks 1\
          --dist-max-chunks 2\
          --dist-nworker 2\
          --dist-nthread 20\
          --dist-ncpu 30\
          --out-casa-gaintables=0\
          --out-name ddcal_take1

@o-smirnov , I have sent you the details of the data location.

[o-smirnov]

Not #187, no, but I confirm the error. Something in the chain, related to my recent modifications no doubt... gimme some time to dig...

[o-smirnov]

Problem is here: https://github.com/ratt-ru/CubiCal/blob/master/cubical/machines/jones_chain_machine.py#L252

Already fixed on my issue-151 and madmax branches, so probably no point fixing master, as the PR is looming. I've already told @KshitijT to try the madmax branch.