running gocubical on test/SUBSET-D147-output.MS gives Table array conformance error with casacore 3.1+

[gijzelaerr]

While redoing the py3 porting work, I encountered this error when running gocubical on test/SUBSET-D147-output.MS both with py2 and py3:

 - 15:10:43 - casa_db_adaptor    [0.2/0.4 1.0/1.3 0.2Gb] Exporting to CASA gaintables
 - 15:10:54 - main               [0.2/0.4 1.0/1.3 0.2Gb] Exiting with exception: RuntimeError(ArrayColumn::putColumn for column OFFSET: Table array conformance error)
 Traceback (most recent call last):
  File "/home/gijs/Work/CubiCal/cubical/main.py", line 479, in main
    stats_dict = workers.run_process_loop(ms, tile_list, load_model, single_chunk, solver_type, solver_opts, debug_opts)
  File "/home/gijs/Work/CubiCal/cubical/workers.py", line 211, in run_process_loop
    return _run_single_process_loop(ms, load_model, single_chunk, solver_type, solver_opts, debug_opts)
  File "/home/gijs/Work/CubiCal/cubical/workers.py", line 363, in _run_single_process_loop
    solver.gm_factory.close()
  File "/home/gijs/Work/CubiCal/cubical/machines/abstract_machine.py", line 862, in close
    db.close()
  File "/home/gijs/Work/CubiCal/cubical/database/casa_db_adaptor.py", line 467, in close
    self.__export()
  File "/home/gijs/Work/CubiCal/cubical/database/casa_db_adaptor.py", line 452, in __export
    casa_caltable_factory.create_B_table(self, "G:gain")
  File "/home/gijs/Work/CubiCal/cubical/database/casa_db_adaptor.py", line 242, in create_B_table
    viscal_label="B Jones" if diag else "D Jones")
  File "/home/gijs/Work/CubiCal/cubical/database/casa_db_adaptor.py", line 58, in init_empty
    t.putcol("OFFSET", db.antoffset[antorder])
  File "/home/gijs/Work/CubiCal/.venv2/local/lib/python2.7/site-packages/casacore/tables/table.py", line 1154, in putcol
    self._putcol(columnname, startrow, nrow, rowincr, value)
RuntimeError: ArrayColumn::putColumn for column OFFSET: Table array conformance error

branch: https://github.com/ratt-ru/CubiCal/tree/py3_v2

note that this is using the unicode PR's for casacore and python-casascore.

[bennahugo]

what version of casacore and python-casacore?

[bennahugo]

input parameters and full log?

[bennahugo]

It used to run though the default test

[gijzelaerr]

note that this is using the unicode PR's for casacore and python-casascore.

input parameters are defaults, just running gocubical --data-ms test/SUBSET-D147-output.MS --model-list DATA --out-subtract-dirs 0

https://gist.github.com/gijzelaerr/8bd2bfe884d6157138da327938b7d8d9

[bennahugo]

ok can't track down the exact cause right now, but you need to set your solution intervals etc This fires the test on the CI server, see the accompanying parset https://github.com/ratt-ru/CubiCal/blob/master/test/d147_test.py https://github.com/ratt-ru/CubiCal/blob/master/test/d147-test.parset

docker run --rm -m 100g --cap-add sys_ptrace \
                   --memory-swap=-1 \
                   --shm-size=150g \
                   --rm=true \
                   --name=cubical$BUILD_NUMBER \
                   -v ${TEST_OUTPUT_DIR}:/workspace \
                   -v ${TEST_OUTPUT_DIR}:/root/tmp \
                   --entrypoint /bin/bash \
                   cubical:${BUILD_NUMBER} \
                   -c "cd /src/cubical && apt-get install -y git && pip install -r requirements.test.txt && nosetests --with-xunit --xunit-file /workspace/nosetests.xml test"

[IanHeywood]

Think this is the same one I encountered: https://github.com/ratt-ru/CubiCal/issues/258

[o-smirnov]

But why does this work on master? Same solution intervals etc. there...

[gijzelaerr]

This is with the latest casacore (3.1+), so I guess they became more strict in checking for flaky data.

[bennahugo]

These are with older gain tables (which can't be dumped outside of CASA, so there is a blank table inside the repo that is filled out with data). It may be that the new casacore cannot read these tables properly.

[bennahugo]

It is not a critical issue so if it breaks it breaks and we'll fix it later