I/O handler for load 0 save None failed with exception: Expected a bounding hull that is at least partially within the image

o-smirnov commented 3 years ago

CubiCal hangs with this error message. Two separate issues here:

If the I/O handler dies unexpectedly (or any of the subprocesses, for that matter), the main process should exit and not just hang indefinitely (that's for me to look at)
@bennahugo can this particular error (region outside model) be caught and reported up front? It's an innocent (and easy) user mistake, so would be nice to have a friendlier error.

viralp commented 3 years ago

Hi, @o-smirnov I am also getting the following error -

INFO      13:49:19 - main               [x01] [6.6/8.8 14.2/23.9 66.9Gb] I/O handler for load 0 save None failed with exception: Table DataManager error: Invalid operation: SSMIndColumn::getShape: no array in row 384905 in column MODEL_DATA of table /data4/vparekh/GCLS_data/J0943.4-7619/file.ms
INFO      13:49:19 - main               [x01] [6.6/8.8 14.2/23.9 66.9Gb] Traceback (most recent call last):
  File "/home/vparekh/CubiCal/cubical/workers.py", line 469, in _io_handler
    tile.load(load_model=load_model)
  File "/home/vparekh/CubiCal/cubical/data_handler/ms_tile.py", line 973, in load
    model0 = self.dh.fetchslice(model_source, subset=table_subset)
  File "/home/vparekh/CubiCal/cubical/data_handler/ms_data_handler.py", line 842, in fetchslice
    subset.getcolnp(str(column), prealloc, startrow, nrows)
  File "/home/vparekh/venv/lib/python3.6/site-packages/casacore/tables/table.py", line 1052, in getcolnp
    return self._getcolvh(columnname, startrow, nrow, rowincr, nparray)
RuntimeError: Table DataManager error: Invalid operation: SSMIndColumn::getShape: no array in row 384905 in column MODEL_DATA of table /data4/vparekh/GCLS_data/J0943.4-7619/file.ms

ddcal_0.log

JSKenyon commented 3 years ago

That looks like a broken MS to me. You can check by doing:

import pyrap.tables as pt

ms = pt.table("ms_name")

ms.getcol("MODEL_DATA", startrow=384904, nrow=10)

If that dies it means portions of the MODEL_DATA column don't exist. It could also be a selection/TAQL bug I guess.

viralp commented 3 years ago

@JSKenyon I got following

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/casacore/tables/table.py", line 1032, in getcol
    return self._getcol(columnname, startrow, nrow, rowincr)
RuntimeError: Table DataManager error: Invalid operation: SSMIndColumn::getShape: no array in row 384905 in column MODEL_DATA of table /data4/vparekh/GCLS_data/J0943.4-7619/file.ms

Actually, this ms file initially processed in the AIPS and later converted to ms file. I ran first DDFacet with --Predict-ColName=MODEL_DATA

JSKenyon commented 3 years ago

Yeah, that means that something is wrong with the MS. It boils down to MODEL_DATA containing empty rows. If MODEL_DATA was generated using DDF, it is possible that something went wrong during the predict. @o-smirnov any thoughts?

o-smirnov commented 3 years ago

Can you look at row 384905 in e.g. casabrowser. What's so special about it -- is it a different field? An autocorrelation? Etc. etc.

One way or another, indeed the MODEL_DATA column in that row did not get populated. We just need to figure out why.

ratt-ru / CubiCal

I/O handler for load 0 save None failed with exception: Expected a bounding hull that is at least partially within the image #435