Table DataManager error: Invalid operation: TSM: no array in row 0 of column FLAG_CATEGORY

[Athanaseus]

Describe the bug After installing the main branch, running goquartica l give below error.

Config description (if applicable)

input_ms:
    path: /home/ramaila/working/PicA/msdir/PicA-PictorA-avg_corr.ms
    data_column: DATA
    sigma_column: SIGMA_SPECTRUM
    time_chunk: '300s'
    freq_chunk: '0'
input_model:
    recipe: MODEL_DATA
solver:
    terms:
        - G
    iter_recipe:
        - 25
output:
    directory: outputs.qc
    overwrite: false
    products:
        - corrected_data
        - corrected_weight
    columns:
        - CORRECTED_DATA
        - WEIGHT_SPECTRUM
dask:
    threads: 32
G:
    type: phase
    time_interval: '32s'
    freq_interval: '256'

To Reproduce goquartical f_slope.yml

Screenshots

2022-05-23 09:49:08 | INFO | preprocess:transcribe_recipe | The following model sources were obtained from --input-model-recipe: 
   Columns: {'MODEL_DATA'} 
   Sky Models: None
2022-05-23 09:49:09 | INFO | ms_handler:read_xds_list | Antenna table indicates 62 antennas were present for this observation.
2022-05-23 09:49:09 | INFO | ms_handler:read_xds_list | Polarization table indicates 4 correlations are present in the measurement set - ['XX', 'XY', 'YX', 'YY'].
2022-05-23 09:49:09 | INFO | ms_handler:read_xds_list | Field table indicates phase centre is at (1.3955158188991927 -0.7989918542020786).
2022-05-23 09:49:12 | WARNING | reads:_group_datasets | Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 0 of column FLAG_CATEGORY in /home/ramaila/working/PicA/msdir/PicA-PictorA-avg_corr.ms/table.f18'
2022-05-23 09:49:14 | ERROR | goquartical:<module> | An error has been caught in function '<module>', process 'MainProcess' (16739), thread 'MainThread' (140549799155520):
Traceback (most recent call last):

> File "/home/ramaila/.virtualenvs/skothane/bin/goquartical", line 11, in <module>
    load_entry_point('quartical', 'console_scripts', 'goquartical')()
    └ <function load_entry_point at 0x7fd44a8b8a60>

  File "/home/ramaila/gitPackages/QuartiCal/quartical/executor.py", line 34, in execute
    _execute(stack)
    │        └ <contextlib.ExitStack object at 0x7fd44b7b9ee0>
    └ <function _execute at 0x7fd2f27243a0>

  File "/home/ramaila/gitPackages/QuartiCal/quartical/executor.py", line 102, in _execute
    data_xds_list, ref_xds_list = read_xds_list(model_columns, ms_opts)
                                  │             │              └ MSInputs(path='/home/ramaila/working/PicA/msdir/PicA-PictorA-avg_corr.ms', data_column='DATA', sigma_column='SIGMA_SPECTRUM',...
                                  │             └ {'MODEL_DATA'}
                                  └ <function read_xds_list at 0x7fd3031168b0>

  File "/home/ramaila/gitPackages/QuartiCal/quartical/data_handling/ms_handler.py", line 82, in read_xds_list
    assert all(c in available_columns for c in columns), \
                    │                          └ ('TIME', 'INTERVAL', 'ANTENNA1', 'ANTENNA2', 'FEED1', 'FEED2', 'FLAG', 'FLAG_ROW', 'UVW', 'DATA', 'SIGMA_SPECTRUM', 'MODEL_DA...
                    └ ['DATA', 'FLAG_ROW', 'BITFLAG', 'PROCESSOR_ID', 'ARRAY_ID', 'CAL_HOTSPOT_MODEL_DATA', 'SIGMA', 'CORRECTED_HOTSPOT_DATA', 'INT...

AssertionError: One or more columns in: ('TIME', 'INTERVAL', 'ANTENNA1', 'ANTENNA2', 'FEED1', 'FEED2', 'FLAG', 'FLAG_ROW', 'UVW', 'DATA', 'SIGMA_SPECTRUM', 'MODEL_DATA') is not present in the data.

Version Ubuntu-20.04 Quartical-0.1.1 Python3.8

[JSKenyon]

This likely means you don't have a SIGMA_SPECTRUM column (or possibly MODEL_DATA?). Could you please check?

[Athanaseus]

Yeah you are tight no SIGMA_SPECTRUM just SIGMA.

In [3]: t.colnames()
Out[3]: 
['UVW',
 'FLAG',
 'FLAG_CATEGORY',
 'WEIGHT',
 'SIGMA',
 'ANTENNA1',
 'ANTENNA2',
 'ARRAY_ID',
 'DATA_DESC_ID',
 'EXPOSURE',
 'FEED1',
 'FEED2',
 'FIELD_ID',
 'FLAG_ROW',
 'INTERVAL',
 'OBSERVATION_ID',
 'PROCESSOR_ID',
 'SCAN_NUMBER',
 'STATE_ID',
 'TIME',
 'TIME_CENTROID',
 'DATA',
 'WEIGHT_SPECTRUM',
 'MODEL_DATA',
 'BITFLAG',
 'BITFLAG_ROW',
 'CORRECTED_DATA',
 'HOTPOT_MODEL_DATA',
 'CORRECTED_HOTSPOT_DATA',
 'CAL_HOTSPOT_MODEL_DATA',
 'CORRECTED_RES_DATA']

[JSKenyon]

So in this case, you may want to use WEIGHT_SPECTRUM instead. QuartiCal has two different, mutually exclusive weight parameters. input_ms.sigma_column will recompute the weights from a SIGMA or SIGMA_SPECTRUM style column, whereas input_ms.weight_column will simply use the specified column directly. Just depends on what you have available.

[JSKenyon]

Oh, and for reference, this message:

2022-05-23 09:49:12 | WARNING | reads:_group_datasets | Ignoring 'FLAG_CATEGORY': Unable to infer shape of column 'FLAG_CATEGORY' due to:
'Table DataManager error: Invalid operation: TSM: no array in row 0 of column FLAG_CATEGORY in /home/ramaila/working/PicA/msdir/PicA-PictorA-avg_corr.ms/table.f18'

is not the error. It is just a warning emitted by dask-ms. It means that although the column is present in your data, it is empty and cannot be read. Nothing to worry about.

[JSKenyon]

Closing for now - the error message was correct, but I may make it clearer at some point.