convert_msv2_to_processing_set SPECTRUM AttributeError

[pford]

Version: xradio-0.0.33

convert_msv2_to_processing_set. gets an AttributeError when trying to convert several MSv2s to zarr processing sets. I used the default partition_scheme. Only 3 of 9 test datasets were successfully converted with this version of xradio (previous conversions worked). One example is a casaguides msv2 downloaded from http://casa.nrao.edu/Data/EVLA/SNRG55/SNR_G55_10s.tar.gz.

File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/convert_msv2_to_processing_set.py", line 100, in convert_msv2_to_processing_set convert_and_write_partition( File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 763, in convert_and_write_partition np.ones(xds.SPECTRUM.shape, dtype=np.float64), ^^^^^^^^^^^^ File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xarray/core/common.py", line 286, in getattr raise AttributeError( AttributeError: 'Dataset' object has no attribute 'SPECTRUM'

[Jan-Willem]

@pford The issue is that SNR_G55_10s.tar.gz has a WEIGHT_SPECTRUM column but it is invalid. I have added a fix and stakeholder test based on SNR_G55_10s.ms. Can you please check if branch 210-convert_msv2_to_processing_set-spectrum-attributeerror works for your datasets?

[pford]

File "/home/groot/casa/github/casagui/casagui/io/_vis_io.py", line 41, in get_processing_set convert_msv2_to_processing_set( File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/convert_msv2_to_processing_set.py", line 100, in convert_msv2_to_processing_set convert_and_write_partition( File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 773, in convert_and_write_partition create_data_variables( File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 596, in create_data_variables xds = get_weight( ^^^^^^^^^^^ File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 624, in get_weight create_attribute_metadata(col, main_column_descriptions) ^^^^^^^^^^^^^^^^^^^^^^^^ NameError: name 'main_column_descriptions' is not defined

[Jan-Willem]

Is this using SNR_G55_10s and what parameters are you giving the convert function?

[pford]

Using SNR_G55_10s.ms and setting in_file and out_file parameters only. main_column_descriptions is not passed in or set in get_weight.

[pford]

Full traceback: Traceback (most recent call last): File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 572, in create_data_variables read_col_conversion( File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/_tables/read.py", line 1243, in read_col_conversion col_dtype = np.array(tb_tool.col(col)[0]).dtype

  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/casacore/tables/tablecolumn.py", line 270, in __getitem__
    return self.getcell(sei[0])
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/casacore/tables/tablecolumn.py", line 127, in getcell
    return self._table.getcell(self._column, rownr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/casacore/tables/table.py", line 959, in getcell
    return self._getcell(columnname, rownr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Table DataManager error: Invalid operation: TSM: no array in row 0 of column WEIGHT_SPECTRUM in /home/groot/tarballs/SNR_G55_10s.ms/table.f19

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/groot/casa/github/casagui/casagui/plots/_vis_raster.py", line 42, in __init__
    self._ps, self._vis_path = get_processing_set(vis)
                               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/groot/casa/github/casagui/casagui/io/_vis_io.py", line 41, in get_processing_set
    convert_msv2_to_processing_set(
  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/convert_msv2_to_processing_set.py", line 100, in convert_msv2_to_processing_set
    convert_and_write_partition(
  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 773, in convert_and_write_partition
    create_data_variables(
  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 596, in create_data_variables
    xds = get_weight(
          ^^^^^^^^^^^
  File "/home/groot/anaconda3/envs/gui_env/lib/python3.11/site-packages/xradio/vis/_vis_utils/_ms/conversion.py", line 624, in get_weight
    create_attribute_metadata(col, main_column_descriptions)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'main_column_descriptions' is not defined

[Jan-Willem]

It is strange that this error is not occurring for the split version of SNR_G55_10s.ms used in the tests. I have pushed a fix.

[pford]

I was able to convert all of my MS datasets. Thanks @Jan-Willem!

[Jan-Willem]

Thank you for your patience and checking.