search

tart-telescope / notebooks

Example python notebooks using TART data
0 stars 2 forks source link

CASA RuntimeError when reading MS created by tart2ms #3

Open rubyvanrooyen opened 2 years ago

rubyvanrooyen commented 2 years ago

Although the error does not occur in wsclean anymore, I cannot get it working in CASA directly. I keep getting the same error when trying to read the tables, but cannot easily find anywhere that may cause the issue.

CASA return RuntimeError: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID

Full implementation and exception trace can be found in notebook: https://github.com/rubyvanrooyen/notebooks/blob/main/application/CASA_TART_workspace.ipynb

Error description from CASA log

2022-10-17 06:43:45 INFO    ::casa  
2022-10-17 06:43:46 INFO    ::casa  Python version 3.7.14
2022-10-17 06:43:46 INFO    ::casa  CASA Version MODULAR 6.5.2.26
2022-10-17 06:43:46 INFO    ::casa  Can't import casatelemetry module.
2022-10-17 06:45:10 INFO    listobs::::casa ##########################################
2022-10-17 06:45:10 INFO    listobs::::casa ##### Begin Task: listobs            #####
2022-10-17 06:45:10 INFO    listobs::::casa listobs( vis='data_test.ms', selectdata=True, spw='', field='', antenna='', uvrange='', timerange='', correlation='', scan='', intent='', feed='', array='', observation='', verbose=True, listfile='', listunfl=False, cachesize=50.0, overwrite=False )
2022-10-17 06:45:10 WARN    listobs::ms::msselect   Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 06:45:10 SEVERE  listobs::ms::summary    ================================================================================
2022-10-17 06:45:10 SEVERE  listobs::ms::summary+              MeasurementSet Name:  /content/data_test.ms      MS Version 2
2022-10-17 06:45:10 SEVERE  listobs::ms::summary+   ================================================================================
2022-10-17 06:45:10 SEVERE  listobs::ms::summary+   Exception Reported: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 06:45:10 SEVERE  listobs::::casa Task listobs raised an exception of class RuntimeError with the following message: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 06:45:10 INFO    listobs::::casa Task listobs complete. Start time: 2022-10-17 06:45:09.830891 End time: 2022-10-17 06:45:09.846064
2022-10-17 06:45:10 INFO    listobs::::casa ##### End Task: listobs              #####
2022-10-17 06:45:10 INFO    listobs::::casa ##########################################
rubyvanrooyen commented 2 years ago

Checking various versioning issues that may cause the error: COLAB currently uses Python 3.7.14 (this cannot be changed by the user)

Checking implementation on local server

python3.7 -m venv env37 (Python 3.7.13)
pip install --upgrade pip wheel

pip install requests
pip install fsspec
pip install dask-ms[xarray]
pip install python-casacore
pip install astropy
pip install tart tart-tools tart2ms

Noting versions installed: dask-2022.2.0 dask-ms-0.2.9 python-casacore-3.5.2 astropy-4.3.1 tart-1.1.0b9 tart-tools-1.1.0b9 tart2ms-0.2.1b4

There is a small issue with one of the astropy libs in the conversion

tart2ms --json data_test.json --ms data_test.ms

Getting Data from file: data_test.json
Writing measurement set 'data_test.ms'...
2022-10-17 11:03:38,406 - root - INFO - Time 5071671511.991
/home/ruby/wipeme/env37/lib/python3.7/site-packages/astropy/time/utils.py:179: RuntimeWarning: overflow encountered in double_scalars
  c = 134217729. * a  # 2**27+1.
/home/ruby/wipeme/env37/lib/python3.7/site-packages/astropy/time/utils.py:181: RuntimeWarning: invalid value encountered in double_scalars
  ah = c - abig
Downloading https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat
|==========================================================================================================================| 1.3k/1.3k (100.00%)         0s
Downloading https://www.ietf.org/timezones/data/leap-seconds.list [Done]
2022-10-17 11:03:40,708 - root - INFO - Pol Feeds ['RR']
2022-10-17 11:03:40,708 - root - INFO - Correlation Types [[5]]
2022-10-17 11:03:40,721 - root - INFO - Pol Types ['RR']
2022-10-17 11:03:40,721 - root - INFO - Pol Responses [[0.7071067811865475, (-0-0.7071067811865475j)]]
2022-10-17 11:03:40,996 - root - INFO - Corr Prod [[0 0]]
2022-10-17 11:03:40,996 - root - INFO - Corr Type [5]
2022-10-17 11:03:40,996 - root - INFO - NUM_CORR dask.array<full_like, shape=(1,), dtype=int32, chunksize=(1,), chunktype=numpy.ndarray>
2022-10-17 11:03:40,997 - root - INFO - Dask Corr Prod (1, 1, 2)
2022-10-17 11:03:40,997 - root - INFO - Dask Corr Type (1, 1)
2022-10-17 11:03:40,998 - root - INFO - Data size 276 1 1
Measurement set writing complete.

Using system CASA installation (CASA Version 5.6.0-60) gives similar error as noted in COLAB

CASA <2>: msfile='data_test.ms'

CASA <3>: listobs(msfile)
2022-10-17 09:06:44 INFO    listobs::::
2022-10-17 09:06:44 INFO    listobs::::+    ##########################################
2022-10-17 09:06:44 INFO    listobs::::+    ##### Begin Task: listobs            #####
2022-10-17 09:06:44 INFO    listobs:::: listobs(vis="data_test.ms",selectdata=True,spw="",field="",antenna="",
2022-10-17 09:06:44 INFO    listobs::::+            uvrange="",timerange="",correlation="",scan="",intent="",
2022-10-17 09:06:44 INFO    listobs::::+            feed="",array="",observation="",verbose=True,listfile="",
2022-10-17 09:06:44 INFO    listobs::::+            listunfl=False,cachesize=50,overwrite=False)
2022-10-17 09:06:44 WARN    listobs::ms::msselect   Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE  listobs::ms::summary    ================================================================================
2022-10-17 09:06:44 SEVERE  listobs::ms::summary+              MeasurementSet Name:  /home/ruby/wipeme/data_test.ms      MS Version 2
2022-10-17 09:06:44 SEVERE  listobs::ms::summary+   ================================================================================
2022-10-17 09:06:44 SEVERE  listobs::ms::summary+   Exception Reported: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE  listobs:::: *** Error *** Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 INFO    listobs:::: ##### End Task: listobs              #####
2022-10-17 09:06:44 INFO    listobs::::+    ##########################################
Out[3]: False
rubyvanrooyen commented 2 years ago

Checking implementation using server default 3.8

python3 -m venv env (Python 3.8.10)
pip install --upgrade pip wheel
pip install requests
pip install fsspec
pip install dask-ms[xarray]
pip install python-casacore
pip install astropy
pip install tart tart-tools tart2ms

Noting versions installed: dask-ms-2022.8.2 python-casacore-3.5.2 astropy-5.1 tart-1.1.0b9 tart-tools-1.1.0b9 tart2ms-0.2.1b4

Updated astropy doesn't have the error for libs in python3.7

tart2ms --json data_test.json --ms data_test.ms

Getting Data from file: data_test.json
Writing measurement set 'data_test.ms'...
2022-10-17 11:15:26,338 - root - INFO - Time 5071671511.991
2022-10-17 11:15:26,888 - root - INFO - Pol Feeds ['RR']
2022-10-17 11:15:26,888 - root - INFO - Correlation Types [[5]]
2022-10-17 11:15:26,921 - root - INFO - Pol Types ['RR']
2022-10-17 11:15:26,921 - root - INFO - Pol Responses [[0.7071067811865475, (-0-0.7071067811865475j)]]
2022-10-17 11:15:27,197 - root - INFO - Corr Prod [[0 0]]
2022-10-17 11:15:27,197 - root - INFO - Corr Type [5]
2022-10-17 11:15:27,197 - root - INFO - NUM_CORR dask.array<full_like, shape=(1,), dtype=int32, chunksize=(1,), chunktype=numpy.ndarray>
2022-10-17 11:15:27,198 - root - INFO - Dask Corr Prod (1, 1, 2)
2022-10-17 11:15:27,198 - root - INFO - Dask Corr Type (1, 1)
2022-10-17 11:15:27,199 - root - INFO - Data size 276 1 1
Measurement set writing complete.

Using system CASA installation (CASA Version 5.6.0-60) gives similar error as noted in COLAB and python3.7

CASA <2>: msfile='data_test.ms'

CASA <3>: listobs(msfile)
2022-10-17 09:06:44 INFO    listobs::::
2022-10-17 09:06:44 INFO    listobs::::+    ##########################################
2022-10-17 09:06:44 INFO    listobs::::+    ##### Begin Task: listobs            #####
2022-10-17 09:06:44 INFO    listobs:::: listobs(vis="data_test.ms",selectdata=True,spw="",field="",antenna="",
2022-10-17 09:06:44 INFO    listobs::::+            uvrange="",timerange="",correlation="",scan="",intent="",
2022-10-17 09:06:44 INFO    listobs::::+            feed="",array="",observation="",verbose=True,listfile="",
2022-10-17 09:06:44 INFO    listobs::::+            listunfl=False,cachesize=50,overwrite=False)
2022-10-17 09:06:44 WARN    listobs::ms::msselect   Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE  listobs::ms::summary    ================================================================================
2022-10-17 09:06:44 SEVERE  listobs::ms::summary+              MeasurementSet Name:  /home/ruby/wipeme/data_test.ms      MS Version 2
2022-10-17 09:06:44 SEVERE  listobs::ms::summary+   ================================================================================
2022-10-17 09:06:44 SEVERE  listobs::ms::summary+   Exception Reported: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE  listobs:::: *** Error *** Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 INFO    listobs:::: ##### End Task: listobs              #####
2022-10-17 09:06:44 INFO    listobs::::+    ##########################################
Out[3]: False
rubyvanrooyen commented 2 years ago

Focusing on CASA and CASA tools versions, installing latest wheel version available

pip install casadata==2022.9.5
pip install casatools==6.5.2.26
pip install casatasks==6.5.2.26
pip install casaplotms==1.8.7
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from casatasks import listobs
>>> msfile = "data_test.ms"
>>> listobs(vis=msfile)
2022-10-17 09:31:09 WARN    listobs::ms::msselect   Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:31:09 SEVERE  listobs::ms::summary    ================================================================================
2022-10-17 09:31:09 SEVERE  listobs::ms::summary+              MeasurementSet Name:  /home/ruby/wipeme/data_test.ms      MS Version 2
2022-10-17 09:31:09 SEVERE  listobs::ms::summary+   ================================================================================
2022-10-17 09:31:09 SEVERE  listobs::ms::summary+   Exception Reported: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:31:09 SEVERE  listobs::::casa Task listobs raised an exception of class RuntimeError with the following message: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatasks/listobs.py", line 159, in __call__
    task_result = _listobs_t( _pc.document['vis'], _pc.document['selectdata'], _pc.document['spw'], _pc.document['field'], _pc.document['antenna'], _pc.document['uvrange'], _pc.document['timerange'], _pc.document['correlation'], _pc.document['scan'], _pc.document['intent'], _pc.document['feed'], _pc.document['array'], _pc.document['observation'], _pc.document['verbose'], _pc.document['listfile'], _pc.document['listunfl'], _pc.document['cachesize'], _pc.document['overwrite'] )
  File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatasks/private/task_listobs.py", line 59, in listobs
    obs_dict = myms.summary(
  File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatools/ms.py", line 577, in summary
    return _dict_dc(self._swigobj.summary(verbose, _str_ec(listfile), listunfl, cachesize, overwrite, wantreturn))
  File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatools/__casac__/ms.py", line 1401, in summary
    return _ms.ms_summary(self, *args, **kwargs)
RuntimeError: Invalid Table data type when accessing column  in ScalarColumn ctor for column TIME_CENTROID
rubyvanrooyen commented 2 years ago

@tmolteno I have verified all the package versions that I think are important and made sure I am using the newest versions, but still no joy Please can you verify which versions you are using, just incase I have missed something.

tmolteno commented 2 years ago

I will have a look at the TIME_CENTROID column in tart2ms. This may well have a similar problem to the TIME column which was of the incorrect order...