Open rubyvanrooyen opened 2 years ago
Checking various versioning issues that may cause the error: COLAB currently uses Python 3.7.14 (this cannot be changed by the user)
Checking implementation on local server
python3.7 -m venv env37 (Python 3.7.13)
pip install --upgrade pip wheel
pip install requests
pip install fsspec
pip install dask-ms[xarray]
pip install python-casacore
pip install astropy
pip install tart tart-tools tart2ms
Noting versions installed: dask-2022.2.0 dask-ms-0.2.9 python-casacore-3.5.2 astropy-4.3.1 tart-1.1.0b9 tart-tools-1.1.0b9 tart2ms-0.2.1b4
There is a small issue with one of the astropy libs in the conversion
tart2ms --json data_test.json --ms data_test.ms
Getting Data from file: data_test.json
Writing measurement set 'data_test.ms'...
2022-10-17 11:03:38,406 - root - INFO - Time 5071671511.991
/home/ruby/wipeme/env37/lib/python3.7/site-packages/astropy/time/utils.py:179: RuntimeWarning: overflow encountered in double_scalars
c = 134217729. * a # 2**27+1.
/home/ruby/wipeme/env37/lib/python3.7/site-packages/astropy/time/utils.py:181: RuntimeWarning: invalid value encountered in double_scalars
ah = c - abig
Downloading https://hpiers.obspm.fr/iers/bul/bulc/Leap_Second.dat
|==========================================================================================================================| 1.3k/1.3k (100.00%) 0s
Downloading https://www.ietf.org/timezones/data/leap-seconds.list [Done]
2022-10-17 11:03:40,708 - root - INFO - Pol Feeds ['RR']
2022-10-17 11:03:40,708 - root - INFO - Correlation Types [[5]]
2022-10-17 11:03:40,721 - root - INFO - Pol Types ['RR']
2022-10-17 11:03:40,721 - root - INFO - Pol Responses [[0.7071067811865475, (-0-0.7071067811865475j)]]
2022-10-17 11:03:40,996 - root - INFO - Corr Prod [[0 0]]
2022-10-17 11:03:40,996 - root - INFO - Corr Type [5]
2022-10-17 11:03:40,996 - root - INFO - NUM_CORR dask.array<full_like, shape=(1,), dtype=int32, chunksize=(1,), chunktype=numpy.ndarray>
2022-10-17 11:03:40,997 - root - INFO - Dask Corr Prod (1, 1, 2)
2022-10-17 11:03:40,997 - root - INFO - Dask Corr Type (1, 1)
2022-10-17 11:03:40,998 - root - INFO - Data size 276 1 1
Measurement set writing complete.
Using system CASA installation (CASA Version 5.6.0-60) gives similar error as noted in COLAB
CASA <2>: msfile='data_test.ms'
CASA <3>: listobs(msfile)
2022-10-17 09:06:44 INFO listobs::::
2022-10-17 09:06:44 INFO listobs::::+ ##########################################
2022-10-17 09:06:44 INFO listobs::::+ ##### Begin Task: listobs #####
2022-10-17 09:06:44 INFO listobs:::: listobs(vis="data_test.ms",selectdata=True,spw="",field="",antenna="",
2022-10-17 09:06:44 INFO listobs::::+ uvrange="",timerange="",correlation="",scan="",intent="",
2022-10-17 09:06:44 INFO listobs::::+ feed="",array="",observation="",verbose=True,listfile="",
2022-10-17 09:06:44 INFO listobs::::+ listunfl=False,cachesize=50,overwrite=False)
2022-10-17 09:06:44 WARN listobs::ms::msselect Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE listobs::ms::summary ================================================================================
2022-10-17 09:06:44 SEVERE listobs::ms::summary+ MeasurementSet Name: /home/ruby/wipeme/data_test.ms MS Version 2
2022-10-17 09:06:44 SEVERE listobs::ms::summary+ ================================================================================
2022-10-17 09:06:44 SEVERE listobs::ms::summary+ Exception Reported: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE listobs:::: *** Error *** Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 INFO listobs:::: ##### End Task: listobs #####
2022-10-17 09:06:44 INFO listobs::::+ ##########################################
Out[3]: False
Checking implementation using server default 3.8
python3 -m venv env (Python 3.8.10)
pip install --upgrade pip wheel
pip install requests
pip install fsspec
pip install dask-ms[xarray]
pip install python-casacore
pip install astropy
pip install tart tart-tools tart2ms
Noting versions installed: dask-ms-2022.8.2 python-casacore-3.5.2 astropy-5.1 tart-1.1.0b9 tart-tools-1.1.0b9 tart2ms-0.2.1b4
Updated astropy doesn't have the error for libs in python3.7
tart2ms --json data_test.json --ms data_test.ms
Getting Data from file: data_test.json
Writing measurement set 'data_test.ms'...
2022-10-17 11:15:26,338 - root - INFO - Time 5071671511.991
2022-10-17 11:15:26,888 - root - INFO - Pol Feeds ['RR']
2022-10-17 11:15:26,888 - root - INFO - Correlation Types [[5]]
2022-10-17 11:15:26,921 - root - INFO - Pol Types ['RR']
2022-10-17 11:15:26,921 - root - INFO - Pol Responses [[0.7071067811865475, (-0-0.7071067811865475j)]]
2022-10-17 11:15:27,197 - root - INFO - Corr Prod [[0 0]]
2022-10-17 11:15:27,197 - root - INFO - Corr Type [5]
2022-10-17 11:15:27,197 - root - INFO - NUM_CORR dask.array<full_like, shape=(1,), dtype=int32, chunksize=(1,), chunktype=numpy.ndarray>
2022-10-17 11:15:27,198 - root - INFO - Dask Corr Prod (1, 1, 2)
2022-10-17 11:15:27,198 - root - INFO - Dask Corr Type (1, 1)
2022-10-17 11:15:27,199 - root - INFO - Data size 276 1 1
Measurement set writing complete.
Using system CASA installation (CASA Version 5.6.0-60) gives similar error as noted in COLAB and python3.7
CASA <2>: msfile='data_test.ms'
CASA <3>: listobs(msfile)
2022-10-17 09:06:44 INFO listobs::::
2022-10-17 09:06:44 INFO listobs::::+ ##########################################
2022-10-17 09:06:44 INFO listobs::::+ ##### Begin Task: listobs #####
2022-10-17 09:06:44 INFO listobs:::: listobs(vis="data_test.ms",selectdata=True,spw="",field="",antenna="",
2022-10-17 09:06:44 INFO listobs::::+ uvrange="",timerange="",correlation="",scan="",intent="",
2022-10-17 09:06:44 INFO listobs::::+ feed="",array="",observation="",verbose=True,listfile="",
2022-10-17 09:06:44 INFO listobs::::+ listunfl=False,cachesize=50,overwrite=False)
2022-10-17 09:06:44 WARN listobs::ms::msselect Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE listobs::ms::summary ================================================================================
2022-10-17 09:06:44 SEVERE listobs::ms::summary+ MeasurementSet Name: /home/ruby/wipeme/data_test.ms MS Version 2
2022-10-17 09:06:44 SEVERE listobs::ms::summary+ ================================================================================
2022-10-17 09:06:44 SEVERE listobs::ms::summary+ Exception Reported: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 SEVERE listobs:::: *** Error *** Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:06:44 INFO listobs:::: ##### End Task: listobs #####
2022-10-17 09:06:44 INFO listobs::::+ ##########################################
Out[3]: False
Focusing on CASA and CASA tools versions, installing latest wheel version available
pip install casadata==2022.9.5
pip install casatools==6.5.2.26
pip install casatasks==6.5.2.26
pip install casaplotms==1.8.7
Python 3.8.10 (default, Jun 22 2022, 20:18:18)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from casatasks import listobs
>>> msfile = "data_test.ms"
>>> listobs(vis=msfile)
2022-10-17 09:31:09 WARN listobs::ms::msselect Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:31:09 SEVERE listobs::ms::summary ================================================================================
2022-10-17 09:31:09 SEVERE listobs::ms::summary+ MeasurementSet Name: /home/ruby/wipeme/data_test.ms MS Version 2
2022-10-17 09:31:09 SEVERE listobs::ms::summary+ ================================================================================
2022-10-17 09:31:09 SEVERE listobs::ms::summary+ Exception Reported: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
2022-10-17 09:31:09 SEVERE listobs::::casa Task listobs raised an exception of class RuntimeError with the following message: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatasks/listobs.py", line 159, in __call__
task_result = _listobs_t( _pc.document['vis'], _pc.document['selectdata'], _pc.document['spw'], _pc.document['field'], _pc.document['antenna'], _pc.document['uvrange'], _pc.document['timerange'], _pc.document['correlation'], _pc.document['scan'], _pc.document['intent'], _pc.document['feed'], _pc.document['array'], _pc.document['observation'], _pc.document['verbose'], _pc.document['listfile'], _pc.document['listunfl'], _pc.document['cachesize'], _pc.document['overwrite'] )
File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatasks/private/task_listobs.py", line 59, in listobs
obs_dict = myms.summary(
File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatools/ms.py", line 577, in summary
return _dict_dc(self._swigobj.summary(verbose, _str_ec(listfile), listunfl, cachesize, overwrite, wantreturn))
File "/home/ruby/wipeme/env/lib/python3.8/site-packages/casatools/__casac__/ms.py", line 1401, in summary
return _ms.ms_summary(self, *args, **kwargs)
RuntimeError: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
@tmolteno I have verified all the package versions that I think are important and made sure I am using the newest versions, but still no joy Please can you verify which versions you are using, just incase I have missed something.
I will have a look at the TIME_CENTROID column in tart2ms. This may well have a similar problem to the TIME column which was of the incorrect order...
Although the error does not occur in wsclean anymore, I cannot get it working in CASA directly. I keep getting the same error when trying to read the tables, but cannot easily find anywhere that may cause the issue.
CASA return
RuntimeError: Invalid Table data type when accessing column in ScalarColumn ctor for column TIME_CENTROID
Full implementation and exception trace can be found in notebook: https://github.com/rubyvanrooyen/notebooks/blob/main/application/CASA_TART_workspace.ipynb
Error description from CASA log