mvftoms creates incorrect ms file

[sourabhpaul]

Hi,

I'm trying to convert some latest UHF meerklass data using the mvftoms tool (The rdb file path is : /idia/raw/hi_im/SCI-20220822-MS-01/1675623808/1675623808/1675623808_sdp_l0.full.rdb). This is my first time using the mvftoms tool and my apologies if I'm missing any important details here.

I did the following on a jupyter notebook:

fname=1675623808
rdb_link="file:///idia/raw/hi_im/SCI-20220822-MS-01/1675623808/1675623808/1675623808_sdp_l0.full.rdb"

!python3 /users/sourabh/OTF/katdal/scripts/mvftoms.py -a --flags cam --scans "" {rdb_link} -o /scratch3/users/sourabh/OTF/UHF/{fname}.ms

The error I get is:

Iterating through scans in dataset(s)...

Traceback (most recent call last):
  File "/users/sourabh/OTF/katdal/scripts/mvftoms.py", line 916, in <module>
    main()
  File "/users/sourabh/OTF/katdal/scripts/mvftoms.py", line 563, in main
    ms_dict['FEED'] = ms_extra.populate_feed_dict(len(dataset.ants), num_receptors_per_feed=2)
  File "/usr/local/lib/python3.8/dist-packages/katdal/ms_extra.py", line 543, in populate_feed_dict
    feed_dict['POL_RESPONSE'] = np.dstack([np.eye(2, dtype=np.complex64) for n in range(num_feeds)]).transpose()
  File "<__array_function__ internals>", line 200, in dstack
  File "/usr/local/lib/python3.8/dist-packages/numpy/lib/shape_base.py", line 723, in dstack
    return _nx.concatenate(arrs, 2)
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate

I get no error if I remove all the options and it creates the ms file:

!python3 /users/sourabh/OTF/katdal/scripts/mvftoms.py {rdb_link} -o /scratch3/users/sourabh/OTF/UHF/{fname}.ms However, when I run listobs on the output ms file, I get the following error:

RuntimeError: Exception: Illegal ANTENNA1 value 60 found in main table. /scratch3/users/sourabh/OTF/UHF/[1675623808.ms/ANTENNA](http://1675623808.ms/ANTENNA) only has 0 rows (IDs).
... thrown by void casa::MSChecker::checkReferentialIntegrity() const at File: /source/casa6/casatools/src/code/msvis/MSVis/MSChecker.cc, line: 73

[ludwigschwardt]

Hi!

How big is your MS? It sounds like it could be empty.

It also sounds like mvftoms could not find any antennas in the dataset, which is unexpected. I get

import katdal
dataset = katdal.open(rdb_link)
print(len(dataset.ants))   # I get 61 here

I would start by deleting the MS and starting mvftoms over again. The script could be confused by the MS that you made before.

[sourabhpaul]

Thanks! I didn't notice that it created a corrupted MS file in the first round. The fresh MS output of !python3 /users/sourabh/OTF/katdal/scripts/mvftoms.py {rdb_link} -o /scratch3/users/sourabh/OTF/UHF/{fname}.ms looks alright now. Any idea, why it didn't work with the options e.g. -a ?

[ludwigschwardt]

MVFtoMS tries to continue with the MS after it crashed, but it assumes that you are rerunning it with the same flags and settings. If not, it will get confused... Something to fix still.

I'm also stumped why you would end up with no antennas in the selection. Maybe it's the --flags cam that upsets it? You can also leave out the --scans "" which should do nothing but you never know...

Maybe try mvftoms.py -a --flags cam or even mvftoms.py -a?

[sourabhpaul]

Thanks!