195 vlass otf support

[Jan-Willem]

This ticket adds support for on-the-fly mosaics that have a continuously changing phase center. This is achieved by allowing a coarser partitioning of the data (multiple fields in a single MSv4).

In convert_msv2_to_processing_set the partition_scheme parameter can be used to set the partitioning scheme:

• partitioning scheme=["FIELD_ID"] #The default that gives the partitioning we have been using up until now.
• partitioning scheme=[] #To be used for datasets with continuously changing phase centers (VLASS, ngVLA mosaics). Maximum "coarseness".

Key Points

• As discussed today intent is renamed obs_mode.
• Schema: https://docs.google.com/spreadsheets/d/14a6qMap9M5r_vjpLnaBKxsR9TF4azN5LVdOxLacOX-s/edit?gid=1230008044#gid=1230008044
• The most important changes were made to xradio/vis/_vis_utils/_ms/_tables/create_field_and_source_xds.py
• The reviews/review_field_and_source_xds.ipynb can be used to see how the schema has changed. In the cell where the conversion is done, you can toggle between partitioning scheme=["FIELD_ID"] and partitioning scheme=[]. The example notebook makes use of an ALMA mosaic instead of the VLASS dataset because the VLASS dataset can't be reasonably converted using partitioning scheme=["FIELD_ID"].
• @FedeMPouzols create_field_and_source_xds::create_field_info_and_check_ephemeris do you know how to do a TaQL query for specific rows in the table (there is no field_ids column but the row numbers should be used)?

[FedeMPouzols]

@Jan-Willem : I think you mean the row numbers/IDs? (as the field IDs are missing in the MSv2 and they use implicitly the row number as field ID). TaQL has this syntax: "select ROWID() as ROWS from ... where ..." which I think should do the trick. There is one example here: https://github.com/casangi/xradio/blob/main/src/xradio/vis/_vis_utils/_ms/partition_queries.py#L328 If we want to use the 'ROWID' in the where, that also seems to work, like SQL pseudo-columns. Even though I do not see examples of this in the TaQL reference, something like the following works for me: select DELAY_DIR, PHASE_DIR from $tb where ROWID() == 0 select DELAY_DIR, PHASE_DIR from $tb where ROWID() in [0,1,2] Is something like taql_where = f"where ROWID() in [{','.join(map(str, unqiue_field_id))}] what you are looking for?

[Jan-Willem]

Thanks @FedeMPouzols for the feedback. I have pushed one change where I added the field table row id TaQL query, and fixed a data selection bug. If there are no objections I will merge after the tests have been completed.