schema_checker issues and conversion errors in datasets from casatestdata/CASA guides

[FedeMPouzols]

I'm creating this issue to have in a branch some fixes for various schema validation issues caught by the schema checker + errors in the converter code, while looking into other issues/errrors.

In casatestdata, out of ~305 datasets we get:

• 42 errors in the conversion code
• 119 datasets have schema checker issues (a majority could be seen as minor issues in some units)

From the schema checker issues, a majority seem minor issues in some units and frames. From the conversion errors, a very common one is issues with dimensions in extract_feed_info

The conversion errors are spread as follows:

• EVLA: 2 errors out of 36
• VLA: 3 errors out of 86
• ALMA: 10 errors out of 83
• ALMA-SD: 16 errors out of 49
• Others: 11 errors out of 51

(note that in these stats there are some 5-10 datasets that I think we do not expect to convert without errors, like for example the crazySourceTable.ms, but I wanted to catch as many validation/conversion issues).

[FedeMPouzols]

With the fixes in this branch the number of MSs with schema checker issues is down from 119 to 6: 3 in ALMA and 3 in Other (BIMA) datasets. All the issues seem related to reference frames.

The number of conversion errors is down to 5:

• EVLA: 0
• VLA: 3
• ALMA: 1 (crazySourceTable.ms), acceptable error: 'Can only process source table with a single time entry for a source_id and spectral_window_id.' (this is a manipulated test dataset)
• ALMA-SD: 0
• Others: 1 (onon.ms), acceptable error: 'Can only process feed table with a single time entry for an feed, antenna and spectral_window_id.' (this is Nobeyama dataset)

From the 3 VLA errors, there is a failure that is not in conversion but in the summary function (invalid coordinate frame when trying to construct an SkyCoord) - this is with the exotic "b1950_vla" which I think is an "FK4" with a custom epoch ("1979.9").

This set of 305 excludes some MSs that have been troublesome in the past (for example a Dysco LOFAR dataset), which should be re-checked later on.

The stakeholder test failure I think is unrelated to this branch. We have a test failure in main, in test_preconverted_alma due to an error with inconsistent xds names: MAIN / correlated_xds that must be related to the ongoing renaming to 'correlated_xds'.

[FedeMPouzols]

The changes in this branch now fix all the schema issues listed above. But recent changes from last week have introduced a relatively common conversion error, apparently when there are different polarization setups / IDs used in the DDI table and those have different NUM_CORR, as used for example in channel_average SPWs. This makes 11 MSs of 83 ALMA test MSs fail, and I suspect most if not all freshly imported ALMA MSs will have similar issues. I'm looking into it.

Before this, the errors remaining were either expected:

• ALMA: 1 (crazySourceTable.ms), acceptable error: 'Can only process source table with a single time entry for a source_id and spectral_window_id.' (this is a manipulated test dataset)
• Others: 1 (onon.ms), acceptable error: 'Can only process feed table with a single time entry for an feed, antenna and spectral_window_id.' (this is Nobeyama dataset)

or related to coordinate frames which should probably dealt with in its own issue:

• Two issues with coordinate frames in VLA datasets ngc7538_ut.ms and 0420+417.ms (schema issues with the frame of FIELD_PHASE_CENTER and failures in the summary function)

In CASA guides, one EVN dataset, n14c3.ms, has conversion errors because of multiple gain curves/times per antennas.