SPWs with different channel numbers causes crash (even if only one SPW is referenced in the main table)

[IanHeywood]

I think this might be a CASA bug rather than a shadems bug, as I don't think the former updates the SPW table after Hanning smoothing has been applied, but:

2020-12-22 17:22:16 - shadems - INFO - : Indexing MS and building dataframes (167778 rows, chunk size is 5000
)
.008705092594_spw27_hanning-minmax-cache.json
2020-12-22 17:22:16 - shadems - INFO - axis: FREQ, range (None, None), discretization None
2020-12-22 17:22:16 - shadems - INFO - axis: amp(CORRECTED_DATA), corr 3, range (None, None), discretization
None
2020-12-22 17:22:16 - shadems - INFO - axis: _(ANTENNA1), corr False, range (None, None), discretization 27
2020-12-22 17:22:16 - shadems - INFO - axis: amp(CORRECTED_DATA), corr 0, range (None, None), discretization
None
2020-12-22 17:22:16 - shadems - INFO -                  : you have asked for 2 plots employing 4 unique datum
s
2020-12-22 17:22:16 - shadems - INFO - : Indexing MS and building dataframes (167778 rows, chunk size is 5000
)
Traceback (most recent call last):
  File "/usr/local/bin/shadems", line 8, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/usr/local/lib/python3.6/dist-packages/shade_ms/main.py", line 675, in main
    row_chunk_size=options.row_chunk_size)
  File "/usr/local/lib/python3.6/dist-packages/shade_ms/data_plots.py", line 199, in get_plot_data
    f"{axis.mapper.fullname}: size {value.shape[0]}, expected {shape[timefreq_axis]}"
AssertionError: frequency: size 64, expected 128

wsclean also crashed with a very similar message:

>>> An invalid channel range was specified! Measurement set only has 64 channels, requested imaging range is 64 -- 128.

[IanHeywood]

Actually hanningsmooth might not be the culprit. The parent MS has a mixture of X-band and P-band in the SPW table (X-band being a pointing scan I guess). The X-band SPWs (0-15) all have 64 channels, the P-band SPWs have 128...

[o-smirnov]

Ah! Now that's an open problem in shadeMS. I was lazy -- there's some shortcuts in there that make assumptions about SPW/DDID. Need to go in and clean that up.

How were you so unfortunate as to end up with an MS like that? Why not keep them separate?

[IanHeywood]

It arrived like this from the archive. There are 16 X-band SPWs that are not referenced by any rows in the main table, which is what made me think it was a pointing scan (I didn't make the SBs).

My first attempt to process this involved using mstransform to split out SPWs 16-31 containing just the P-band data. I then ran split-ms-spw.py to get a MS per SPW. This preserves the full SPECTRAL_WINDOW table, even though the main table only references one row in it, which is what seems to be causing the tricolour issue.

My second attempt I skipped the mstransform step and went straight for split-ms-spw.py. This took the X-band SPWs with it, and the varying number of channels between the two halves of the SPW table are (I guess) what's causing this trouble.

So I do want to keep it all separate, firstly because this is going to be a very hands-on reduction and it's more manageable this way, and secondly because having multiple SPWs seems to be a complete pain in the ass for all my favourite software.