FedeMPouzols
commented
6 months ago About the awful performance when converting large pointing tables (create_pointing_xds() ), I think it is now clear after some more tests that the vast lion's share of the inefficiencies comes from using the by-row reading functions of the casacore table tool.
From past experiences with the pointing table I had the vague idea that reading it through the generic read functions was easily slower by a factor of 2x as compared to the "readcol" functions normally used to read the main table / large columns. The generic and slower by-row reading is needed to deal with any potential complicated variable size array columns, which happen always or often in known subtables like SPECTRAL_WINDOW or HISTORY, and also in unexpected extension columns.
With a bit more of experimenting, replacing all the row-reading calls with by-column reads (getcol()s) ,the speedup factor can be as high as > ~20-30x, and probably increasing as the pointing table size increases. In particular the slicing and many calls to tablerow.py:48(get)) is clearly dominating runtimes and seems to trigger very large amounts of Python allocations (with memory use also being higher by a factor of >= ~3x).
| Some examples of runtimes (all timing values approx.): | MS | MS size | POINTING size | "before (s)" (using row()) | "after (s)" (using getcol()) | create_pointing_xds() down to (s): |
|---|---|---|---|---|---|---|
| A) twhya.short.ms from casatestdata) | 550MB | 127MB | 461 | 22 | 0.96 | |
| B) uid___A002_X8ca70c_X5_shortened.ms (from casatestdata) | 499MB | 211MB | 335 | 80 | 1.82 | |
| C) uid___A002_Xfd764e_X4e4c_targets.ms" PL dataset, 7m mosaic | 2.2GB | 190MB | 8974 | 344 | 1.53 |
(These are datasets where only the "main" / science SPWs have been kept. The ratio would be higher for fuller (pre-calibration) datasets with many small SPWs, which is very common in ALMA at least throughout calibration.)
I'd be optimistic that this will probably sort out most if not all of the awful performance issue, and the other related topics listed in this issue can be addressed without being too conditioned by performance considerations.
We'll need a specialized efficient by-column reading for the POINTING subtable (and possibly others if they can be large enough) and the generic read for other tables to be able to handle misbehaving variable size columns, as well as non-standard extension columns that we can see in MSs from various observatories and are hard to anticipate. I'm leaning towards handling this alternative loading (by-row vs. by-col) deep in the code that reads the columns, keeping the rest of the code the same, but for that I have to reorganize the code still a bit more.
We do have one version of read_pointing that is fast (uses getcol()) for "read_vis()" but that one uses chunked reads with dask.delayed, which is not a good option for the convert functionality.
Suggestions for Improvements:
Performance Issue: