casangi / xradio

Xarray Radio Astronomy Data IO
https://xradio.readthedocs.io/en/latest/
Other
15 stars 7 forks source link

Clarifications in the MSv4 description #313

Open dpetry opened 1 day ago

dpetry commented 1 day ago

As discussed also in person with Jan-Willem Steeb, I think the MSv4 is a good step forward, but to make the the transition to it easier for long-term users of the MSv2, it would be good to extend the MSv4 documentation/description further by (a) clarifying the high-level structure of the MSv4 as a database. and (b) providing a "cheat sheet" of corresponding concepts between MSv2 and 4.

Details: Concerning the MSv4 structure as a database, I would like to see something similar to the following diagram which I created long ago for the MSv2: msv2structure The MSv2 is a relational database made up of tables which are organised in the levels with the MAIN table and a few others in the top level, the subtables referenced by the top level tables in the second level, and the subtables referenced by the level-2 tables in the third level. Effectively, the MSv4 must have a similar layout. It would be good to see if it is also organised in three layers or if there are more.

Concerning the MSv2_to_4 "cheat sheet": I don't mean a table like the already existing one here: https://xradio.readthedocs.io/en/latest/measurement_set_overview.html#Translating-between-MS-v2 That table is also somewhat useful, but I would request to fill a table explaining the correspondence between implementation concepts in the two MS versions:

Here the beginnings of a list of implementation concepts of the MSv2 which should be part of the cheat sheet:

  1. casacore Table
  2. Table row
  3. Table column
  4. Table keyword
  5. Table field keyword
  6. QuantumUnits
  7. MEASINFO
  8. Array Column
  9. Variable (width) Array Column
  10. Optional Column
  11. Key Column (column which forms part of the key uniquely identifying a Table row)
  12. Subtable
  13. Optional Subtable
  14. Multi-MS (MMS)
  15. Virtually concatenated Table
  16. Column Storage Manager ...