FedeMPouzols
commented
7 months ago Hi @fdulwich , I took a look at this create_weather_xds() function. I think that it is very close to a working version in its current shape.
The main issue that I found is with the order of the dimensions. If you try this branch on example MSs you'll probably get errors like:
`ValueError: conflicting sizes for dimension 'antenna_id': length 1 on 'antenna_id' and length 276 on {'time': 'PRESSURE', 'antenna_id': 'PRESSURE'}That is because in this weather_xds we are now using (time, antenna_id,...) ordering but in the earlier MS/casacore read functions we were using (antenna_id, time...) ordering. This can be fixed by applying a change like this to create_weather_xds()
"wind_speed": "WIND_SPEED",
}
data_variable_dims = {
- "h2o": ["time", "antenna_id"],
- "ionos_electron": ["time", "antenna_id"],
- "pressure": ["time", "antenna_id"],
- "rel_humidity": ["time", "antenna_id"],
- "temperature": ["time", "antenna_id"],
- "dew_point": ["time", "antenna_id"],
- "wind_direction": ["time", "antenna_id"],
- "wind_speed": ["time", "antenna_id"],
+ "h2o": ["antenna_id", "time", ],
+ "ionos_electron": ["antenna_id", "time"],
+ "pressure": ["antenna_id", "time"],
+ "rel_humidity": ["antenna_id", "time"],
+ "temperature": ["antenna_id", "time"],
+ "dew_point": ["antenna_id", "time"],
+ "wind_direction": ["antenna_id", "time"],
+ "wind_speed": ["antenna_id", "time"],
}Or alternatively we can change the order of the dimensions in read.py:redimension_ms_subtable().
Now that this came up I'm not sure if (time, antenna_id,...) is necessarily the only or best order for this table. In the pointing_xds we are using the same order but I'm equally not sure if that's the most natural order. In any case it is just a matter of order of dimensions when creating the data arrays / giving the list of dimensions of the data vars.
Then if we want to move from "antenna_id" to "station_id" we'd need several changes I think. There are some comments in the spreadsheet/WEATHER and I probably missed the discussion so I'm not sure if the new station_id is just a rename or if we want to re-generate indices. For a start, something along the lines of this diff would apply the rename to "station_id" and produce the coords and dims that I think gets us closer to the specs:
diff --git a/src/xradio/vis/_vis_utils/_ms/msv4_sub_xdss.py b/src/xradio/vis/_vis_utils/_ms/msv4_sub_xdss.py
index f17dfd3..fd06bbe 100644
--- a/src/xradio/vis/_vis_utils/_ms/msv4_sub_xdss.py
+++ b/src/xradio/vis/_vis_utils/_ms/msv4_sub_xdss.py
@@ -114,22 +114,23 @@ def create_weather_xds(in_file: str):
"wind_speed": "WIND_SPEED",
}
data_variable_dims = {
- "h2o": ["time", "antenna_id"],
- "ionos_electron": ["time", "antenna_id"],
- "pressure": ["time", "antenna_id"],
- "rel_humidity": ["time", "antenna_id"],
- "temperature": ["time", "antenna_id"],
- "dew_point": ["time", "antenna_id"],
- "wind_direction": ["time", "antenna_id"],
- "wind_speed": ["time", "antenna_id"],
+ "h2o": ["station_id", "time", ],
+ "ionos_electron": ["station_id", "time"],
+ "pressure": ["station_id", "time"],
+ "rel_humidity": ["station_id", "time"],
+ "temperature": ["station_id", "time"],
+ "dew_point": ["station_id", "time"],
+ "wind_direction": ["station_id", "time"],
+ "wind_speed": ["station_id", "time"],
}
to_new_coord_names = {
- "time": "time",
- "antenna_id": "antenna_id",
+ # No MS data cols are turned into xds coords
}
coord_dims = {
- "time": ["time"],
- "antenna_id": ["antenna_id"],
+ # No MS data cols are turned into xds coords
+ }
+ to_new_dim_names = {
+ "antenna_id": "station_id",
}
# Read WEATHER table into a Xarray Dataset.
@@ -138,6 +139,7 @@ def create_weather_xds(in_file: str):
"WEATHER",
rename_ids=subt_rename_ids["WEATHER"],
)
+ generic_weather_xds = generic_weather_xds.rename_dims(to_new_dim_names)
weather_column_description = generic_weather_xds.attrs["other"]["msv2"]["ctds_attrs"][
"column_descriptions"
@@ -148,8 +150,8 @@ def create_weather_xds(in_file: str):
weather_xds = xr.Dataset()
coords = {
- "antenna_id": np.arange(generic_weather_xds.dims["antenna_id"]),
- "xyz_label": ["x", "y", "z"],
+ "station_id": np.arange(generic_weather_xds.dims["station_id"]),
+ "time": generic_weather_xds["time"],
}
for key in generic_weather_xds:
msv4_measure = column_description_casacore_to_msv4_measure(When looking into this I saw more potential tricky details that I'm not sure we have discussed. Form what I see in a few test MSs, a relatively common practice in the WEATHER subtable is:
- to leave the "ANTENNA_ID" to -1!
- (sometimes, not always) to use a "non-standard extension" column, NS_WX_STATION_ID (+ another non-standard NS_WX_STATION_POSITION
From what I see in the CASA "ASDM=>MS filler" this is always done when importing ASDM into MS.
That NS_WX_STATION_ID is taken directly from the stationId key of the ASDM WEATHER table. If we wanted to use it we'd need to take its values to define the "station_id" coord (the coords = { ... dict above.
fdulwich
Create a function that will convert a MS v2 weather table to a MS v4 weather dataset (https://docs.google.com/spreadsheets/d/14a6qMap9M5r_vjpLnaBKxsR9TF4azN5LVdOxLacOX-s/edit?usp=sharing).
The function should appear in xradio/vis/_vis_utils/_ms/msv4_sub_xdss.py (see create_ant_xds as an example) and be called in the convert_and_write_partition function (xradio/vis/_vis_utils/_ms/conversion.py) after
ant_xds = create_ant_xds(in_file)and then saved to zarr underif storage_backend == "zarr":. The functionread_processing_set(src/xradio/vis/read_processing_set.py) should also be updated so to that the datatset can be read.