b-reyes
commented
2 years ago Thank you @leewujung for putting this together! I have been working on this conversion problem, so I am happy to work on several of these items. I think that once we establish the sensor-specific files in #596, then we can proceed with making the code forward compatible. The following is the first step I would take (input is welcome) to address this issue. Note that this is primarily focused on changes to downstream items related to open_converted.
- Find the version of echopype that created the zarr/nc file and use the sensor-specific files created in #596 to open and read in the file into a EchoData object.
- This will require changes to
__load_file.- Replace
self.group_map.items()with a sensor-specific dict obtained from #596. - Replace lines 616-620 with a function that correctly assigns the group based on the sensor-specific file.
- Add a depreciation warning/prompt to save to zarr/nc, if conversion is necessary
- Apply change to variable names, if necessary
- Replace
- This will require changes to
From my understanding, if we end up implementing datatree, we would only need to modify the above items slightly.
emiliom
leewujung
Context
We are coming up against multiple planned breaking changes in v0.6.0. The changes include netCDF group structures as well as coordinates and variable names and attributes.
We know that many users, including both OOI and NCEI have converted some non-trivial volume of data into zarr or nc using echopype v0.5.x, so it is imperative that all processing can accommodate these already converted data.
In light of the group structure changes, we also want to ease the transition for ourselves such that we do not disturb the entire codebase significantly all at once (so that we can focus on convention compliance for v0.6.0).
Problems and Approaches
There are two types of changes we need to deal with:
Group structure changes
After discussions (#567) we concluded that a cleaner approach is for us to use DataTree functionality to allow accessing the data like a dictionary or like in the netcdf4 library, such as
EchoData["Sonar/Beam_groupX"], because our current attribute setup would requireEchoData.sonar.beam_group1which can be quite involved to implement. The downside to this are:open_treeon v0.5.x data, the groups will not be in the right placeThe last one requires some more discussion, so I wonder if an intermediate step we can take in v0.6.0 is to avoid this (for now) by making our code forward compatible to the v0.6.0 data: Basically, we would hook up:
EchoData.beamtoroot/Sonar/Beam_group1EchoData.beam_power(if exists) toroot/Sonar/Beam_group2EchoData.sonarstill toroot/Sonar(no action)EchoData.nmeastill toroot/Platform/NMEA(no action) This way all the downstream functions do not have to change in v0.6.0, and can be dealt with later once we figure out how to use DataTree to handle v0.5.x data.Variable name changes
For these changes, I propose that instead of taking the forward compatibility route as above, here we actually change the variable names in the code once the correct groups are read into xarray datasets.
For example, we would:
range_bin--> immediately change that torange_sample--> change all the downstream processing that usesrange_binalso torange_samplefrequency--> immediately change that tochannel--> change all the downstream processing that usesrange_binalso torange_sampleDecision?
I want to list out all the changes we need to make in v0.6.0 here, but that requires that we decide what approach to take for the group structure changes problem. So, pinging @emiliom @b-reyes @lsetiawan for inputs!
Once we arrive at a decision I'll add specific tasks to this issue.