Cooperate with IFREMER multibeam netCDF convention

[gavinmacaulay]

IFREMER are developing a netCDF-based convention for storing multi-beam data, including water column data. We need to modify the SONAR-netCDF convention to better support some of the concepts in the IFREMER format, such as:

• enhance group structure to allow both conventions to exist in the same netCDF file
• permit separate transmitting and receiving transducers (e.g., location, orientation)

[cyrilleponcelet]

Hi Gavin, I'm working with Laurent Berger on an extension of the Sonar-netcdf4 format allowing to store bathymetry data and Simrad ADCP data.

In order to be more generic I would like to submit several change request on the format. I would like your feeling about it before I proceed in a more detailed way because most of these changes are mandatory to us.

1) Make platform description more generic. Bathymetry Sonar are made up of several antennas being either emitting antennas (Tx) or Receiving Antennas (Rx). I would like to submit a proposal allowing to describe those categories.

2) Allow multiple sensors : In my systems I could have several position and attitude sensors connected, I would proposed a way to organize those description in order to support those type of multi sensors data.

3) We would like to submit the idea, as a general rule to add domain specificities as subgroup of Sonar/Beam_group1 . For example in my bathymetric case, I will add all my sounding detection information as a group in Sonar/Beam_group1/Bathymetry, for ADCP data they would be located in Sonar/Beam_group1/ADCP

4) We need to have interpolated data at the time of the ping (typically position and attitude). Since dimension cannot be shared amoung group except descending ones, we need to add those data as a subgroup of Beam_group1. Thus we will have a platform subgroup of Beam_group1. This data will match position and attitude data effectively used by the sounder at the time of the ping. What is your feeling about this ?

Shall we proceed with the pull request or shall we discuss furthermore about those subjects ?

[gavinmacaulay]

Items 1 and 2 sound fine. They are something that wasn't considered when developing v1.0 of the convention, but had we thought of it, they would of been included.

For item 3, are you suggesting to store processed/derived data under the associated Beam_groupX? I received in March 2019 some ADCP suggestions from Simrad where they proposed a toplevel group ADCP and subgroups of ADCP/BeamData and ADCP/CurrentVelocity. Comments on this? At the time their suggestion seemed fine, but your suggestion seems more consistent with the sonar-netcdf4 idea that everything derived from a group of beams is stored under the relevant Beam_groupX. This also makes the format more similar for different types of acoustic systems. There might then be the need for a 'beam_purpose' variable in each Beam_groupX that says what purpose the data is mostly collected for (e.g., omnisonar, echosounder, adcp, multibeam).

For item 4, I thought that dimension variables could be shared across all groups (the mechanism appears to use id's that are independent of the group hierarchy)? I had done some testing with this and it seemed to work. Hence variables in /Platform could use the /Sonar/Beam_groupX/ping_time dimension variable (or the opposite)? A related question is whether you see a need for variables to hold full sample rate /Platform data (e.g., position and attitude) and data at the ping times? If yes, we should allow for that with extra variables and/or groups.

I would like to see these 4 items done via separate pull requests (to keep the changes organised). Items 1 & 2 are no problem - go ahead. I think we need to discuss items 3 & 4 a little more first.

[cyrilleponcelet]

• For Item 1 and 2 we will create the pull request

• For Item 3 : Yes, I am suggesting that. Ifremer is currently commenting the ADCP suggestions from Simrad, I think they will talk about this with Simrad and come back to us. And a multiple enum value beam_purpose could really make sense

• For Item 4 : In which language you made you test ? I don't think you can share variable across all groups except if you define it at the root node: (https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_model.html). "Dimensions have a special scope: they may be seen by all variables in their group, and all descendant groups. This allows the user to define dimensions in a top-level group, and use them in many sub-groups." I made a simple test in python showing that it failed in my case and even if it worked it could lead to some ambiguity. In my example, what ping dimension should be used for the "lat" variable ?

with nc.Dataset(file_path,mode="w")as dataset:
     grp1= dataset.createGroup("/sonar/beam/Beam_group1/")
     gpr2= dataset.createGroup("/sonar/beam/Beam_group2/")
     grp1.createDimension("ping",10)
     gpr2.createDimension("ping",20)
     plat=dataset.createGroup('platform')
     plat.createVariable("lat","f8",("ping"))

Yes, I see a need for variables to hold the data at full sample rate (100Hz on my MBES) and the interpolated values (interpolated by the sounder) at the ping times. For a multibeam echosounder the interpolated values will allow to know the exact platform data at ping time used for positioning the soundings on the seafloor For an ADCP it will allow to know the exact platform data during transmission and reception used for computation of current velocities. In my cases I added those variables as new variables in the beamGroup. I used a generic rule implying that interpolated variables at ping time are located in the beamGroup (which is where is defined the ping_time). Apart the technical issue of sharing dimension, I think this is the best place to put or we need a rule to define how to give a name to a set of interpolated variables since we will have one set of those variables per beamGroup

[gavinmacaulay]

The 4 items have been split into 4 separate issues (#12, #13, #14, and #15) to improve traceability.

[gavinmacaulay]

Further items under the top level heading (cooperate with IFREMER) should be done as separate issues.