ices-eg / wg_WGFAST

Working Group on Fisheries Acoustics, Science and Technology
11 stars 16 forks source link

Cooperate with IFREMER multibeam netCDF convention #2

Closed gavinmacaulay closed 5 years ago

gavinmacaulay commented 6 years ago

IFREMER are developing a netCDF-based convention for storing multi-beam data, including water column data. We need to modify the SONAR-netCDF convention to better support some of the concepts in the IFREMER format, such as:

cyrilleponcelet commented 5 years ago

Hi Gavin, I'm working with Laurent Berger on an extension of the Sonar-netcdf4 format allowing to store bathymetry data and Simrad ADCP data.

In order to be more generic I would like to submit several change request on the format. I would like your feeling about it before I proceed in a more detailed way because most of these changes are mandatory to us.

1) Make platform description more generic. Bathymetry Sonar are made up of several antennas being either emitting antennas (Tx) or Receiving Antennas (Rx). I would like to submit a proposal allowing to describe those categories.

2) Allow multiple sensors : In my systems I could have several position and attitude sensors connected, I would proposed a way to organize those description in order to support those type of multi sensors data.

3) We would like to submit the idea, as a general rule to add domain specificities as subgroup of Sonar/Beam_group1 . For example in my bathymetric case, I will add all my sounding detection information as a group in Sonar/Beam_group1/Bathymetry, for ADCP data they would be located in Sonar/Beam_group1/ADCP

4) We need to have interpolated data at the time of the ping (typically position and attitude). Since dimension cannot be shared amoung group except descending ones, we need to add those data as a subgroup of Beam_group1. Thus we will have a platform subgroup of Beam_group1. This data will match position and attitude data effectively used by the sounder at the time of the ping. What is your feeling about this ?

Shall we proceed with the pull request or shall we discuss furthermore about those subjects ?

gavinmacaulay commented 5 years ago

Items 1 and 2 sound fine. They are something that wasn't considered when developing v1.0 of the convention, but had we thought of it, they would of been included.

For item 3, are you suggesting to store processed/derived data under the associated Beam_groupX? I received in March 2019 some ADCP suggestions from Simrad where they proposed a toplevel group ADCP and subgroups of ADCP/BeamData and ADCP/CurrentVelocity. Comments on this? At the time their suggestion seemed fine, but your suggestion seems more consistent with the sonar-netcdf4 idea that everything derived from a group of beams is stored under the relevant Beam_groupX. This also makes the format more similar for different types of acoustic systems. There might then be the need for a 'beam_purpose' variable in each Beam_groupX that says what purpose the data is mostly collected for (e.g., omnisonar, echosounder, adcp, multibeam).

For item 4, I thought that dimension variables could be shared across all groups (the mechanism appears to use id's that are independent of the group hierarchy)? I had done some testing with this and it seemed to work. Hence variables in /Platform could use the /Sonar/Beam_groupX/ping_time dimension variable (or the opposite)? A related question is whether you see a need for variables to hold full sample rate /Platform data (e.g., position and attitude) and data at the ping times? If yes, we should allow for that with extra variables and/or groups.

I would like to see these 4 items done via separate pull requests (to keep the changes organised). Items 1 & 2 are no problem - go ahead. I think we need to discuss items 3 & 4 a little more first.

cyrilleponcelet commented 5 years ago
with nc.Dataset(file_path,mode="w")as dataset:
     grp1= dataset.createGroup("/sonar/beam/Beam_group1/")
     gpr2= dataset.createGroup("/sonar/beam/Beam_group2/")
     grp1.createDimension("ping",10)
     gpr2.createDimension("ping",20)
     plat=dataset.createGroup('platform')
     plat.createVariable("lat","f8",("ping"))

Yes, I see a need for variables to hold the data at full sample rate (100Hz on my MBES) and the interpolated values (interpolated by the sounder) at the ping times. For a multibeam echosounder the interpolated values will allow to know the exact platform data at ping time used for positioning the soundings on the seafloor For an ADCP it will allow to know the exact platform data during transmission and reception used for computation of current velocities. In my cases I added those variables as new variables in the beamGroup. I used a generic rule implying that interpolated variables at ping time are located in the beamGroup (which is where is defined the ping_time). Apart the technical issue of sharing dimension, I think this is the best place to put or we need a rule to define how to give a name to a set of interpolated variables since we will have one set of those variables per beamGroup

gavinmacaulay commented 5 years ago

The 4 items have been split into 4 separate issues (#12, #13, #14, and #15) to improve traceability.

gavinmacaulay commented 5 years ago

Further items under the top level heading (cooperate with IFREMER) should be done as separate issues.