Include additional environment variables stored in EK80 files to converted dataset

[leewujung]

EK80 files contain a lot more environmental variables in addition to the 5 ("temperature", "depth", "acidity", "salinity", "sound_speed") that we currently save in the converted dataset. For example, the current test file D20170912-T234910.raw contains the following:

We should add those that are of the nature of environmental variables (such as "sound_velocity_profile", "sound_velocity_source") to the output variables into the Environment group, and others, such as "water_level_draft" and "water_lebel_draft_is_manual" into the Platform group, so on and so forth.

I am not sure why there is a "latitude" variable in this...

This is related to #531 .

[emiliom]

Is that additional information currently not retained somewhere, even if in unparsed form? In other words, if you convert an EK80 raw file, does the resulting EchoData object contain that information anywhere?

[gavinmacaulay]

I am not sure why there is a "latitude" variable in this...

Latitude is in the environment variables because it is an input to the sound speed equations. and presumably Kongsberg is including all input parameters that they used when calculating the sound speed.

[leewujung]

Latitude is in the environment variables because it is an input to the sound speed equations. and presumably Kongsberg is including all input parameters that they used when calculating the sound speed.

Ah I see, somehow I didn't make that immediate connection. So Kongsberg is probably using the new one for sound speed calculation in EK80 then, since in EK60 latitude is not included. We should include that in utils.uwa so things are matching. Thanks @gavinmacaulay !

[leewujung]

Is that additional information currently not retained somewhere, even if in unparsed form? In other words, if you convert an EK80 raw file, does the resulting EchoData object contain that information anywhere?

The additional information is currently not retained anywhere.

Another piece of info from the SONAR-netCDF4 convention p.7 as we move to include these:

[leewujung]

@gavinmacaulay : More questions as I put together an issue for adding a new option in sound speed calculation (Leroy et al 2008 JASA). Do you know if Kongsberg actually uses the latest equation on sound speed from Leroy et al. 2008, or they use the location info and depth to produce pressure, and then use the Mackenzie equation (or some other equation)?

[gavinmacaulay]

The EK80 manual says that it uses the 'UNSECO' algorithm and then gives these references for seawater:

• C-T. Chen and F.J. Millero, Speed of sound in seawater at high pressures (1977) Journal of the Acoustic Society of America, Volume 62, Issue 5, pages 1129 to 1135

• N.P. Fofonoff and R.C. Millard Jr. Algorithms for computation of fundamental properties of seawater (1983), UNESCO technical papers in marine science. No. 44, Division of Marine Sciences. UNESCO, Place de Fontenoy, 75700 Paris.

• G.S.K. Wong and S Zhu, Speed of sound in seawater as a function of salinity, temperature and pressure (1995) Journal of the Acoustic Society of America, Volume 97, Issue 3, pages 1732 to 1736

  and these for freshwater:

• V.A. Del Grosso, New equation for the speed of sound in natural waters (with comparisons to other equations) (1974) Journal of the Acoustic Society of America, Volume 56, Issue 4, pages 1084 to 1091

• G.S.K. Wong and S Zhu, Speed of sound in seawater as a function of salinity, temperature and pressure (1995) Journal of the Acoustic Society of America, Volume 97, Issue 3, pages 1732 to 1736

[leewujung]

@imranmaj : along with your work on #592, could you also take this on to add the variables that are in the EK80 raw data but not currently stored in the Environment group? Thanks!

[imranmaj]

File D20170912-T234910.raw has the following sound_velocity_profile:

This could be pairs of depth, sound speed

[leewujung]

Yes this looks like the (depth, sound speed) pair.

@emiliom : any pointers on if we name the first dimension as depth and the units (here it is most likely in meters) and other attributes?

[emiliom]

SONAR-netCDF4 v1 recommends the use of the "NCEI NetCDF 'profile' template, v2.0 or greater", for environmental profile data. The NCEI netcdf templates are here. See the profile and maybe the timeSeriesProfile feature types.

See the sample profile CDL file for more details.

any pointers on if we name the first dimension as depth

The variable name is up to the user, though conventional names should be used (z, depth, etc). What matters is the attributes -- standard_name, axis, etc.

and the units (here it is most likely in meters) and other attributes?

Units in meters would be perfect. The CF requirement is UDUNITS strings, with a few exceptions. For other attributes, see the CDL. But no need to adopt anywhere near the entire suite of global attributes listed!! The NCEI netcdf templates are a gold standard for data archiving. We should take baby steps here.

[leewujung]

Thanks @emiliom !

The variable name is up to the user, though conventional names should be used (z, depth, etc). What matters is the attributes -- standard_name, axis, etc.

Good to know this rule of thumb! 😀

The NCEI netcdf templates are a gold standard for data archiving. We should take baby steps here.

Great -- that template is intimidating...

[emiliom]

Great -- that template is intimidating...

It's strict CF + ACDD + some NCEI additions

[leewujung]

@emiliom : I tried to make up something below, please chime in on any suggestions/corrections! I used the sound_speed_indicative attributes for the variable, and added some info in the comment.

• variable name: "sound_velocity_profile"
  • attributes:
  • long_name: "sound velocity profile"
  • standard_name: "speed_of_sound_in_sea_water"
  • units: "m/s"
  • valid_min: 0.0
  • comment: "parsed from raw data files as (depth, sound_speed) value pairs"
• coordinates: "depth"
  • attributes:
  • standard_name: "depth"
  • units: "m"
  • axis: "Z"
  • positive: "down"
  • valid_min: 0.0

[imranmaj]

The environment group already has a depth variable; what other name should be used? Does sound_velocity_profile_depth work?

[leewujung]

@imranmaj : I think sound_velocity_profile_depth works. @emiliom : any thoughts on the above other attributes?

[emiliom]

Side note:

EK80 files contain a lot more environmental variables in addition to the 5 ("temperature", "depth", "acidity", "salinity", "sound_speed") that we currently save in the converted dataset.

None of those existing variables have any attributes assigned (eg, units). That should be an easy addition. The task could be tracked via a new, separate issue.

[emiliom]

Regarding the attributes -- and structure -- of the new profile data (finally got to it! :disappointed:): the attributes and names look good. But what about the time dimension? Is each (depth, sound_speed) profile associated with a single time stamp? If so, is there a profile for each ping_time value? In other words, is sound_velocity_profile actually a function of both depth and ping_time? Are the depth bins identical across a raw file? If those conditions are met, we're dealing with a timeSeriesProfile CF feature type rather than a simple profile.

A couple of questions or issues come to mind too if we were to think of the Environment data from a strict CF perspective (@imranmaj : I'm thinking out loud here!). If you go to the trouble of following a CF feature type structure, typically you'd want to label it as such via dedicated global attributes, especially Conventions and featureType. But CF specifies that a data file can only contain data for one feature type, whereas this Environment group would hold both a profile (or timeSeriesProfile) variable and several timeSeries variables that are only a function of ping_time (temperature", "depth", "acidity", "salinity", "sound_speed"). That would violate CF. Hmm.

[leewujung]

Is each (depth, sound_speed) profile associated with a single time stamp?

Currently in the raw data there is no time dimension in the sound speed profile. I don't know how this profile is generated in EK80, or whether it is input by user. If the latter, it is likely entered/imported when setting up the instrument. In this situation, we can potentially run into the situation where sound_velocity_profile is a function of both depth and ping_time when multiple EK80 files with different sound velocity profiles are combined together using combine_echodata. If we want to accommodate this case, maybe we can add a time coordinate and use the first ping_time for the timestamp of the profile.

As I am typing I am just realizing that this timestamp issue could occur for other environmental variables (temperature", "depth", "acidity", "salinity", "sound_speed") too. The difference here between EK60 and EK80 is that:

• for EK60: the environmental variables are part of the sample datagram so there is a natural dimension along ping_time
• for EK80: these env variables are just numbers (at least in this test file) recorded in an environment datagram (the echo data are in other datagrams) so won't have a natural ping_time dimension.

Ugh...

I am not sure what to comment for the CF violation part.

[emiliom]

Currently in the raw data there is no time dimension in the sound speed profile. I don't know how this profile is generated in EK80, or whether it is input by user.

Ah. Just so I'm 100% clear, there's just one profile per raw file?

[gavinmacaulay]

Currently in the raw data there is no time dimension in the sound speed profile.

There will be a time available from the .raw file datagram timestamp. If the profile data comes from a previously entered profile, the timestamp will be pretty much the time when that file was created. The EK80 (and EAxx) software accepts sound speed profiles sent automatically from other instruments, which can arrive at any time. I suggest allowing for a time coordinate on the profiles, noting that that time will not be a ping_time. There can be multiple profiles per raw file.

[leewujung]

Oh that's true that there is a timestamp associated with that particular datagram, even if not the profile, we can use that!

[leewujung]

EK60: environment variables are in the "sample datagram" so naturally has a ping_time dimension.

EK80: environment variables are in separate environment datagram so can have different timestamps compared to the acoustic data datagrams

[imranmaj]

• All fields (see top-most screenshot) will be converted to variables, not attributes
• transducer_sound_speed will be moved from Sonar to Environment
• transducer_name will be moved from Sonar to Environment
• drop_keel_offset_is_manual and water_level_draft_is_manual will remain in Platform

[emiliom]

A small clarification: transducer_sound_speed and transducer_name are only found in EK80 (or at the very least, they're not found in EK60, nor are they from the convention). That "clarification" wasn't strictly necessary, because this issue is focused on EK80.

[leewujung]

Added in #616, closing this now 🚀