OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
98 stars 73 forks source link

Expose transducer serial number on the converted data from the raw unconverted data. #212

Closed lsetiawan closed 2 years ago

lsetiawan commented 3 years ago

Overview

For OOI data, there are serial numbers somewhere within the raw data files. It would be great if this can be shown in the converted nc/zarr file.

leewujung commented 3 years ago

Currently we have the serial numbers for the transducer and for the transceiver in the EK80 nc/zarr files (but probably not in the most obvious place!). I need to check where things are in the OOI EK60 files. I know they are parsed, but not sure if it made it's way into the nc/zarr... (again the decisions related to correspondence/discrepancy with the convention #210 )

emiliom commented 3 years ago

I need to check where things are in the OOI EK60 files. I know they are parsed, but not sure if it made it's way into the nc/zarr...

At least for EK60, it's not being written into the sonar_serial_number attribute; see here. It looks to me like it'd be straightforward to fix this, if the information can be parsed easily from the EK60 datagrams.

leewujung commented 3 years ago

if the information can be parsed easily from the EK60 datagrams

Let's add this to TODO, I don't know immediately where in the parsed config datagram that is stored.

emiliom commented 2 years ago

It would be nice to address this long-standing issue in the near future. Maybe @imranmaj could tackle it?

leewujung commented 2 years ago

@imranmaj @lsetiawan : I looked into 3 EK60 test files we have, and the only place that I think has info resembling the serial number type is in field "channel_id" in parses_obj (if you set break point after here).

It doesn't seem to me that the transducer serial number was recorded in these files.

@gavinmacaulay :


File from OOI-CE02SHBP

path: ./echopype/test_data/ek60/ooi/CE02/SHBP-MJ01C-07-ZPLSCB101_OOI-D20191201-T000000.raw

Screen Shot 2022-02-27 at 9 13 01 AM

Last time I spoke with the Simrad sales rep, he mentioned some custom modification they needed to do for the OOI transducers to force one of the split-beam transducers to be a single beam, which I suspect goes to the transducer type here (OOI.38|200, OOI38|200). Recall in the OOI setup, 38 kHz and 200 kHz channels are set up as single beam, 120 kHz channel is split-beam.

File from Hake survey

path: ./echopype/test_data/ek60/ncei-wcsd/Summer2017-D20170719-T211347.raw

Screen Shot 2022-02-27 at 9 11 26 AM

Test file from AFSC

path: ./echopype/test_data/ek60/DY1801_EK60-D20180211-T164025.raw

Screen Shot 2022-02-27 at 9 13 45 AM

leewujung commented 2 years ago

And more investigation into EK80: The transducer serial number is recorded in test EK80 files we have, in the field channel_id_short and transducer_custom_name (see below).

However, I think is a hard-coded number users put in when setting up the instrument and is not read from the communication with the transceiver or transducer. (Pinging @gavinmacaulay to confirm?)


File from Hake survey

path: ./echopype/test_data/ek80/D20170912-T234910.raw

Screen Shot 2022-02-27 at 9 31 43 AM

More info is probably useful here: Screen Shot 2022-02-27 at 9 34 10 AM

File from @FletcherFT with the 2-in-1 18-200 combi

path: ./echopype/test_data/ek80/Green2.Survey2.FM.short.slow.-D20191004-T211557.raw Screen Shot 2022-02-27 at 9 37 36 AM

Similarly, exposing more info is probably useful here: Screen Shot 2022-02-27 at 9 40 16 AM

leewujung commented 2 years ago

Finally, @imranmaj I think we've discussed overhauling this section that parses the EK80 configuration XML datagram. We've been through some very specific changes to make sure the XML can parse data from standard WBT and WBT mini.

Not needed to address this issue, but it seems time for us to take that on in one of the v0.6.X so that the code will be easier to maintain in the future.

gavinmacaulay commented 2 years ago

And more investigation into EK80: The transducer serial number is recorded in test EK80 files we have, in the field channel_id_short and transducer_custom_name (see below).

However, I think is a hard-coded number users put in when setting up the instrument and is not read from the communication with the transceiver or transducer. (Pinging @gavinmacaulay to confirm?)

transducer serial number is manually entered when adding a transceiver/transducer to the EK80 software. But some transducers can report their own serial number. I'm not sure if this is used yet, but it was the intention to support that in the EK80.

gavinmacaulay commented 2 years ago

@imranmaj @lsetiawan : I looked into 3 EK60 test files we have, and the only place that I think has info resembling the serial number type is in field "channel_id" in parses_obj (if you set break point after here).

It doesn't seem to me that the transducer serial number was recorded in these files.

@gavinmacaulay :

  • Do you know if the serial number is actually stored in EK60 raw files? I don't see it in the spec of the configuration datagram
  • Do you know what the middle string ("009072034d45", "0090720346a8", etc) are? edit: looking at EK80 files, it seems that these are ethernet addresses?
  • To confirm: the "2-1", "6-1", "6-2" are: "transceiver channel-transducer number in that channel", is this correct?

Transducer serial number is not in EK60 raw files. The middle string is a GPT identification of sorts - I think it is the ethernet MAC address of the GPT. I think the x-y part is as you say, where the transceiver channel number is ordered as per the order of installation in the EK60/EK80 software (so is not necessarily sorted by any transceiver parameter).

leewujung commented 2 years ago

This can now be closed:

emiliom commented 2 years ago

Thanks for the summary conclusions!

EK60: transducer serial number is NOT stored in the data file

Let's add this comment to https://github.com/OSOceanAcoustics/echopype/blob/dev/echopype/convert/set_groups_ek60.py#L163, where sonar_serial_number is assigned, so we store this finding where it'll be most useful and always be seen. I'll try to get this started right away before I forget. That'll be a tiny, one-line PR!

leewujung commented 2 years ago

Good call. Thanks!