search

OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
95 stars 73 forks source link

EK80 pulse compressed Sv output values #308

Closed leewujung closed 1 year ago

leewujung commented 3 years ago

There are currently unresolved issues related to matching echopype pulse compressed Sv with those generated by Matlab code and EchoView. The Matlab and Echoview pulse compression outputs are also different. This was originally in #278 but needs further investigation so better to be an independent issue.

The mismatch with Matlab pulse compressed Sv are likely due to a hard-coded range offset (which selects for the starting sample index to be one closest to 0.02 m) in the Matlab code which we should investigate for its reason. The mismatch is smaller at longer range, which is consistent with this potential reason.

We should connect with EchoView to see how they do pulse compression under the hood to make sure that we know and can reproduce their results.

leewujung commented 2 years ago

Currently we only have outputs from Matlab routines (Read EK80 raw data.zip from Dr. Dezhang Chu, with other authors) to test against for EK80 calibration. Here the calibration refers to the band-summation one based on center frequency instead of the frequency-dependent TS. However there is unresolved discrepancy between the values generated by Matlab and by echopype.

@gavinmacaulay if you could take a look at these in comparison with Echoview and LSSS that will be really great! I don't have access to either of them. Chu helped generated some pulse compressed outputs for us to compare with for 1 file (test), but we also had quite some discrepancy there.

The matlab code has a selection of minRange and maxRange when selecting which sample to include in pulse compression, where the minRange in the example I got is set to 0.02 m. I suspect that the range calculation is where the discrepancy between echopype and Matlab code comes from. The range-related code is in EK80readRawFiles_rev.m which calls EK80readRawV3_rev.m. Starting point is EK80_read_raw_file.m at the root level.

The related echopype tests with unresolved issues are here: https://github.com/OSOceanAcoustics/echopype/blob/main/echopype/tests/calibrate/test_calibrate.py#L109-L158

To run Sv computation, these 3 test functions point to the files to use (they are in the CI_test_data folder) and how the the calibration routines are called: https://github.com/OSOceanAcoustics/echopype/blob/main/echopype/tests/calibrate/test_calibrate.py#L161-L187

The actual calibration code is in https://github.com/OSOceanAcoustics/echopype/blob/main/echopype/calibrate/calibrate_ek.py Range is computed via compute_range and the calibration code is in _cal_complex.

gavinmacaulay commented 2 years ago

The code that Chu provided is 95% written by me - I haven't updated it for about 6 years (once LSSS fully supported EK80 data I stopped actively using that code) so am not surprised that it doesn't give the correct answers given all the changes in the EK80 data formats since then.

leewujung commented 2 years ago

Awesome - I am so happy to be able to finally track these things down! 😃 From just the code it is hard to tell if some parameters are hard-wired for the system (i.e. should not change).

A note on the cross-correlation implementation: I was experimenting with xarray functionality for rolling/construct at the time. There are a few ways to try to speed things up like using optimized fft routines, once we have the operations sorted out.

leewujung commented 2 years ago

Note from WGFAST 2022: Lars Anderson from Simrad now has python code implemented for these, so we can cross check and resolve this!

leewujung commented 2 years ago

Note from working on #657 that when we get to this we can clean up the current use of freq_center as a selector for CW/BB waveform mode, since with the coordinate frequency --> channel change most computations are now indexed by channel.