leewujung
commented
2 years ago @brandynlucca : Thanks for reporting the bugs and the very thorough investigation!! I believe you didn't break anything and it is the code that needs improvements.
The NaN padding is a strategy we use to get data from all channels into a gridded form for slicing, and it is great (!) to know that the current calibration routine breaks with the interleaving NaNs for sequential pings. I quickly looked into the section where things broke and saw a TODO for exactly the problem you saw: https://github.com/OSOceanAcoustics/echopype/blob/12693a72badcf54a18408e694b72e7b0e6d48023/echopype/calibrate/calibrate_ek.py#L528-L529
I will need probably until the weekend to get to this. If you wouldn't mind sharing a small example file that will really help with changing/debugging the implementation. This gives me a push to look into the EK80 cal code too, so thanks! (I think in addition to this issue the current implementation is also pretty slow due to the mechanism used to do pulse compression)
For the MVBS problem, it is a regression bug at v0.6.0 that should be fixed in #736 (but not merged yet). Could you give it a try to see if that resolved the problem?
Also, thanks for pointing out the >=3.8 problem. We'll get that fixed!
brandynlucca
petejan
ctuguinay
Hi all. I didn't see this in either of the current open or closed issues, but I am running into a data processing issue with Simrad *.raw files containing broadband multiple frequency data. I followed the threads and chain of issues/pulls in both #308 and #657, but those didn't seem to directly help address the issues I was encountering. I apologize if I missed a thread that does, in fact, address this exact issue. Namely, some of the issues addressed between 0.5.6 and 0.6.0 didn't appear to resolve errors involving changing data values for
transmit_power,slope,transmit_duration_nominal,frequency_start, andfrequency_end.Background
OS: Windows 10.0.19043 Python version: 3.7.9 and 3.8.10 Echopype version: 0.5.6 and 0.6.0 Simrad EK80 software version: 2.0.0
An aside: I ran
Echopype 0.5.6inPython 3.7.9andEchopype 0.6.0inPython 3.8.10. ForPython 3.7.x, there is an error in trying to installEchopype 0.6.0due to certain dependencies requiringPython >= 3.8(namelyxarray). So the recommended Python version in the Installation Guide may need to be amended for Windows. This was detected via:pip install git+https://github.com/OSOceanAcoustics/echopype.git#egg=echopypeIssue I am trying to call in EK80 raw files that includes four frequencies (38, 70, 120, 200 kHz) where the 38 and 200 kHz come from a dual frequency transducer (38/200 Combi D). All four comprise broadband/complex measurements. The fileset I am currently processing is from May 2021. I haven't run into any significant issues when trying to open the raw files. However, I do run into an issue when trying to compute Sv or attempt really any other processing function. This also includes trying to visualize the echogram. I have been able to successfully run the examples from the provided notebook using EK60 data.
The issue appears to be related to how some of the arrays for certain variables are mapped to the
frequencyandping_timecoordinates. So the initial Error message I receive is as follows (Note: ultimately, the flagged error is identical in bothEchopype 0.5.6andEchopype 0.6.0):Resulting error
Looking under the hood in
beamfrom myEchoDataobject (I used the equivalent indexing notation, e.g.ed.beam, for theEchopype 0.5.6release):I wasn't sure if the padded
nanvalues here were related to #681, but when checking whether there is a different transmit duration included in this array:It makes sense since pings were sequential so the
ping_timecoordinate won't include actual data for each frequency. Looking at the first 6 pings:So that matches what would be expected since the 38/200 pinged simultaneously followed by the 70 and 120 kHz transducers, respectively. So, presumably, this should be the same for other variables mapped with the same coordinates (i.e.
frequencyandping_time), which it is (e.g.slope):So when I just replace the
nanplaceholders with the samefloat32value, the issue gets passed onto the next variable (soslopein this case):Changing slope:
Same error, just for
slopenow:Repeating this by replacing the
nanplaceholders intransmit_power,frequency_start, andfrequency_end(the latter two required a slightly different brute force approach since they apparently have a different dimension than the other data variables listed above) appears to resolve the issue since functions likecompute_Sv(...)will now operate as expected:But this hack to brute force
compute_Sv(...)unfortunately appears to potentially break other features (or this may be just a general 'feature' on how these *.raw files are formatted regardless of the above hack). For instance, I am able to generate the unbinned Sv echograms without much trouble (attached is the related figure for the 70 kHz):I can manually toggle through each of them to pull the channel/frequency I would like. But as soon as I average by bin via
compute_MVBS(...), things seem to break for all frequencies except 38 kHz where the echogram is able to be generated.It's certainly quite possible that I broke something in the process and that is what is triggering the above issue, but I have replicated the exact issue and 'hack' in 10 unique EK *.raw files from other cruises where we also collected broadband measurements with sequential pings. Any guidance would be greatly appreciated!