search

OSOceanAcoustics / echopype

Enabling interoperability and scalability in ocean sonar data analysis
https://echopype.readthedocs.io/
Apache License 2.0
95 stars 73 forks source link

​ValueError: ​​No parser available for type RAW version 4 #713

Closed emiliom closed 2 years ago

emiliom commented 2 years ago

From @cornejotux:

​I have been trying to read my EK80 files but I just can't get it to work, I get the message:

11:32:41  parsing file reclas sesgo 0522-D20220516-T131807.raw, time of first ping: 2022-may-16 13:18:07
​​ValueError: ​​No parser available for type RAW version 4

Do you have any idea of what I'm doing wrong?

cornejotux commented 2 years ago

Thanks, @emiliom, I don't know what I didn't think about opening an issue myself.

emiliom commented 2 years ago

Can you include the exact statement you used that generated this error? I'm assuming it's from open_raw, but it'll be very helpful to see the actual statement. Also, what echopype version are you using? We released a major new version, 0.6.0, a few days ago! See https://echopype.readthedocs.io/en/stable/whats-new.html

Also, if your raw file isn't very large, would you be able to share it with us so we can look more closely?

@leewujung may have insight on the error message

Update: From my initial inspection it looks to me like echopype doesn't currently support EK80 RAW version 4. @leewujung will be able to verify this more definitively. I see support for version 0, 3 and possibly 1. I don't know enough about Simrad RAW versions to say anything about version 4 and how different it is from version 3.

cornejotux commented 2 years ago

Ok.. thanks... Maybe there is a way to replay the files and save them as an older version. I will try to do that and if that solves the problem I will report it here. Thanks for your help.

gavinmacaulay commented 2 years ago

RAW4 contains complex sample data from the transmit pulse (intended to allow calculation of transducer impedance). RAW4 datagrams can be ignored by echopype as the normal RAW3 will still be in the files (see page 149 of https://www.simrad.online/ek80/interface/ek80_interface_en_us.pdf).

emiliom commented 2 years ago

Thanks @gavinmacaulay ! That's very promising. @cornejotux if you could share a small raw file (ideally < 100 MB), that would let us investigate what it might take to do what @gavinmacaulay is suggesting.

leewujung commented 2 years ago

Thanks all for the discussions! I think it'll be relatively straightforward to add support for RAW4 sample datagram into echopype. If @cornejotux could share a sample file that'll let us test the new support.

I think we can quickly put in a "non-patch" for now to skip it with a warning, and put the RAW4 parsingi n the next minor release.

cornejotux commented 2 years ago

Ok... Here is the code that I used:

import echopype as ep
from echopype import open_raw
print(ep.__version__) 
ed = open_raw('test-D20220514-T172704.raw', sonar_model='EK80')  # for EK80 file

Echopype version is 0.5.6 and the error is the came.

I'm attaching the raw file so you can test it test-D20220514-T172704.zip

Thanks in advance.

cornejotux commented 2 years ago

Not sure if this helps, but I just try with the 0.6 version and got this:

`>>> print(ep.version) 0.6.0

ed = open_raw('test/test-D20220514-T172704.raw', sonar_model='EK80') # for EK80 file 10:22:08 parsing file test-D20220514-T172704.raw, time of first ping: 2022-May-14 17:27:04 Traceback (most recent call last): File "", line 1, in File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/api.py", line 428, in open_raw parser.parse_raw() File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/parse_base.py", line 101, in parse_raw self._read_datagrams(fid) File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/parse_base.py", line 206, in _read_datagrams new_datagram = fid.read(1) File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/utils/ek_raw_io.py", line 437, in read return self._read_next_dgram() File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/utils/ek_raw_io.py", line 347, in _read_next_dgram nice_dgram = self._convert_raw_datagram(raw_dgram, bytes_read) File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/utils/ek_raw_io.py", line 376, in _convert_raw_datagram nice_dgram = parser.from_string(raw_datagram_string, bytes_read) File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/utils/ek_raw_parsers.py", line 81, in fromstring id, version = self.validate_data_header(header) File "/home/cornejo/.virtualenvs/pythonProject/lib/python3.8/site-packages/echopype/convert/utils/ek_raw_parsers.py", line 72, in validate_data_header raise ValueError("No parser available for type %s version %d" % (self._id, version)) ValueError: No parser available for type RAW version 4 `

leewujung commented 2 years ago

Thanks @cornejotux ! there was no change on the parser side from v0.5.6 to v0.6.0, so this is the same error.

emiliom commented 2 years ago

@cornejotux thanks for providing the run time information and the sample file; at 3.8 MB, that's a perfect size.

And great to see that @leewujung is already on it (PR #714)! :exploding_head:

emiliom commented 2 years ago

I can confirm that @leewujung 's PR #714 lets your sample file. Here's a screenshot, using a conda environment based on the PR. One small note: in the screenshot I hide the run-time messages from ep.open_raw b/c there are a lot of "Unknown datagram type: RAW4" lines; I assume those are the RAW4 datagrams that are ignored.

If you want to be on the bleeding edge and try this, you can install her branch directly from her repo (but I make no promises on how smooth the installation will be!):

pip install git+https://github.com/leewujung/echopype@add-raw4

PS. I use squeeze() because after inspecting the echodata object I saw that the file has only one channel (frequency).

image

leewujung commented 2 years ago

I just pushed up something that handles that datagram so you won't get that warning. At the moment in my branch RAW4 data are parsed and organized but not saved into the EchoData object yet. That's the next task in #714.

leewujung commented 2 years ago

@cornejotux: #714 should be ready to go, and there's just a last thing about a dimension (https://github.com/OSOceanAcoustics/echopype/pull/714#issuecomment-1146745401). I added the file you provided into our tests.

Could you try to run this branch on your other files to see if things work fine?

Do you happen to have a file with more than 1 channel? It would be good to see if everything behaves correctly using such a file.

Thanks!

cornejotux commented 2 years ago

ok, I try other files and the open_raw works like a charm. I do not have multibeam files that we can test.

leewujung commented 2 years ago

@cornejotux: great! We'll get that into v0.6.1 and push out that release soon.

I do not have multibeam files that we can test.

No a multibeam, but a multi-transducer file (so more than 1 frequency). Do you happen to have one of those? No worries if not!

cornejotux commented 2 years ago

I do not...

leewujung commented 2 years ago

Closing as it is fixed in #714 .

cornejotux commented 1 year ago

I will try it during the week, but I do not have multichannel files.

Jorge F. Cornejo-Donoso PhD Marine Science @.*** http://www.jorgecornejo.net

El dom, 5 jun 2022 a la(s) 01:36, Wu-Jung Lee @.***) escribió:

@cornejotux https://github.com/cornejotux: #714 https://github.com/OSOceanAcoustics/echopype/pull/714 should be ready to go, and there's just a last thing about a dimension (#714 (comment) https://github.com/OSOceanAcoustics/echopype/pull/714#issuecomment-1146745401 ). I added the file you provided into our tests.

Could you try to run this branch on your other files to see if things work fine?

Do you happen to have a file with more than 1 channel? It would be good to see if everything behaves correctly using such a file.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/OSOceanAcoustics/echopype/issues/713#issuecomment-1146745904, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACN5LEXO6WM4VNGYRR4KZ4LVNQ4ELANCNFSM5XWU4W5Q . You are receiving this because you were mentioned.Message ID: @.***>