open_raw of EA640 encounter self.parser_obj.environment["temperature"]], KeyError: 'temperature'

[cywhale]

I have some .raw files from two ships, both by EA640. For one ship open_raw is ok, but fails for the other. I use echopype version 0.5.5

ed = ep.open_raw(
    raw_file="xxx.raw", 
    sonar_model="EK80", 
)

Error message:

KeyError                                  Traceback (most recent call last)
/tmp/ipykernel_70694/522230037.py in <module>
      1 #echodata = ep.open_raw(raw_file=ek_nor3, sonar_model="ek80")
----> 2 ed0 = ep.open_raw(
      3     raw_file=rawfiles[0],
      4     sonar_model="EK80",
      5 ) #files[0] #just debug

~/py3.8/lib/python3.8/site-packages/echopype/convert/api.py in open_raw(raw_file, sonar_model, xml_path, convert_params, storage_options)
    441             sonar_model=sonar_model, date_created=parser.ping_time[0]
    442         )
--> 443     echodata.environment = setgrouper.set_env()
    444     echodata.platform = setgrouper.set_platform()
    445     if sonar_model in ["EK60", "ES70", "EK80", "ES80"]:

~/py3.8/lib/python3.8/site-packages/echopype/convert/set_groups_ek80.py in set_env(self, env_only)
     29                 "temperature": (
     30                     ["ping_time"],
---> 31                     [self.parser_obj.environment["temperature"]],
     32                 ),
     33                 "depth": (["ping_time"], [self.parser_obj.environment["depth"]]),

KeyError: 'temperature'

Is there something wrong with the .raw? or any way to prevent this error if I just want to parse the NMEA log out from the raw file. Thanks.

[cywhale]

I just found a much smaller .raw that can be tested for this issue and uploaded to my repo: https://github.com/cywhale/ODB/blob/ff7aa49e11403ae10d17729d211598631dbe8691/EK/test.raw

echodata = ep.open_raw(raw_file="test.raw", sonar_model="ek80")

and got KeyError: 'temperature' FYI

[leewujung]

@cywhale : Awesome, thanks for reporting this and providing the test file! I wonder whether it is just that that key does not exist in the file so the parsed dict does not content that. Let me take a look.

[leewujung]

@cywhale : somehow the error I ran into for this test.raw is SimradEOF instead of the temperature setting problem. Could you test again?

[cywhale]

@cywhale : somehow the error I ran into for this test.raw is SimradEOF instead of the temperature setting problem. Could you test again?

@leewujung Thanks for the help. But I just tested it by downloading this test.raw again from my repo, used open_raw (in a jupyter notebook), and got the same error. (I never met SimradEOF before) See result (clipped png), https://github.com/cywhale/ODB/blob/master/EK/test_raw.png

[leewujung]

@cywhale : You're right - somehow the files I downloaded earlier were both corrupted with different sizes (I tried twice because I was thinking maybe it was that). Not sure why... I'll look into the set_env problem.

[leewujung]

@cywhale : Could you please try #537 to see if the fix can successfully convert your other larger files?

Also, do you have access to the recording software? If so, could you please try to collect a smaller file that we could use as a test file? Assuming the recording software is very similar to EK80 (I don't have experience with EA640), this can be done in 2 ways: collect a new data file, or "replay" an existing file while recording. Just a few pings would be fine, to make sure that the code run through the conversion mechanism without problem. Thanks!

[cywhale]

@leewujung I just tested a .row with 300Mb size, and it works. Thanks. Got a warning: "WARNING: The water_level_draft was not in the file. Value set to NaN." seems due to #540

$pip freeze | grep echopype
echopype @ git+git://github.com/OSOceanAcoustics/echopype.git@39ff073f537e93edfa1c27d835791179688eb665

Our research vessels will not have cruises till March and should be in Years Repair. But I still have many other .raw of two vessels (say V2, V3, and both EA640) with different file size. Although most of them are huge. I can try to find some raw files for testing if needed?

BTW, is there any chance that open_raw could lost some NMEA datagram information? Maybe it's another issue, or just some operational problem for EA640 on these two vessels. Now I use new commit of echopype to open raw, but only V3 has $SDDPT (water depth relative to transducer) in NMEA datagram, no $SDDPT in V2 (but still has $GPGGA). It confuses me when testing these raw files.

[leewujung]

@cywhale : Great that the fix works out! I've included this in the upcoming release (v0.5.6 -- it is a relatively smaller one) and we're aiming for this upcoming week.

You are correct that the water_level_draft setting warning is because that is not present in the data file.

For the test file, I think that if you could replay the file in the recording software, you can hit "record" and then just record a few pings. We can talk about this when we connect on Thursday. :)

For the NMEA datagram, echopype uses pynmea2 to parse NMEA info. The package is active so I suspect that the datagrams are not dropped. The NMEA datagram can come from different sources, and often the ship, so you could ask the marine tech if both of their devices have that sentence set up.

The water level values echopype stores is from the environment datagram, which is an XML string, so not from the NMEA string.

One thing I'd mention as an FYI in case it has any bearing here: EA640 has an additional datagram NME1, compared to only NME0 in EK80. I wonder if that's because there can be 2 devices and therefore 2 sources of NMEA string data?

[cywhale]

@leewujung I think I cannot access that software. These raw files were copied by V2 technicians to ODB. Thanks for the information about NMEA and NME1. I don't know that. I'll check it if possible.

[leewujung]

This is addressed by #537 .