Error parsing split-beam EK60 file with problematic phase angle information

[emiliom]

@lgarzio we are posting much of our email exchange here so it's much easier to track and resolve. I'm making small edits for clarity.

(2021-6-18, from @lgarzio)

I’ve been working with a small group of people here at Rutgers trying to develop tools to process our echosounder data – both shipboard data (EK60) and an AZFP mounted on a glider.

Right now I’m trying to do the shipboard data and I’m running into an issue with converting some of the raw files.

Screen shot of the main error:

What is ping_data_dict['angle']? The best we can tell, it might be equivalent to ‘angular position of raw pings’ in Echoview but we weren’t entirely sure. This is what it looks like in a file that parses correctly:

I can provide more details when I raise the issue in github, but I first wanted to see if you know off the top of your head what that variable is.

[emiliom]

(2021-6-18, from @leewujung)

This is the electric phase angle recorded by the split beam echosounder. It's curious that this errors out since we have been parsing EK files with both single and split beams without problem. Can you see what beam_type is in self.parser_obj.config_datagram["transceivers"] for the channel that errors out?

You can see a TODO at line 532 in set_groups_ek60.py -- it is possible that your echosounder was set to collect power only even though it is a split beam. If you would consider submitting a PR that would be really great!

(2021-6-18, response from @lgarzio)

I wasn’t on this cruise so I don’t know what the echosounder settings were, so I guess anything is possible.

The weird thing is that I have 2 different sets of files from this echosounder (corresponding with 2 different net tows). I am able to parse files from net tow 1, it’s just the files from net two 2 I can’t parse. My colleague is able to load and read these files in echoview if that helps.

I’ll work on submitting all the info to github, but in the meantime if you’re interested:

• Processing code: https://github.com/lgarzio/echosounder-processing/blob/master/RossSeaEAGER/EAGER_ship_acoustics_process.py
• Environment: https://github.com/lgarzio/echosounder-processing/blob/master/environment.yml

The example .raw files are in a Rutgers Box folder – I added you as an editor to the folder. The file NBP_B050N-D20180116-T015348-good.raw can be parsed (from net tow 1) and NBP_B050N-D20180117-T215840-bad.raw cannot be parsed (from net tow 2)

[emiliom]

(2021-6-21, from @lgarzio)

Ok I updated echopype to v. 0.5.1 (I was using 0.5.0 on Friday) and debugged to line 538 in set_groups_ek60.py and tried self.convert_obj.ping_data_dict['mode'][ch] but got the error “AttributeError: 'SetGroupsEK60' object has no attribute 'convert_obj'”

But self.parser_obj.ping_data_dict['mode'][ch] gave me a list of all 1s with length=1585 so I think that’s probably what you meant?

But yes, that’s definitely the issue because self.parser_obj.ping_data_dict["angle"][ch] = None. From what I can tell, that’s set in line 129 of parse_base.py

Here, when data_type = “angle”, k=1 and v is a list of None with length=1585 so it doesn’t find any data in that channel and sets self.ping_data_dict[data_type][k] = None (which in this case is self.ping_data_dict['angle'][1])

I got there by debugging to parser.parse_raw() in api.py

I’ve cc’d Grace (the PI), Schuyler (grad student that was on the cruise), Nick (@nlbeaird, PI helping with the python code), and Ailey (grad student helping with the python code).

[emiliom]

(2021-6-21, from @leewujung)

Thanks, Lori, for doing the investigation and looping in everyone!

debugged to line 538 in set_groups_ek60.py and tried self.convert_obj.ping_data_dict['mode'][ch] but got the error “AttributeError: 'SetGroupsEK60' object has no attribute 'convert_obj'”

Oh gosh I think that is a typo or legacy code: it should be parser_obj instead of convert_obj that contains the recording mode.

self.parser_obj.ping_data_dict['mode'][ch] gave me a list of all 1s with length=1585 so I think that’s probably what you meant?

Yes, I think this is where the problem is (and what that TODO is) -- the recording mode was set up to record only power data but not the split beam phase angles. Usually for split beam echosounders people also record the phase angle data, since it is what the split beam is for. :) The code currently checks for whether it is a split beam, but it should really be checking what data are actually being recorded. As a sanity check: the file should have 1585 pings, and the power data should have 1585 pings also.

(2021-6-21, response from @lgarzio)

The power data has 1585 pings and I think the file has 1585 pings? Not entirely sure how to check that, but I do know there are 3 channels and the length of the timestamps for all 3 channels is 1585 so I’m guessing that’s right.

len(parser.ping_data_dict['timestamp'][1]) = 1585

Thanks so much for your help with this! Is there a quick workaround that I can apply locally just so I can get these files to parse until you all can work on the TODO?

[emiliom]

(2021-6-22, from @lgarzio)

I just wanted to clarify one more thing. The files that can’t be parsed by echopype can be read/loaded by echoview. For those files in echoview, there is a variable called ‘angular position of raw pings’ that contains what looks like valid data. This sounds like it’s the split beam angles that aren’t found by echopype’s parser but we’re not entirely sure. Do you know if they are the same?

I just want to make sure we’re narrowing down what the actual issue is. In my view it could be one of two issues:

1. the echosounder didn’t collect angle data even though it was split beam, and your TODO should fix that issue, or
2. somehow the raw files stored the data slightly differently and the echopype parser isn’t picking up the angle data.

I’m honestly not sure if 2 is even possible but I just want to make sure we’re thinking through all of the possibilities.

(2021-6-22, response from @leewujung )

Do you mean that EchoView gives you the phase angles for the files that cannot be converted by echopype? (also, the point you got to is for setting the various groups in the netCDF file, the parser stage is before that and the data were parsed successfully.) I don't think echopype's parser has issues with not parsing the angle data correctly if they are in there, the mode=1 indicates strongly towards that. I haven't had a chance to look at the files you shared though, but I think the source is likely that TODO item.

If anyone in your team knows how the EK60 was set up that would very really helpful in resolving the source of this issue. Let us know what you find. I am quite tied up with preparing for an upcoming fieldwork so won't be able to get to this myself very soon, but if through the back and forth here we can identify the issue more narrowly, someone in our team may be able to help resolve it -- perhaps including it in v0.5.2 or .3.

(2021-6-22, response from @lgarzio)

We’re not entirely sure, but EchoView gives us something that looks like the phase angles. I’ll try to find a definitive answer to that question. We’ll also try to figure out who knows how the echosounder was set up, but unfortunately we think it might have been a marine technician on the ship in 2008 so that might not be an option.

[emiliom]

(2021-7-16, from @lgarzio)

Wu-Jung/Emilio – we have some more information that I’m hoping will help clarify this issue.

Schuyler (cc’d here) looked at the “good” and “bad” .raw echosounder files in Echoview, and added some information in the Box folder that you should have access to (because these files are too large to send over email).

“Angular position of raw pings” in Echoview is: “Sample values are two angles (degrees) describing the detected position of the sample in the beam (determined from split beam phase measurements)” which I believe is the same variable we’re having issues with in echopype.

In the shared Box folder there are several other files (other than the .raw echosounder files):

1. The “good” and “bag” .png files are screenshots of the “angular position raw pings T1” variable (120 kHz) in Echoview. The “axis angles” variable in the left panel is showing the two number combination for an individual ping – so both files appear to have this data that Echoview is reading, at least at one individual ping.
   • the “bad_zoomed” .png file is a screenshot zoomed in a bit more so you can see every ping has a different 2 number combination (1m horizontal x 10m vertical gridlines added for scale)
2. There are two exported angles.csv files - “ping angles bad” and “ping angles good”. It looks like the ping angle data matrix starts in column N. In the good.csv file, it looks like the values just repeat for each row in columns N-U, then the values are all different for the rest of the columns. I’m assuming these are the actual measurements of the ping angles. For the bad.csv, however, the same values repeat for each row for columns N-W and AB-CY (then I stopped scrolling). So, at least to my untrained eye, it appears that there isn’t much useful ping angle data in the “bad” file, and maybe Echoview just populates the missing data with random values?

We sent these files to an expert at Echoview to review and tell us what values Echoview is spitting out for ping angle when it doesn’t look like there are any valid data there. He responded today, and I put the screen shots he attached in the shared Box folder:

“Just keeping you in the loop, I’ve looked over your data and my current thinking is that something has corrupted the raw angular position values in the files, particularly the minor-axis values (example screenshots attached). Given that it occurs on all 3 channels, I’m thinking it was due to a glitch in the logging software rather than a dud transducer sector. However, our programmers are taking a closer look to confirm that we’re not doing something funny with the data at our end for some reason.”

His hunch makes sense, since we have files that don’t appear to have any issues from before and after this time period (I believe the EK60 and software were turned off then restarted for each of the 4 experiments, Schuyler/Grace please correct me if I’m wrong).

All of this makes me think that your TODO item in echopype will fix the issue (and we won’t be losing any ping angle data that might be hidden somewhere). Do you have any updates on a timeline for completing the TODO?

[emiliom]

(2021-9-20, from @lgarzio)

I hope you’re both doing well! I just wanted to follow-up on this issue and see if you have an estimated timeline for the TODO item in set_groups_ek60.py?

# TODO: below needs to be changed to use
#  self.convert_obj.ping_data_dict['mode'][ch] == 3
#  1 = Power only, 2 = Angle only 3 = Power & Angle
# Set angle data if in split beam mode (beam_type == 1)
# because single beam mode (beam_type == 0) does not record angle data

[lgarzio]

@emiliom just checking in to see if there are any updates on this issue? Thanks!!

[lgarzio]

I wrote a quick work-around to be able to parse the split-beam files I have with no beam angles recorded. I'm sure this isn't the best way to do it but it's working for me, and is still working for files that have values for beam angle (using echopype version 0.5.6):

When I debug echopype.open_raw, setgrouper.parser_obj.ping_data_dict['angle'] is:

defaultdict(<class 'list'>, {1: None, 2: None, 3: None})

Since there are no beam angles, lines 596 and 601 in set_groups_ek60.py throw errors.

line 596: self.parser_obj.ping_data_dict["angle"][ch][:, :, 0] line 601: self.parser_obj.ping_data_dict["angle"][ch][:, :, 1]

I replaced lines 591-605 with the code below. I creates an array filled with nans with the same shape as backscatter_r for angle_athwartship and angle_alongship, and I included a comment that the beam angles aren't there (most likely an error with the echosounder).

https://github.com/OSOceanAcoustics/echopype/blob/main/echopype/convert/set_groups_ek60.py#L591-L605

        # modified by lgarzio to account for split beam mode (beam_type == 1)
        # echosounders that didn't record angle data
        if self.parser_obj.config_datagram["transceivers"][ch]["beam_type"] == 1:
            if isinstance(self.parser_obj.ping_data_dict["angle"][ch], np.ndarray):
                ds_tmp = ds_tmp.assign(
                    {
                        "angle_athwartship": (
                            ["ping_time", "range_bin"],
                            self.parser_obj.ping_data_dict["angle"][ch][:, :, 0],
                            {"long_name": "electrical athwartship angle"},
                        ),
                        "angle_alongship": (
                            ["ping_time", "range_bin"],
                            self.parser_obj.ping_data_dict["angle"][ch][:, :, 1],
                            {"long_name": "electrical alongship angle"},
                        ),
                    }
                )
            else:
                vals = np.ones(np.shape(ds_tmp.backscatter_r.values))
                vals[:] = np.nan
                ds_tmp = ds_tmp.assign(
                    {
                        "angle_athwartship": (
                            ["ping_time", "range_bin"],
                            vals,
                            {"long_name": "electrical athwartship angle",
                             "comment": "angle data not recorded by split beam echosounder in error"},
                        ),
                        "angle_alongship": (
                            ["ping_time", "range_bin"],
                            vals,
                            {"long_name": "electrical alongship angle",
                             "comment": "angle data not recorded by split beam echosounder in error"},
                        ),
                    }
                )
        # end lgarzio modifications

[leewujung]

Thanks @lgarzio , your description above gives a pretty good diagnosis of how we may be able to fix this.

Could you please check (via debugging open_raw) what the values of self.convert_obj.ping_data_dict['mode'] are for all the channels?

As I had in the TODO, this should reflect the setting of what data were supposed to be collected

# 1 = Power only, 2 = Angle only 3 = Power & Angle

If it is set to 3 but you have empty angle data, then it is likely an error of the instruments. If it is set to 1, then angle data were (correctly) not stored.

[leewujung]

As a reference that may be usefull, this file from the OOI has 2 channels that do not have angle data and 1 channel that has angle data.

[leewujung]

We are in the position work on this now for v0.6.1.

@lgarzio: Could you please provide a small test file?

[lgarzio]

@leewujung example file here: https://marine.rutgers.edu/~lgarzio/echosounder/

[leewujung]

@lgarzio: Could you try out #718 on your other files to see if that resolves the problem?

Also, do you have smaller files (~<5 MB) that we could include in the tests? The file you provided is too large and would result in very long CI runs because of the need to copy file in the process. Thanks!

[lgarzio]

Sure I'm happy to test, but I've never done something like this before so I don't really know how to go about getting the updated code if it's not just updating the echopype version?

And no, sorry the smallest file I have with this particular issue is ~14 MB

[leewujung]

@lgarzio : you can install my branch directly using pip install git+https://github.com/leewujung/echopype@power-angle-mode. This branch is built on top of what's already in v0.6.0, so if you have a new environment from that but instead of using the conda-forge echopype, pip install echopype from this branch, you should have it.

Could you give us that ~14 MB file? It is still wayyy smaller than the 100 MB one you linked to previously. Thanks.

[lgarzio]

@leewujung ~14 MB file added here: https://marine.rutgers.edu/~lgarzio/echosounder/. I can work on testing the fix on the rest of the files I have. Thank you!!!

[lgarzio]

@leewujung looks good to me!! I was able to successfully parse all of files that threw the error before. I'll have to update my plotting code to fully check the entire deployment, but I did look at one of the files and the timestamps and Sv values I extracted are exactly the same as the previous version that I had manually done myself. Thank you so much this is great!!

[leewujung]

Fantastic @lgarzio! Thanks for testing this out and the test file. I will put the file in our CI and get this merged. It will be in v0.6.1 so you don't have to do the pip install from that branch after that. We're aiming for that to be a minor with just a few things, so should be relatively quick.

[leewujung]

Closing this as the fix is merged into dev.