EGN and Bank Shadows

[AdrianPinchbeck]

Just having a little test using my own data, and going through some user parameters. A couple of observations: The available option on line 91 of 0 returns an error on EGN, though changing line 90 to false seems to produce the output I would have expected above. Leave shadows option of 0 on line 109 still seems to remove bank shadows for some reason.

[AdrianPinchbeck]

Checking the logs, I noticed that just after the bedpick plots are created, remShadow defaults to 2 for substrata mapping. So maybe it changes a little early if that helps!

[CameronBodine]

Can you post the log and main file from the meta folder in the output directory?

[CameronBodine]

Substrate mapped requires shadow removal for the models to function as expected. So remShadow needs to default to 2 at that point in the script.

Are you wanting to export EGN with shadows present? The EGN workflow needs to remove shadows. If there are a lot of shadows present in the scan, the related range bin average is subsequently pulled down,which has undesirable effects on the corrected image. It may be possible to not mask the shadows in the output, however. I'll look.

[AdrianPinchbeck]

I wasn't looking for any shadow removal, but if it's require to function correctly then that's fine. I couldn't seem to avoid removal of bank shadows with either option above, even setting line 90 to false. log_2023-09-08_1039.txt Ah, it doesn't want me to send the python file, I'll see if I can change it somehow...

[CameronBodine]

You can change the file extension to .txt

[AdrianPinchbeck]

Here we go, you beet me to it! main_2023-09-08_1039 via notepad.txt

[CameronBodine]

Ah I see! I'll dig into this. Helpful stuff!

[AdrianPinchbeck]

No worries, it's great we're using it in different scenarios.

[CameronBodine]

Ok, I think I addressed the issues:

• egn_stretch now allows a value of 0 (stretch to global min and max).
• Removed option to stretch based on standard deviation. Only allowing stretch based on global min/max, or the percent of the histogram tails to clip.
• It's now possible to not mask the shadows and still do the EGN correction. Note that the correction was designed to work the best when removing the water column and removing bank shadows.

Go ahead and update PINGMapper and see if it is performing as expected. I'm going to be in the field beginning tomorrow for a week, so I will likely not be responding during that time. But post any errors, including console output, log file, and main script and I will address when I return. Good luck!

[AdrianPinchbeck]

I think this issue maybe the far ping returns are assumed to be bank edge, being relatively dark, thus resulting in in a plot that looks like it has a bankside crop. In my case the transect gradually shallows and the proportion of dark returns increase, and as a result the overall width of the mosaic narrows as the water becomes shallow.

[CameronBodine]

Can you share some screenshots slo I can see what you are talking about?

[CameronBodine]

Better yet, can you share the recording in question?

https://drive.google.com/drive/folders/1-0uBe4E2C5O3ME6gLuQ1DG-w05Egcb33

[AdrianPinchbeck]

Hopefully the recording files are winging their way over to you... Line 90: egn = True Line 108: remShadow = 0

[CameronBodine]

Ok I took a look and I have an explanation. I have found it optimal to first mask the shadows before calculating the average. That's because the shadow regions pull the mean intensity down, which will result in pushing higher intensities up. Using your recording as an example, this would cause the cobble boulder piles to be very bright at higher ranges, and thus, not interpretable. By setting remShadow=0, the correction doesn't account for these lower intensities, making the average higher, and pushing low intensities to 0. That's why you see gaps in the data; it's because those intensities were pushed to zero.

I think the only real workaround, to my knowledge and experience, would be to plan tighter transects in shallow areas. I don't think I can code a way out of this one, unfortunately.

I'm working on a formal description of the EGN correction in a forthcoming manuscript which will hopefully lend clarity to the procedure.

Just for reference, here is a screenshot to show the variation in sonar intensity due to high depth variation:

And a screenshot showing, from left to right, raw backscatter, EGN corrected backscatter with shadows present, and EGN corrected backscatter with shadows removed.

Hope this sheds a little light. Let me know if you have any more questions.

[CameronBodine]

Ok, I made it so that shadows are not masked during the EGN calculation if remShadow=0. Hopefully it functions as expected. Go ahead and update and let me know how it goes!

[AdrianPinchbeck]

Great work Cameron, and thanks for the explanation. One of the issues with preplanning waypoints for this mission is our almost six metre tidal range and variable direction change on the flood, juggling these along with weather windows and available launch times for an area of sloping ground was a challenge. I may have to think about developing a dynamic waypoint grid layout for future use! It looks as though you may have cracked it above though, I'll be able to test this out later this evening and then provide you with an update. Many thanks.

[AdrianPinchbeck]

Had a chance to test this morning, and can confirm that it is working as anticipated. I must say that the local subtleties in softness and texture are quite a surprise, and appear to confirm differences the drop-down camera has shown, eg pure sand verses sand and small stone mixture. Will be very interesting to run the whole site through using EGN. Thanks again!

[CameronBodine]

Very glad to hear it. Turns out I originally had the ability to correct the images with or without the shadows. I turned it off for some reason. I'm glad you are kicking the tires!!!