MEGA data processing errors

[philw6]

There are a few errors that pop up when processing MEGA data, not sure if its only for this transducer set-up.

The first is that the horizontal distance of the output processed data is much less than actual. For instance the right bank is ~21m from the boat track in the attached file, where as PyHum is reading it as ~7m. . This does not happen if I set the transducer to the incorrect "1199" (instead of "'mega'"), whereby the correct horizontal distance is returned.

Shadow removal does seem to remove anything beyond about 10m from the transducer where as the data was collected out to 30m. Contouring only returns a thin strip at the edge of the data area (see attached).

The second issue is that I get a memory error in the gridding process where by it keeps trying larger and larger resolutions until infinity, regardless of having "99" or "1" in the res setting. I have to hit exit/enter in the CMD terminal to get it to stop. This also happens when I used the (incorrect) 1199 transducer type in the script.

I dont know if its relevant but I get the "memory mapping failed in sliding window - trying memory intensive version" in the initial read module. It seems to continue from this point OK though. Using a Windows 10 machine, i7-8550 with 16GB RAM.

Does anyone know the correct transducer array length for the MEGA units (XM 9 20 MSI T)? It physically measures about 16-18cm but I cant find details about the internal array length.

Script below:

-- coding: utf-8 --

syntax for script without file path included: (pyhum) C:\Users\philw>python bhp_pyhum_pw1.py -i C:\Users\philw\BHPsonar\R00170.DAT -s C:\Users\philw\BHPsonar\R00170\

""" Spyder Editor

This is a temporary script file. """ import sys, getopt

from Tkinter import Tk from tkFileDialog import askopenfilename, askdirectory

import PyHum import os

if name == 'main':

argv = sys.argv[1:]
humfile = ''; sonpath = ''

# parse inputs to variables
try:
   opts, args = getopt.getopt(argv,"hi:s:")
except getopt.GetoptError:
     print 'error'
     sys.exit(2)
for opt, arg in opts:
   if opt == '-h':
     print 'help'
     sys.exit()
   elif opt in ("-i"):
      humfile = arg
   elif opt in ("-s"):
      sonpath = arg

# prompt user to supply file if no input file given
if not humfile:
   print 'An input file is required!!!!!!'
   Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
   humfile = askopenfilename(filetypes=[("DAT files","*.DAT")]) 

# prompt user to supply directory if no input sonpath is given
if not sonpath:
   print 'A *.SON directory is required!!!!!!'
   Tk().withdraw() # we don't want a full GUI, so keep the root window from appearing
   sonpath = askdirectory() 

# print given arguments to screen and convert data type where necessary
if humfile:
   print 'Input file is %s' % (humfile)

if sonpath:
   print 'Son files are in %s' % (sonpath)

doplot = 1 #yes

# reading specific settings
cs2cs_args = "epsg:4326" #WGS84
bedpick = 1 # auto bed pick
c = 1450 # speed of sound fresh water
t = 0.108 # length of transducer for MEGA unit not confirmed
draft = 0.3 # draft in metres
flip_lr = 0 # flip port and starboard
model = 'mega' # humminbird model
calc_bearing = 0 #no
filt_bearing = 0 #no
chunk = 'd100' # distance, 100m
#chunk = 'p1000' # pings, 1000
#chunk = 'h10' # heading deviation, 10 deg

# correction specific settings
maxW = 1000 # rms output wattage for MEGA unit
dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
ph = 7.0 # acidity on the pH scale
temp = 26.0 # water temperature in degrees Celsius 20 at S4 dike and 27 at Mascarenhas in Oct 2017
salinity = 0.0 #salinity = 0.0

# for shadow removal
shadowmask = 0 #auto shadow removal
win =10

# for mapping
res = 99 # grid resolution in metres
# if res==99, the program will automatically calc res from the spatial res of the scans
mode = 1 # gridding mode (simple nearest neighbour)
#mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
#mode = 3 # gridding mode (gaussian weighted nearest neighbour)
dowrite = 0 #disable writing of point cloud data to file
scalemax = 60 # max color scale value (60 is a good place to start)

## read data in SON files into PyHum memory mapped format (.dat)
PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk) #cog

## correct scans and remove water column
PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)

## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
win = 10
numclasses = 4
PyHum.texture2(humfile, sonpath, win, doplot, numclasses)

## grid and map the scans
cs2cs_args = "epsg:4326"
res = 99
mode = 1
nn = 64
numstdevs = 4
use_uncorrected = 0
scalemax = 60
PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected, scalemax) #dowrite, 

[dbuscombe-usgs]

Interesting. Please provide this data set and I will investigate these issues. Thanks for your patience as these issues with the 'new' data formats get ironed out

[philw6]

The .SON files are very large... can I send them to you via FTP link? To your email address perhaps?

[dbuscombe-usgs]

sure thing. however you want to send is fine with me

[dbuscombe-usgs]

I've fixed the issues associated with incorrect reported range, but I am also getting errors in the map module, because either the heading is incorrect or some other reason that is creating latitudes outside of the range -90 to +90. I haven't been able to figure it out. I've tried various epsg codes etc. Is there something unusual about your setup, such as an external non-Humminbird GPS?

[dbuscombe-usgs]

Ok, I figured it out. For some reason, your data doesn't work with

epsg:4326

I managed to run the map module with

epsg:29101

using the latest version 1.4.6

[philw6]

Hi Daniel,

I almost have it working except now the requirement to add in values for all variables has me stumped on a few. Please see the python input script and the cmd output. It seems to be missing some values for the plot_contours module which I cant find any documentation to work out what they should be.

Are you able to take a quick look at the attached and advise on what I need to do from here?

On another note, we are a company based in Australia that does a lot of river and marine geomorphic analysis and so could potentially collaborate with you given that much of your research appears to be in the same area. We are going to the field again in March/April to the Brazil location that I provided data for previously. There are actually 20 sites across a 650km river system including a variety of substrates and habitat types. We are also taking Hydroacoustic (fish biomass) measurements in conjunction with the Humminbird sonar data, using a Biosonics DT-X system. The hydroacoustic biomass is actually the main data for the project where as we are using the Humminbird as a supporting dataset to look at habitats (e.g. visually identifying woody debris etc and using Reefmaster to extract E1, E2 and PSV values). If you were every interested in collaboration/consulting let me know.

Regards,

Phil

-----Original message----- From: Daniel Buscombe notifications@github.com Sent: Saturday 27th January 2018 14:12 To: dbuscombe-usgs/PyHum PyHum@noreply.github.com Cc: Phil Whittle Phil.Whittle@hydrobiology.biz; Author author@noreply.github.com Subject: Re: [dbuscombe-usgs/PyHum] MEGA data processing errors (#52)

Ok, I figured it out. For some reason, your data doesn't work with

epsg:4326

I managed to run the map module with

epsg:29101

using the latest version 1.4.6

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

  (PyHum) C:\Users\philw>python bhp_pyhum_pw1.py -i C:\Users\philw\BHPsonar\R00170.DAT -s C:\Users\philw\BHPsonar\R00170\ Input file is C:\Users\philw\BHPsonar\R00170.DAT Son files are in C:\Users\philw\BHPsonar\R00170\ Input file is C:\Users\philw\BHPsonar\R00170.DAT Son files are in C:\Users\philw\BHPsonar\R00170\ cs2cs arguments are epsg:4326 Draft: 0.3 Celerity of sound: 1450.0 m/s Transducer length is 0.108 m Bed picking is auto Chunks based on distance of 100 m Data is from the MEGA series Checking the epsg code you have chosen for compatibility with Basemap ... ... epsg code compatible WARNING: Because files have to be read in byte by byte, this could take a very long time ... low-frq. downward scan not available port sonar data will be parsed into 8.0, 99 m chunks starboard sonar data will be parsed into 8.0, 99 m chunks memory-mapping failed in sliding window - trying memory intensive version high-freq. sonar data will be parsed into 8.0, 99 m chunks Processing took 68.0569080364seconds to analyse Done!

Input file is C:\Users\philw\BHPsonar\R00170.DAT Sonar file path is C:\Users\philw\BHPsonar\R00170\ Max. transducer power is 1000.0 W Salinity is 0.001 ppt pH is 7.0 Temperature is 26.0 Suspended sediment size/conc. file is None Processing took 41.5166579791seconds to analyse Done!

Input file is C:\Users\philw\BHPsonar\R00170.DAT Sonar file path is C:\Users\philw\BHPsonar\R00170\ Window is 10 square pixels Threshold dissimilarity (shadow is <) is 3 Threshold correlation (shadow is <) is 0 Threshold contrast (shadow is <) is 6 Threshold energy (shadow is >) is 0 Threshold mean intensity (shadow is <) is 4 Processing took 61.9968000025seconds to analyse Done!

Input file is C:\Users\philw\BHPsonar\R00170.DAT Sonar file path is C:\Users\philw\BHPsonar\R00170\ Window is 10 square pixels Number of sediment classes: 4 [Default] Number of processors is 8 processing port side ... processing starboard side ... Plotting ... Plotting ... Plotting ... Plotting ... Plotting ... Plotting ... Plotting ... Plotting ... Traceback (most recent call last): File "bhp_pyhum_pw1.py", line 112, in PyHum.texture2(humfile, sonpath, win, doplot, numclasses) File "C:\Users\philw\Anaconda2\envs\PyHum\lib\site-packages\PyHum_pyhum_texture2.py", line 324, in texture2 plot_contours(dist_m, shape_port, class_fp[p], ft, humfile, sonpath, base, numclasses, p) TypeError: plot_contours() takes exactly 11 arguments (9 given)

(PyHum) C:\Users\philw>

[dbuscombe-usgs]

Apologies for the delay in my response - the plotting routines in the texture module need an overhaul. I promise I will try to make it a priority soon. In the meantime, I suggest playing with the input parameter 'win', or just making your own plots with the exported data. Remember, PyHum is an API. You can incorporate it into your own scripts for making plots. I am planning a series of ipython notebooks that demonstrate some examples of this. Please bear with me - things progress slowly.

Also, you may use the e1e2 module in pyhum to compute E1 and E2 values. This is a little-used function within PyHum that I'd be happy to revise as directed

Always interested in collaboration - just let me know how I can help!