Extracting Information from .SON and/or .IDX file

[ryanhefley]

This is not a technical issue per se, but the reason that I have stumbled upon PyHum is because of a problem that one of our engineers was having. He has .SON and .IDX files (B001.IDX,B001.SON,B002.IDX,B002.SON,B003.IDX,B003.SON,B004.IDX,B004.SON)

This was done on the Arkansas River, in Arkansas which our coordinate system we use is NAD 83 15N

I was showing him PyHum, and he really doesn't care to much about the graphs and images, but rather he would like to extract the following information {lat, long, depth}

How could I use PyHum to extract this information in a CSV that allows him to view this information so we can plot in ArcGIS?

Thank you so much, sorry for my lack of knowledge but I tried to dig into the documentation, and was getting confused fairly quickly.

[danhamill]

IIUC I think you are looking for the file that ends with 'rawdat.csv'. Is that what you are looking for?

[ryanhefley]

@danhamill , this is exactly what he needs!

So now that I have PyHum working I beleive using the Anaconda Prompt, well at least if I do the following within: activate PyHum python -c"Import PyHum;PyHum.dotest()

What procedure would I need to do in order for me to take in these 4 files with the respected .IDX and .SON and output our own data?

I apologize if I'm asking to much. I'm just really needing this data for a project and I'm very new to Python and esp working with libraries

[danhamill]

You need to create your own python scripts. The test.py is included within the python package to check the functionality of the program on your machine.

I would create a new python script (lets say its called myscript.py) in your desired working directory that looks something like:

import PyHum

def my_func():

   # copy files over to somewhere read/writeable

   # general settings   
   humfile = #<-----your path here
   sonpath = #<-----your path here

   doplot = 1 #yes

   # reading specific settings
   cs2cs_args = "epsg:26949" #arizona central state plane
   bedpick = 1 # auto bed pick
   c = 1450 # speed of sound fresh water
   t = 0.108 # length of transducer
   draft = 0.3 # draft in metres
   flip_lr = 1 # flip port and starboard
   model = 998 # humminbird model
   calc_bearing = 1 #1=yes
   filt_bearing = 1 #1=yes
   chunk = '1' ##'d100' # distance, 100m
   #chunk = 'p1000' # pings, 1000
   #chunk = 'h10' # heading deviation, 10 deg

   # correction specific settings
   maxW = 1000 # rms output wattage
   dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
   correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
   ph = 7.0 # acidity on the pH scale
   temp = 10.0 # water temperature in degrees Celsius
   salinity = 0.0

   # for shadow removal
   shadowmask = 0 # 0= automatic shadow removal, 1=manual
   win = 31
   dissim=3
   correl=0.2
   contrast=6
   energy=0.15
   mn=4

   # for texture calcs
   shift = 50 ##10 # pixel shift
   density =win/2 # win/2 
   numclasses = 8 #4 # number of discrete classes for contouring and k-means
   maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis)
   notes = 4 # Notes per octave (for wavelet analysis)

   # for mapping
   res = 0.1 #99 # grid resolution in metres
   # if res==99, the program will automatically calc res from the spatial res of the scans
   mode = 1 # gridding mode (simple nearest neighbour)
   #mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
   #mode = 3 # gridding mode (gaussian weighted nearest neighbour)
   use_uncorrected = 0

   nn = 64 #number of nearest neighbours for gridding (used if mode > 1)
   ##influence = 1 #Radius of influence used in gridding. Cut off distance in meters 
   numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept 

   # for downward-looking echosounder echogram (e1-e2) analysis
   beam = 20.0
   transfreq = 200.0 # frequency (kHz) of downward looking echosounder
   integ = 5
   numclusters = 3 # number of acoustic classes to group observations

   ## read data in SON files into PyHum memory mapped format (.dat)
   PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk) #cog

   ## correct scans and remove water column
   PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

   ## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
   PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot, dissim, correl, contrast, energy, mn)

   win = 10
   PyHum.texture2(humfile, sonpath, win, doplot, numclasses)

   ## grid and map the scans
   PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite, 

   ## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar
   PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)

   res = 1 # grid resolution in metres
   numstdevs = 5

   ## grid and map the texture lengthscale maps
   #PyHum.map_texture(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs)

   #res = 0
   #nn = 5 # noise threshold in dB W
   #noisefloor = 10 # noise threshold in dB W
   #weight = 1 ##based on grazing angle and inverse distance weighting

   ## create mosaic out of all chunks with weighting according to distance from nadir, grazing angle, or both
   #PyHum.mosaic(humfile, sonpath, cs2cs_args, res, nn, noisefloor, weight)

   ##win = 200 #100 # pixel window 
   ## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
   ##PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)

   ## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
   ## implemented using the superpixel approach
   ##PyHum.texture_slic(humfile, sonpath, doplot, numclasses, maxscale, notes)

if __name__ == '__main__':
  my_func()

Then if you are within your working directory with the python script in the anaconda prompt:

activate pyhum 
python my_script.py

[ryanhefley]

@danhamill , so do you need a .DAT file for the "humfile"? If so, we have no .DAT file available. All i am given is the .SON and .IDX files

[danhamill]

Thats correct. The dat file is required to "decode" the data stream. @dbuscombe-usgs will probably have a more verbose answer.

[ryanhefley]

@danhamill that is what I was thinking myself. Okay, I'm going to drive in some questions to the engineering requesting this and figure out why we don't have .DAT files available. Thank you so much for all the helpful information. I really do appreciate it.

[danhamill]

@rhefley Thanks for using PyHum

[ryanhefley]

@danhamill , I swear I will get this working sometime today! I'm almost there I think!

import PyHum

def my_func():

   # copy files over to somewhere read/writeable

   # general settings   
   humfile = "C:\\Users\\ryan.hefley\\Desktop\\R00001.dat"#<-----your path here
   sonpath = "C:\\Users\\ryan.hefley\\Desktop\\R00001"#<-----your path here

   doplot = 1 #yes

   # reading specific settings
   cs2cs_args = "epsg:26915" #ARKANSAS UTM 1983 15N
   bedpick = 1 # auto bed pick
   c = 1450 # speed of sound fresh water
   t = 0.186 # length of transducer
   draft = 0.3 # draft in metres
   flip_lr = 1 # flip port and starboard
   model = 998 # humminbird model
   calc_bearing = 1 #1=yes
   filt_bearing = 1 #1=yes
   chunk = '1' ##'d100' # distance, 100m
   #chunk = 'p1000' # pings, 1000
   #chunk = 'h10' # heading deviation, 10 deg

   # correction specific settings
   maxW = 1000 # rms output wattage
   dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
   correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
   ph = 7.0 # acidity on the pH scale
   temp = 15.0 # water temperature in degrees Celsius
   salinity = 0.0

   # for shadow removal
   shadowmask = 0 # 0= automatic shadow removal, 1=manual
   win = 31
   dissim=3
   correl=0.2
   contrast=6
   energy=0.15
   mn=4

   # for texture calcs
   shift = 50 ##10 # pixel shift
   density =win/2 # win/2 
   numclasses = 8 #4 # number of discrete classes for contouring and k-means
   maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis)
   notes = 4 # Notes per octave (for wavelet analysis)

   # for mapping
   res = 0.1 #99 # grid resolution in metres
   # if res==99, the program will automatically calc res from the spatial res of the scans
   mode = 1 # gridding mode (simple nearest neighbour)
   #mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
   #mode = 3 # gridding mode (gaussian weighted nearest neighbour)
   use_uncorrected = 0

   nn = 64 #number of nearest neighbours for gridding (used if mode > 1)
   ##influence = 1 #Radius of influence used in gridding. Cut off distance in meters 
   numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept 

   # for downward-looking echosounder echogram (e1-e2) analysis
   beam = 20.0
   transfreq = 200.0 # frequency (kHz) of downward looking echosounder
   integ = 5
   numclusters = 3 # number of acoustic classes to group observations

   ## read data in SON files into PyHum memory mapped format (.dat)
   PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk) #cog

   ## correct scans and remove water column
   PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

   ## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
   PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot, dissim, correl, contrast, energy, mn)

   win = 10
   PyHum.texture2(humfile, sonpath, win, doplot, numclasses)

   ## grid and map the scans
   PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite, 

   ## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar
   PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)

   res = 1 # grid resolution in metres
   numstdevs = 5

   ## grid and map the texture lengthscale maps
   #PyHum.map_texture(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs)

   #res = 0
   #nn = 5 # noise threshold in dB W
   #noisefloor = 10 # noise threshold in dB W
   #weight = 1 ##based on grazing angle and inverse distance weighting

   ## create mosaic out of all chunks with weighting according to distance from nadir, grazing angle, or both
   #PyHum.mosaic(humfile, sonpath, cs2cs_args, res, nn, noisefloor, weight)

   ##win = 200 #100 # pixel window 
   ## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
   ##PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)

   ## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
   ## implemented using the superpixel approach
   ##PyHum.texture_slic(humfile, sonpath, doplot, numclasses, maxscale, notes)

if __name__ == '__main__':
  my_func()

Above is the code that I have modified from your example. I have changed the following [humfile location, sonpath location, transducer length, temp, cs2cs_args]

When I try to run this, it acts like it is about to start up and not give me 10 billion errors (which I am so grateful for!)

It gives me an error saying the epsg:26915 does not work with this base map. Do you know what I could use instead for Arkansas?

I was in fact able to find the .DAT file and this is how I have my setup

[danhamill]

Not sure why, but Basemap doesn't work very well with UTM. I would advise trying a state plane projection. Maybe EPSG 3433 or 3434

[ryanhefley]

@danhamill Yes, right before I saw your comment, I changed it to 3433 and tried to run it again. Now I'm given another type of error And the following new code:

import PyHum

def my_func():

   # copy files over to somewhere read/writeable

   # general settings   
   humfile = "C:\Users\ryan.hefley\Desktop\R00001.dat"#<-----your path here
   sonpath = "C:\Users\ryan.hefley\Desktop\R00001"#<-----your path here

   doplot = 1 #yes

   # reading specific settings
   cs2cs_args = "epsg:3433" #ARKANSAS STATE PLANE NORTH
   bedpick = 1 # auto bed pick
   c = 1450 # speed of sound fresh water
   t = 0.186 # length of transducer
   draft = 0.3 # draft in metres
   flip_lr = 1 # flip port and starboard
   model = 998 # humminbird model
   calc_bearing = 1 #1=yes
   filt_bearing = 1 #1=yes
   chunk = '1' ##'d100' # distance, 100m
   #chunk = 'p1000' # pings, 1000
   #chunk = 'h10' # heading deviation, 10 deg

   # correction specific settings
   maxW = 1000 # rms output wattage
   dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)
   correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections)
   ph = 7.0 # acidity on the pH scale
   temp = 15.0 # water temperature in degrees Celsius
   salinity = 0.0

   # for shadow removal
   shadowmask = 0 # 0= automatic shadow removal, 1=manual
   win = 31
   dissim=3
   correl=0.2
   contrast=6
   energy=0.15
   mn=4

   # for texture calcs
   shift = 50 ##10 # pixel shift
   density =win/2 # win/2 
   numclasses = 8 #4 # number of discrete classes for contouring and k-means
   maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis)
   notes = 4 # Notes per octave (for wavelet analysis)

   # for mapping
   res = 0.1 #99 # grid resolution in metres
   # if res==99, the program will automatically calc res from the spatial res of the scans
   mode = 1 # gridding mode (simple nearest neighbour)
   #mode = 2 # gridding mode (inverse distance weighted nearest neighbour)
   #mode = 3 # gridding mode (gaussian weighted nearest neighbour)
   use_uncorrected = 0

   nn = 64 #number of nearest neighbours for gridding (used if mode > 1)
   ##influence = 1 #Radius of influence used in gridding. Cut off distance in meters 
   numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept 

   # for downward-looking echosounder echogram (e1-e2) analysis
   beam = 20.0
   transfreq = 200.0 # frequency (kHz) of downward looking echosounder
   integ = 5
   numclusters = 3 # number of acoustic classes to group observations

   ## read data in SON files into PyHum memory mapped format (.dat)
   PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, chunk) #cog

   ## correct scans and remove water column
   PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

   ## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)
   PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot, dissim, correl, contrast, energy, mn)

   win = 10
   PyHum.texture2(humfile, sonpath, win, doplot, numclasses)

   ## grid and map the scans
   PyHum.map(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs, use_uncorrected) #dowrite, 

   ## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar
   PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)

   res = 1 # grid resolution in metres
   numstdevs = 5

   ## grid and map the texture lengthscale maps
   #PyHum.map_texture(humfile, sonpath, cs2cs_args, res, mode, nn, numstdevs)

   #res = 0
   #nn = 5 # noise threshold in dB W
   #noisefloor = 10 # noise threshold in dB W
   #weight = 1 ##based on grazing angle and inverse distance weighting

   ## create mosaic out of all chunks with weighting according to distance from nadir, grazing angle, or both
   #PyHum.mosaic(humfile, sonpath, cs2cs_args, res, nn, noisefloor, weight)

   ##win = 200 #100 # pixel window 
   ## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
   ##PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)

   ## Calculate texture lengthscale maps using the method of Buscombe et al. (2015)
   ## implemented using the superpixel approach
   ##PyHum.texture_slic(humfile, sonpath, doplot, numclasses, maxscale, notes)

if __name__ == '__main__':
  my_func()

I changed the file path from "C:\Users\ryan.hefley\..." back to just single backslash thinking that would fix the above error

[danhamill]

Looks like your files are within a folder named pyhum. your script is pointing to your desktop

[ryanhefley]

@danhamill , Correct. Right now I just have all the files and script in a folder on my desktop. Where should this be saved? Within the Anaconda folder?

[danhamill]

All that matters is you save them in a place where you have read and write permissions. Your path is incorrect in your script. it needs to be something like ...Desktop\pyhum\...

[ryanhefley]

Well.... I feel like a complete idiot. That was completly overlooked on my part. In coding, it's always the simple things that throw errors. And if it isn't one thing, it's another. I got past the first hangup with that simple error, and now I'm given a whole new error. This time possibly in the Pyread module? TypeError: 'NoneType' object is not subscriptable

[danhamill]

Hmm. How large are your son files?

[ryanhefley]

@danhamill All 4 .SON files are 2,709KB (2.709MB) All 4 .IDX files are 13KB

[danhamill]

Ok. I wanted to make sure they are large enough. I think you might be running into a firmware issue. Could you try different model numbers in your script?

    model: int, *optional* [Default=998]
       A 3 or 4 number code indicating the model number
       Examples: 998, 997, 1198, 1199

[ryanhefley]

Upon changing the model number to 1199, the program worked!!!

[danhamill]

Great.

[dbuscombe-usgs]

Thanks @danhamill for this excellent advice! @rhefley - I'm glad you're up and running