version 1.3.7 PyHum.read() pyread.pyx module returning NoneType

[debboutr]

I have installed the most recent version 1.3.7 and installed using

pip install -e 'path/to/repo'

It installs and builds with no errors and I can import it, but when using the PyHum.read() function I am getting the following error:

Traceback (most recent call last): File "", line 1, in File "PyHum_pyhum_read.py", line 396, in read metadat = data.getmetadata() File "pyread.pyx", line 479, in PyHum.pyread.pyread.getmetadata (PyHum\pyread.c:9841) File "pyread.pyx", line 485, in PyHum.pyread.pyread.getmetadata (PyHum\pyread.c:8879) TypeError: 'NoneType' object is not subscriptable

if I step through the _pyhum_read.py script, I am unable to:

import pyread

All of the files that we have collected with our Hummingbird model 1198 have only 3 files associated with them, running PyHum.read() with model = 1198 doesn’t seem to work, but if I switch it to 1199, it does and produces good plots up until I get to the PyHum.map() function which is where the problem exists that I mentioned in the github issue.

I have since tried to install PyHum onto a personal computer running Ubuntu 15.10 . Using the package installed through ‘pip install PyHum’ I was able to get to the map function, with PyHum version 1.3.7 installed I am getting the error which I have attached as a .txt (‘PyHum.read_output’) if I use model = 1199. Below is the error I get when I use model = 1198 in version 1.3.7 running PyHum.read():

Input file is C:\Users\Rdebbout\SideScanData\TEST\R00015.DAT Son files are in C:\Users\Rdebbout\SideScanData\TEST\R00015 cs2cs arguments are epsg:26791 Draft: 0.3 Celerity of sound: 1450.0 m/s Transducer length is 0.108 m Bed picking is auto Chunks based on distance of 100 m Data is from the 1198 series Heading based on course-over-ground Bearing will be calculated from coordinates Checking the epsg code you have chosen for compatibility with Basemap ... ... epsg code compatible WARNING: Because files have to be read in byte by byte, this could take a very long time ... something went wrong with the parallelised version of pyread ... Traceback (most recent call last): File "", line 1, in File "PyHum_pyhum_read.py", line 396, in read metadat = data.getmetadata() File "pyread.pyx", line 479, in PyHum.pyread.pyread.getmetadata (PyHum\pyread.c:9841) File "pyread.pyx", line 485, in PyHum.pyread.pyread.getmetadata (PyHum\pyread.c:8879) TypeError: 'NoneType' object is not subscriptable

The method that I used to install PyHum after cloning the git repository is as such, pip install -e D:\Projects\SideScanSonar\PyHum. If I try to use the ‘python setup.py install’ method I get the output in the txt file attached ‘setup.py_Install_Ubuntu.txt’. I can’t seem to use the setup.py method in either windows or Ubuntu. In windows, I get the error:

D:\Projects\SideScanSonar\PyHum>python setup.py install C:\Users\Rdebbout\AppData\Local\Continuum\Anaconda\lib\distutils\dist.py:267: UserWarning: Unknown distribution option: 'install_req uires' warnings.warn(msg) running install running build running build_py running build_ext skipping 'PyHum\cwt.c' Cython extension (up-to-date) building 'PyHum.cwt' extension error: Unable to find vcvarsall.bat

            I have installed Microsoft Visual C++ Compiler for Python 2.7 and have added the directory to the vcvarsall.bat file to my environment variables, I have also tried to add a VS110COMNTOOLS variable with the same directory listed, no luck. I don’t expect you to know how to fix this, I just thought it might be worth mentioning because it seems to be a common problem with compiling packages in the Windows environment, one which I can’t figure out. Looking at the PKG-INFO on github it seems like you are using a Linux OS, so I thought that doing it on Ubuntu might work out better, but didn’t.

PyHum.read_output.txt setup.py_Install_Ubuntu.txt

Thanks for all of your help!

[dbuscombe-usgs]

On the install errors, try installing directly from git:

pip uninstall PyHum
pip install git+https://github.com/dbuscombe-usgs/PyHum.git

On the 1198 / 1199 issues, it is probably a data format issue. Unfortunately, I don't know what happens to units when firmware updates take place, but it is possible the data format in the SON files differ. Please send me an example data set (DAT, SON and IDX files, if you have them) and I will investigate

[debboutr]

Here is a link to the files that produced this Error:

https://drive.google.com/open?id=0B_WKyKlsG-QjeG5OZ29GQ2pTREE

Traceback (most recent call last): File "processData2.py", line 87, in PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, chunk=chunk) File "/home/rick/anaconda2/lib/python2.7/site-packages/PyHum/_pyhum_read.py", line 654, in read x, bed = humutils.auto_bedpick(ft, dep_m, chunkmode, port_fp, c) File "/home/rick/anaconda2/lib/python2.7/site-packages/PyHum/utils.py", line 106, in auto_bedpick x = np.min(np.vstack((np.squeeze(bed),np.squeeze(autobed))), axis=0)
File "/home/rick/anaconda2/lib/python2.7/site-packages/numpy/core/shape_base.py", line 230, in vstack return _nx.concatenate([atleast_2d(_m) for _m in tup], 0) ValueError: all the input array dimensions except for the concatenation axis must match exactly

[debboutr]

Maybe I need to start back at the beginning here. Should it matter if I am running the commands at the command line vs. inside a console in spyder? Is the best way to run PyHum with a script that takes the sonfile and DAT file locations as arguments? The RAM seems to get flooded when I call the _correct() function inside of a console in spyder, would this be prevented if I use a script at the command line?

[danhamill]

Are you using windows or ubuntu?

[debboutr]

I'm trying with both, Windows7 and Ubuntu 15.10, mostly Windows7 I have Ubuntu on my home machine ha I thought might solve some problems

[danhamill]

In windows environments, I have run into similar problems and I only use the command line to call my python scripts. Are you using run or execfile() command from the console to run your script? Or are you copy and pasting individual modules into the console?

I would reccomend using the format in the test.py script. For example, if you only wanted to run the .read module:

import PyHum

def myfunction()
   # general settings   
   humfile = #your path 
   sonpath = #your path
   doplot = 1 #yes

   # reading specific settings
   cs2cs_args = "epsg:26949" #arizona central state plane
   bedpick = 1 # auto bed pick
   c = 1450 # speed of sound fresh water
   t = 0.108 # length of transducer
   draft = 0.3 # draft in metres
   flip_lr = 1 # flip port and starboard
   model = 998 # humminbird model
   cog = 1 # GPS course-over-ground used for heading
   calc_bearing = 0 #no
   filt_bearing = 0 #no
   chunk = 'd100' # distance, 100m
   ## read data in SON files into PyHum memory mapped format (.dat)
   PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, cog, chunk)

if __name__ == '__main__':
  myfunction()

If you don't use the if __name__ == '__main__': wrapper, the script will recursively spawn subprocesses go into an infinite loop and use 100% of your computational resources. This is windows problem and originates because it does not support forking of subprocesses. If you have been using the wrapper, it could just be it has entered the parallel processing phase of the .correct module and it should finish eventually.

[debboutr]

I was just using 'python process_mysidescandata.py -i DATpath -s SONpath' to run this stuff, but I just tried with the above code and adjusted model, flip_lr, and cs2cs_args. I ran and here is the output which I can't figure.

ProjError.txt

[dbuscombe-usgs]

All your errors probably have the same root cause: the data type you have is not currently supported by the program. Please be patient - I have the files you shared, and I'm getting around to working on your data. Thanks

[dbuscombe-usgs]

I just got round to looking at this problem with the data provided, when I realised you haven't supplied the .DAT file. Please provide, thanks

[debboutr]

https://drive.google.com/folderview?id=0B_WKyKlsG-QjMW8wR25NTmJyb2s&usp=sharing

this should have it I was using R00005, but if you can get any of these files to map I'd like to know. Thanks

[dbuscombe-usgs]

Got it, thanks. I'll dig into it and see what I find

[dbuscombe-usgs]

I just wanted to update you. I have been working with your R0004 and R0005 data sets. The model is in fact 1199, which I think may explain some of the weird bugs you were getting in the read module. But your heading readings are strange and produce strange outputs in the final map of sidescan intensities. Was there something different about the heading sensor or GPS unit you used?

[Mark-epa]

Dr. Buscombe, We here in Duluth-EPA did the side-scan surveys. Rick is in our Corvallis office and has experience with Python/programming. Our Humminbird side-scan model an 1198c, with v7.09 being the most recent software update. We wired in a Garmin GPSMAP model 78 to record our track line (lat/long) every 3 seconds. This setup was based on Kaeser and Litts (2010;2012). I have attached the instructions for how our side-scan system is wired (from Kaeser and Litts). I hope this answers some questions. As a side note prior to using PyHum we processed the our recordings using Sonar TRX software and the output appears complete. I have also attached one of these output files. I believe it is R0012 and Dat 0012. Rick can confirm that. Cheers. It appears I cannot attach anything I will send them to Rick to send to you.

[dbuscombe-usgs]

Mark, That's very helpful info. I could process the scans setting model=1999. What that does in the program is dictate how the data from SON files are read in. Specifically, it means there are 68 rather than 67 header bytes per data packet. It also changes some aspects of the byte-ordering of data written to each header packet, which makes it difficult to parse out heading and location info. It is probable that it is the firmware, not the model, which dictates this. In that regard, analyzing your data was very enlightening. I would need to modify the program to automatically 'catch' this, which apparently in what SonarTRX is doing. But, the long-and-short of it is, model=1999 works for you.

The other issue of heading is probably related to you using an external GPS. I don't know what convention that uses, but it possible it is different from a course-over-ground north-east-up convention expected by PyHum. Therefore, using the flag cog=0 in the read module might be the solution. I will test this tomorrow when I'm back in the office

Dan

[Mark-epa]

Dan,

I looked at our settings for our GPS unit (Garmin GPSMap 78) they were set for the following: Map datum: WGS 84 Heading: true north Position format: hddd.ddddd

Mark S. Pearson Aquatic Biologist USEPA/ORD/NHEERL/MidContinent Ecology Divsion Duluth, MN Office: 218-529-5205 Fax: 218-529-5003

From: Daniel Buscombe [mailto:notifications@github.com] Sent: Wednesday, February 10, 2016 7:52 PM To: dbuscombe-usgs/PyHum PyHum@noreply.github.com Cc: Pearson, Mark Pearson.Mark@epa.gov Subject: Re: [PyHum] version 1.3.7 PyHum.read() pyread.pyx module returning NoneType (#12)

Mark, That's very helpful info. I could process the scans setting model=1999. What that does in the program is dictate how the data from SON files are read in. Specifically, it means there are 68 rather than 67 header bytes per data packet. It also changes some aspects of the byte-ordering of data written to each header packet, which makes it difficult to parse out heading and location info. It is probable that it is the firmware, not the model, which dictates this. In that regard, analyzing your data was very enlightening. I would need to modify the program to automatically 'catch' this, which apparently in what SonarTRX is doing. But, the long-and-short of it is, model=1999 works for you.

The other issue of heading is probably related to you using an external GPS. I don't know what convention that uses, but it possible it is different from a course-over-ground north-east-up convention expected by PyHum. Therefore, using the flag cog=0 in the read module might be the solution. I will test this tomorrow when I'm back in the office

Dan

— Reply to this email directly or view it on GitHubhttps://github.com/dbuscombe-usgs/PyHum/issues/12#issuecomment-182670714.

[dbuscombe-usgs]

Ok, from what I've discovered there's essentially nothing wrong with PyHum. I have processed R0004 and R0005 from the example data sets made available to me. The following settings work well (note that, for reasons described before, model=1999 is correct):

   # reading specific settings
   cs2cs_args = "epsg:26791"
   bedpick = 1 # auto bed pick
   c = 1450 # speed of sound fresh water
   t = 0.108 # length of transducer
   draft = 0.3 # draft in metres
   flip_lr = 1 # flip port and starboard
   model = 1199 # humminbird model
   cog = 1 # GPS course-over-ground used for heading
   calc_bearing = 1 #0=no, 1=yes
   filt_bearing = 0 #0=no, 1=yes
   chunk = 'd1000' # distance, 1000m

I have set the 'calc_bearing' flag to 1, because recalculating headings works better in your case. I have zipped up all my processing files which are available here:

ftp://ftpext.usgs.gov/pub/wr/az/flagstaff/dbuscombe/Debbout_R00004.zip

and here:

ftp://ftpext.usgs.gov/pub/wr/az/flagstaff/dbuscombe/Debbout_R00005.zip

Before you download, note these are really big files.

You should notice that the scans sometimes didn't georectify well. I believe this is the result of using a GPS input only every 3 seconds. This is rather slow (much slower than a 'native' Humminbird GPS receiver which has worked quite well for me) and may not capture changes in heading. If, as you say, the scans map correctly using SonarTRX, then I wish I knew the magic trick being used there. I will investigate further, seeing if I can use this data as a case study for how to fix bad georectifications. If I make any progress, I will let you know. I used 'd1000' for chunk argument (1 km long chunks) but maybe smaller, say 200m chunk lengths, would give you more usable data

Also, there are gaps in the sonar coverage. That is because you tended to conduct your surveys rather too fast (> 2 m/s for most of the survey in R00005, for example). Given the pulse length and coverage, in your case <1.5 m/s would have ensured greater coverage. Of course, I realise that may not have been possible. If you don't see these gaps using sonarTRX, I suggest that that program is merely 'interpolating' across the gaps. There are ways to do that in PyHum too, or an external program

Anyway, sorry for the delay. I hope that helps.

Dan

[dbuscombe-usgs]

Can I close this issue?

[Mark-epa]

Yes. We just downloaded the data today and are looking at the data. Rick may have questions down the road. It does look like our external GPS was the issue and I can certainly use the Humminbird GPS in future surveys. Thank you for the effort !

Mark

Mark S. Pearson Aquatic Biologist USEPA/ORD/NHEERL/MidContinent Ecology Divsion Duluth, MN Office: 218-529-5205 Fax: 218-529-5003

From: Daniel Buscombe [mailto:notifications@github.com] Sent: Thursday, February 18, 2016 5:14 PM To: dbuscombe-usgs/PyHum PyHum@noreply.github.com Cc: Pearson, Mark Pearson.Mark@epa.gov Subject: Re: [PyHum] version 1.3.7 PyHum.read() pyread.pyx module returning NoneType (#12)

Can I close this issue?

— Reply to this email directly or view it on GitHubhttps://github.com/dbuscombe-usgs/PyHum/issues/12#issuecomment-185975220.