search

BenthicSubstrateMapping / PyHum

Python code to read, display, export and analyse Humminbird sidescan sonar data
Other
68 stars 28 forks source link

joblib error #15

Closed dbuscombe-usgs closed 8 years ago

dbuscombe-usgs commented 8 years ago

The new install has thrown a JobLibError

LZarri commented 8 years ago

A new error has occurred: JoblibIndexError. First, here is the script which results in the JobLibIndexError:

-- coding: utf-8 --

""" """

import PyHum

def dotest():

humfile = 'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00220.DAT' sonpath = 'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00220'

doplot = 1 #yes

reading specific settings

cs2cs_args = "epsg:32100" #Montana bedpick = 1 # auto bed pick c = 1450 # speed of sound fresh water t = 0.108 # length of transducer draft = 0.3 # draft in metres flip_lr = 0 # flip port and starboard model = 998 # humminbird model cog = 1 # GPS course-over-ground used for heading calc_bearing = 1 #no filt_bearing = 0 #no chunk = 65 # distance, 65m

chunk = 'p1000' # pings, 1000

chunk = 'h10' # heading deviation, 10 deg

correction specific settings

maxW = 1000 # rms output wattage dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans) correct_withwater = 0 # don't retain water column in radiometric correction (1 = retains water column for radiomatric corrections) ph = 7.0 # acidity on the pH scale temp = 10.0 # water temperature in degrees Celsius salinity = 0.0

for shadow removal

shadowmask = 0 #automatic shadow removal

for texture calcs

win = 100 # pixel window shift = 10 # pixel shift density = win/2 numclasses = 4 # number of discrete classes for contouring and k-means maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis) notes = 4 # Notes per octave (for wavelet analysis)

for mapping

dogrid = 1 #yes res = 0.2 # grid resolution in metres

if res==99, the program will automatically calc res from the spatial res of the scans

mode = 1 # gridding mode (simple nearest neighbour)

mode = 2 # gridding mode (inverse distance weighted nearest neighbour)

mode = 3 # gridding mode (gaussian weighted nearest neighbour)

dowrite = 0 #disable writing of point cloud data to file

nn = 128 #number of nearest neighbours for gridding (used if mode > 1) influence = 1 #Radius of influence used in gridding. Cut off distance in meters numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept

for downward-looking echosounder echogram (e1-e2) analysis

beam = 20.0 transfreq = 200.0 # frequency (kHz) of downward looking echosounder integ = 5 numclusters = 3 # number of acoustic classes to group observations

read data in SON files into PyHum memory mapped format (.dat)

PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, cog, chunk)

correct scans and remove water column

PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)

remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)

PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)

Calculate texture lengthscale maps using the method of Buscombe et al. (2015)

PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)

grid and map the scans

PyHum.map(humfile, sonpath, cs2cs_args, res, dowrite, mode, nn, influence, numstdevs)

res = 0.5 # grid resolution in metres numstdevs = 5

grid and map the texture lengthscale maps

PyHum.map_texture(humfile, sonpath, cs2cs_args, dogrid, res, mode, nn, influence, numstdevs)

calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar

PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)

if name == 'main': dotest()

Below is the error message:

JoblibIndexError: JoblibIndexError

Multiprocessing exception: ........................................................................... C:\Users\user\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\start_ipython_kernel.py in () 200 # ipythonkernel to not have problems while starting kernels 201 change_edit_magic(ipythonshell) 202 ipythonshell.register_magic_function(varexp) 203 204 # Start the (infinite) kernel event loop. --> 205 ipythonkernel.start() 206 207 208 209

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\ipykernel\kernelapp.py in start(self=) 398

399 if self.poller is not None: 400 self.poller.start() 401 self.kernel.start() 402 try: --> 403 ioloop.IOLoop.instance().start() 404 except KeyboardInterrupt: 405 pass 406 407 launch_new_instance = IPKernelApp.launch_instance

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\zmq\eventloop\ioloop.py in start(self=) 146 PollIOLoop.configure(ZMQIOLoop) 147 return PollIOLoop.instance() 148

149 def start(self): 150 try: --> 151 super(ZMQIOLoop, self).start() self.start = > 152 except ZMQError as e: 153 if e.errno == ETERM: 154 # quietly return on ETERM 155 pass

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\tornado\ioloop.py in start(self=) 878 self._events.update(event_pairs) 879 while self._events: 880 fd, events = self._events.popitem() 881 try: 882 fd_obj, handler_func = self._handlers[fd] --> 883 handler_func(fd_obj, events) handler_func = fd_obj = events = 1 884 except (OSError, IOError) as e: 885 if errno_from_exception(e) == errno.EPIPE: 886 # Happens when the client closes the connection 887 pass

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\tornado\stack_context.py in null_wrapper(args=(, 1), kwargs={}) 270 # Fast path when there are no active contexts. 271 def null_wrapper(args, kwargs): 272 try: 273 current_state = _state.contexts 274 _state.contexts = cap_contexts[0] --> 275 return fn(args, *kwargs) args = (, 1) kwargs = {} 276 finally: 277 _state.contexts = current_state 278 null_wrapper._wrapped = True 279 return null_wrapper

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\zmq\eventloop\zmqstream.py in _handle_events(self=, fd=, events=1) 428 # dispatch events: 429 if events & IOLoop.ERROR: 430 gen_log.error("got POLLERR event on ZMQStream, which doesn't make sense") 431 return 432 if events & IOLoop.READ: --> 433 self._handle_recv() self._handle_recv = > 434 if not self.socket: 435 return 436 if events & IOLoop.WRITE: 437 self._handle_send()

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\zmq\eventloop\zmqstream.py in _handle_recv(self=) 460 gen_log.error("RECV Error: %s"%zmq.strerror(e.errno)) 461 else: 462 if self._recv_callback: 463 callback = self._recv_callback 464 # self._recv_callback = None --> 465 self._run_callback(callback, msg) self._run_callback = > callback = msg = [, , , , , , ] 466

467 # self.update_state() 468

469

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\zmq\eventloop\zmqstream.py in _run_callback(self=, callback=, args=([, , , , , , ],), kwargs={}) 402 close our socket.""" 403 try: 404 # Use a NullContext to ensure that all StackContexts are run 405 # inside our blanket exception handler rather than outside. 406 with stack_context.NullContext(): --> 407 callback(args, kwargs) callback = args = ([, , , , , , ],) kwargs = {} 408 except: 409 gen_log.error("Uncaught exception, closing connection.", 410 exc_info=True) 411 # Close the socket on an uncaught exception from a user callback

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\tornado\stack_context.py in null_wrapper(args=([, , , , , , ],), kwargs={}) 270 # Fast path when there are no active contexts. 271 def null_wrapper(args, kwargs): 272 try: 273 current_state = _state.contexts 274 _state.contexts = cap_contexts[0] --> 275 return fn(args, *kwargs) args = ([, , , , , , ],) kwargs = {} 276 finally: 277 _state.contexts = current_state 278 null_wrapper._wrapped = True 279 return null_wrapper

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\ipykernel\kernelbase.py in dispatcher(msg=[, , , , , , ]) 255 if self.control_stream: 256 self.control_stream.on_recv(self.dispatch_control, copy=False) 257 258 def make_dispatcher(stream): 259 def dispatcher(msg): --> 260 return self.dispatch_shell(stream, msg) msg = [, , , , , , ] 261 return dispatcher 262 263 for s in self.shell_streams: 264 s.on_recv(make_dispatcher(s), copy=False)

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\ipykernel\kernelbase.py in dispatch_shell(self=, stream=, msg={'buffers': [], 'content': {u'allow_stdin': True, u'code': u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')", u'silent': False, u'stop_on_error': True, u'store_history': True, u'user_expressions': {}}, 'header': {u'date': datetime.datetime(2016, 2, 1, 16, 32, 12, 280000), u'msg_id': u'49151e52-4a23-43db-8d00-0bbd27dc177b', u'msg_type': u'execute_request', u'session': u'a22a4acf-7b72-4e4c-b56d-e29754497eef', u'username': u'username', u'version': u'5.0'}, 'metadata': {}, 'msg_id': u'49151e52-4a23-43db-8d00-0bbd27dc177b', 'msg_type': u'execute_request', 'parent_header': {}}) 207 self.log.error("UNKNOWN MESSAGE TYPE: %r", msg_type) 208 else: 209 self.log.debug("%s: %s", msg_type, msg) 210 self.pre_handler_hook() 211 try: --> 212 handler(stream, idents, msg) handler = > stream = idents = ['a22a4acf-7b72-4e4c-b56d-e29754497eef'] msg = {'parent_header': {}, 'msg_type': u'execute_request', 'msg_id': u'49151e52-4a23-43db-8d00-0bbd27dc177b', 'content': {u'code': u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')", u'silent': False, u'stop_on_error': True, u'store_history': True, u'user_expressions': {}, u'allow_stdin': True}, 'header': {u'username': u'username', u'version': u'5.0', u'msg_type': u'execute_request', u'msg_id': u'49151e52-4a23-43db-8d00-0bbd27dc177b', u'session': u'a22a4acf-7b72-4e4c-b56d-e29754497eef', u'date': datetime.datetime(2016, 2, 1, 16, 32, 12, 280000)}, 'buffers': [], 'metadata': {}} 213 except Exception: 214 self.log.error("Exception in message handler:", exc_info=True) 215 finally: 216 self.post_handler_hook()

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\ipykernel\kernelbase.py in execute_request(self=, stream=, ident=['a22a4acf-7b72-4e4c-b56d-e29754497eef'], parent={'buffers': [], 'content': {u'allow_stdin': True, u'code': u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')", u'silent': False, u'stop_on_error': True, u'store_history': True, u'user_expressions': {}}, 'header': {u'date': datetime.datetime(2016, 2, 1, 16, 32, 12, 280000), u'msg_id': u'49151e52-4a23-43db-8d00-0bbd27dc177b', u'msg_type': u'execute_request', u'session': u'a22a4acf-7b72-4e4c-b56d-e29754497eef', u'username': u'username', u'version': u'5.0'}, 'metadata': {}, 'msg_id': u'49151e52-4a23-43db-8d00-0bbd27dc177b', 'msg_type': u'execute_request', 'parent_header': {}}) 365 if not silent: 366 self.execution_count += 1 367 self._publish_execute_input(code, parent, self.execution_count) 368 369 reply_content = self.do_execute(code, silent, store_history, --> 370 user_expressions, allow_stdin) user_expressions = {} allow_stdin = True 371 372 # Flush output before sending the reply. 373 sys.stdout.flush() 374 sys.stderr.flush()

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\ipykernel\ipkernel.py in do_execute(self=, code=u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')", silent=False, store_history=True, user_expressions={}, allow_stdin=True) 170 171 reply_content = {} 172 # FIXME: the shell calls the exception handler itself. 173 shell._reply_content = None 174 try: --> 175 shell.run_cell(code, store_history=store_history, silent=silent) shell.run_cell = > code = u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')" store_history = True silent = False 176 except: 177 status = u'error' 178 # FIXME: this code right now isn't being used yet by default, 179 # because the run_cell() call above directly fires off exception

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py in run_cell(self=, raw_cell=u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')", store_history=True, silent=False, shell_futures=True) 2897 self.displayhook.exec_result = result 2898 2899 # Execute the user code 2900 interactivity = "none" if silent else self.ast_node_interactivity 2901 self.run_ast_nodes(code_ast.body, cell_name, -> 2902 interactivity=interactivity, compiler=compiler, result=result) interactivity = 'last_expr' compiler = 2903 2904 # Reset this so later displayed values do not modify the 2905 # ExecutionResult 2906 self.displayhook.exec_result = None

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py in run_ast_nodes(self=, nodelist=[<_ast.Expr object>], cell_name='', interactivity='last', compiler=, result=) 3007 return True 3008 3009 for i, node in enumerate(to_run_interactive): 3010 mod = ast.Interactive([node]) 3011 code = compiler(mod, cell_name, "single") -> 3012 if self.run_code(code, result): self.run_code = > code = at 0000000018D33F30, file "", line 1> result = 3013 return True 3014 3015 # Flush softspace 3016 if softspace(sys.stdout, 0):

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py in run_code(self=, code_obj= at 0000000018D33F30, file "", line 1>, result=) 3061 outflag = 1 # happens in more places, so it's easier as default 3062 try: 3063 try: 3064 self.hooks.pre_run_code_hook() 3065 #rprint('Running code', repr(code_obj)) # dbg -> 3066 exec(code_obj, self.user_global_ns, self.user_ns) code_obj = at 0000000018D33F30, file "", line 1> self.user_globalns = {'dh': [u'C:\Users\user\CodingWorkingFiles\Tutorial'], '': '', 'builtin': , 'quit': , 'i1': u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')", 'package_': None, 'exit': , 'getipython': >, 'i': u'', '**doc': '\n', 'builtins': , 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', 'ih': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'PyHum': <module 'PyHum' from 'C:\Users\user\Anaconda2\lib\site-packages\PyHum_init.pyc'>, 'name': 'main', '': '', '': '', '_sh': , 'dotest': , '_iii': u'', '_ii': u'', 'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], '_oh': {}, 'Out': {}} self.user_ns = {'_dh': [u'C:\Users\user\CodingWorkingFiles\Tutorial'], '': '', 'builtin': , 'quit': , 'i1': u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')", 'package_': None, 'exit': , 'get_ipython': >, 'i': u'', '_doc': '\n', 'builtins': , 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', 'ih': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'PyHum': <module 'PyHum' from 'C:\Users\user\Anaconda2\lib\site-packages\PyHum_init.pyc'>, 'name': 'main', '**': '', '': '', '_sh': , 'dotest': , '_iii': u'', '_ii': u'', 'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], '_oh': {}, 'Out': {}} 3067 finally: 3068 # Reset our crash handler in place 3069 sys.excepthook = old_excepthook 3070 except SystemExit as e:

........................................................................... C:\Users\user\CodingWorkingFiles\Tutorial in () ----> 1 2 3 4 5 6 runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial') 7 8 9 10

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py in runfile(filename=u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', args=None, wdir=u'C:/Users/user/CodingWorkingFiles/Tutorial', namespace={'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'Out': {}, 'PyHum': , '': '', '': '', '': '', 'builtin': , 'builtins': , 'doc': '\n', 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', ...}) 694 except (UnicodeError, TypeError, AttributeError): 695 # UnicodeError, TypeError --> eventually raised in Python 2 696 # AttributeError --> systematically raised in Python 3 697 pass 698 os.chdir(wdir) --> 699 execfile(filename, namespace) filename = u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py' namespace = {'dh': [u'C:\Users\user\CodingWorkingFiles\Tutorial'], '_': '', 'builtin': , 'quit': , 'i1': u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')", 'package_': None, 'exit': , 'get_ipython': >, 'i': u'', 'doc_': '\n', 'builtins': , 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', 'ih': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'PyHum': <module 'PyHum' from 'C:\Users\user\Anaconda2\lib\site-packages\PyHum_init.pyc'>, 'name': 'main', '': '', '': '', '_sh': , 'dotest': , '_iii': u'', '_ii': u'', 'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], '_oh': {}, 'Out': {}} 700 sys.argv = [''] 701 namespace.pop('__file') 702 703 builtins.runfile = runfile

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py in execfile(fname=u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', glob={'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'Out': {}, 'PyHum': , '': '', '_': '', '': '', 'builtin': , 'builtins': , 'doc': '\n', 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', ...}, loc={'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tuto...dir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'Out': {}, 'PyHum': , '': '', '': '', '': '', '**builtin_': , 'builtins': , 'doc': '\n', 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', ...}) 69 # ascii. Let's do the conversion before calling compile 70 if isinstance(fname, unicode): 71 filename = encode(fname) 72 else: 73 filename = fname ---> 74 exec(compile(scripttext, filename, 'exec'), glob, loc) scripttext = '# -- coding: utf-8 --\n"""\n"""\n\nimport PyHum\n\ndef dotest():\n\n humfile = \'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00221.DAT\'\n sonpath = \'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00221\'\n\n doplot = 1 #yes\n\n # reading specific settings\n cs2cs_args = "epsg:32100" #Montana\n bedpick = 1 # auto bed pick\n c = 1450 # speed of sound fresh water\n t = 0.108 # length of transducer\n draft = 0.3 # draft in metres\n flip_lr = 0 # flip port and starboard\n model = 998 # humminbird model\n cog = 1 # GPS course-over-ground used for heading\n calc_bearing = 1 #no\n filt_bearing = 0 #no\n chunk = 65 # distance, 65m\n #chunk = \'p1000\' # pings, 1000\n #chunk = \'h10\' # heading deviation, 10 deg\n \n # correction specific settings\n maxW = 1000 # rms output wattage\n dofilt = 0 # 1 = apply a phase preserving filter (WARNING!! takes a very long time for large scans)\n correct_withwater = 0 # don\'t retain water column in radiometric correction (1 = retains water column for radiomatric corrections)\n ph = 7.0 # acidity on the pH scale\n temp = 10.0 # water temperature in degrees Celsius\n salinity = 0.0\n\n # for shadow removal\n shadowmask = 0 #automatic shadow removal\n\n # for texture calcs\n win = 100 # pixel window\n shift = 10 # pixel shift\n density = win/2 \n numclasses = 4 # number of discrete classes for contouring and k-means\n maxscale = 20 # Max scale as inverse fraction of data length (for wavelet analysis)\n notes = 4 # Notes per octave (for wavelet analysis)\n\n # for mapping\n dogrid = 1 #yes\n res = 0.2 # grid resolution in metres\n # if res==99, the program will automatically calc res from the spatial res of the scans\n mode = 1 # gridding mode (simple nearest neighbour)\n #mode = 2 # gridding mode (inverse distance weighted nearest neighbour)\n #mode = 3 # gridding mode (gaussian weighted nearest neighbour)\n dowrite = 0 #disable writing of point cloud data to file\n\n nn = 128 #number of nearest neighbours for gridding (used if mode > 1)\n influence = 1 #Radius of influence used in gridding. Cut off distance in meters \n numstdevs = 5 #Threshold number of standard deviations in sidescan intensity per grid cell up to which to accept \n\n # for downward-looking echosounder echogram (e1-e2) analysis\n beam = 20.0\n transfreq = 200.0 # frequency (kHz) of downward looking echosounder\n integ = 5\n numclusters = 3 # number of acoustic classes to group observations\n\n ## read data in SON files into PyHum memory mapped format (.dat)\n PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, cog, chunk)\n\n ## correct scans and remove water column\n PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity)\n\n ## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline)\n PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot)\n\n # Calculate texture lengthscale maps using the method of Buscombe et al. (2015)\n PyHum.texture(humfile, sonpath, win, shift, doplot, density, numclasses, maxscale, notes)\n\n # grid and map the scans\n PyHum.map(humfile, sonpath, cs2cs_args, res, dowrite, mode, nn, influence, numstdevs)\n\n res = 0.5 # grid resolution in metres\n numstdevs = 5\n \n # grid and map the texture lengthscale maps\n PyHum.map_texture(humfile, sonpath, cs2cs_args, dogrid, res, mode, nn, influence, numstdevs)\n\n ## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar\n #PyHum.e1e2(humfile, sonpath, cs2csargs, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot)\n\n\nif name == \'main\':\n dotest()\n\n\n\n' filename = 'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py' glob = {'dh': [u'C:\Users\user\CodingWorkingFiles\Tutorial'], '': '', 'builtin': , 'quit': , 'i1': u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')", 'package_': None, 'exit': , 'get_ipython': >, 'i': u'', 'doc_': '\n', 'builtins': , 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', 'ih': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'PyHum': <module 'PyHum' from 'C:\Users\user\Anaconda2\lib\site-packages\PyHuminit_.pyc'>, 'name': 'main', '': '', '': '', '_sh': , 'dotest': , '_iii': u'', '_ii': u'', 'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], '_oh': {}, 'Out': {}} loc = {'_dh': [u'C:\Users\user\CodingWorkingFiles\Tutorial'], '': '', 'builtin': , 'quit': , 'i1': u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')", 'package_': None, 'exit': , 'get_ipython': >, 'i': u'', '_doc': '\n', 'builtins': , 'file': u'C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', 'ih': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], 'PyHum': <module 'PyHum' from 'C:\Users\user\Anaconda2\lib\site-packages\PyHuminit.pyc'>, 'name': 'main', '*': '', '_': '', '_sh': , 'dotest': , '_iii': u'', '_ii': u'', 'In': ['', u"runfile('C:/Users/user/CodingWorkingFiles/Tutorial/BuscombeFix00221.py', wdir='C:/Users/user/CodingWorkingFiles/Tutorial')"], '_oh': {}, 'Out': {}} 75 else: 76 def execfile(fname, where): 77 if isinstance(fname, unicode): 78 filename = fname.encode(sys.getfilesystemencoding())

........................................................................... C:\Users\user\CodingWorkingFiles\Tutorial\BuscombeFix00221.py in () 88 ## calculate and map the e1 and e2 acoustic coefficients from the downward-looking sonar 89 #PyHum.e1e2(humfile, sonpath, cs2cs_args, ph, temp, salinity, beam, transfreq, integ, numclusters, doplot) 90 91 92 if name == 'main': ---> 93 dotest() 94 95 96 97

........................................................................... C:\Users\user\CodingWorkingFiles\Tutorial\BuscombeFix00221.py in dotest() 66 67 ## read data in SON files into PyHum memory mapped format (.dat) 68 PyHum.read(humfile, sonpath, cs2cs_args, c, draft, doplot, t, bedpick, flip_lr, model, calc_bearing, filt_bearing, cog, chunk) 69 70 ## correct scans and remove water column ---> 71 PyHum.correct(humfile, sonpath, maxW, doplot, dofilt, correct_withwater, ph, temp, salinity) humfile = 'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00221.DAT' sonpath = 'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00221' maxW = 1000 doplot = 1 dofilt = 0 correct_withwater = 0 ph = 7.0 temp = 10.0 salinity = 0.0 72 73 ## remove acoustic shadows (caused by distal acoustic attenuation or sound hitting shallows or shoreline) 74 PyHum.rmshadows(humfile, sonpath, win, shadowmask, doplot) 75

........................................................................... C:\Users\user\Anaconda2\lib\site-packages\PyHum_pyhum_correct.py in correct(humfile=r'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00221.DAT', sonpath=r'E:\Sturgeon\2015Substrate\Humminbird\BeenUploaded\R00221\', maxW=array(1000.0), doplot=1, dofilt=0, correct_withwater=0, ph=array(7.0), temp=array(10.0), salinity=0.0, dconcfile=None) 514 del Zt 515 516 #we are only going to access the portion of memory required 517 hi_fp = io.get_mmap_data(sonpath, base, '_data_dwnhi_l.dat', 'float32', shape_hi) 518 --> 519 Zt = correct_scans2(hi_fp, TL_fp) Zt = undefined hi_fp = memmap([[[ 28.47058868, 28.58823586, 28.58823586, ..., 26.58823586, 27.52941132, 26.47058868], [ 28.70588303, 28.58823586, 28.47058868, ..., 26.58823586, 27.64705849, 26.11764717], [ 28.58823586, 28.47058868, 28.47058868, ..., 26.8235302 , 25.88235283, 26.70588303], ..., [ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. , 0. , 0. , ..., 0. , 0. , 0. ]],

[[ 28.35294151, 26.35294151, 28.11764717, ..., 27.52941132, 27.05882263, 26.8235302 ], [ 28.35294151, 27.41176414, 28.23529434, ..., 27.1764698 , 27.76470566, 26.58823586], [ 28.11764717, 28.11764717, 27.76470566, ..., 27.1764698 , 28.23529434, 24.8235302 ], ..., [ 0. , 0. , 0. , ..., 0. ,

  1. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
  2. , 0. ], [ 0. , 0. , 0. , ..., 0. ,

  3. , 0. ]]], dtype=float32) TL_fp = memmap([[[ 0.02716745, 0.02690375, 0.0266379 , ..., 0.03162833, 0.0314817 , 0.03130488], [ 0.02969706, 0.029452 , 0.02920508, ..., 0.03414679, 0.03401045, 0.0338461 ], [ 0.03206417, 0.03183534, 0.03160489, ..., 0.03650322, 0.03637585, 0.03622236], ..., [ 0. , 0. , 0. , ..., 0. ,

    1. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    2. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    3. , 0. ]],

    [[ 0.0310825 , 0.03082864, 0.03057281, ..., 0.02253968, 0.02253968, 0.02253968], [ 0.03363949, 0.03340374, 0.03316629, ..., 0.02504963, 0.02504963, 0.02504963], [ 0.03602947, 0.03580946, 0.03558797, ..., 0.02740089, 0.02740089, 0.02740089], ..., [ 0. , 0. , 0. , ..., 0. ,

    1. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    2. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    3. , 0. ]]], dtype=float32) 520 521 # create memory mapped file for Z 522 shape_hi = io.set_mmap_data(sonpath, base, '_data_dwnhi_la.dat', 'float32', Zt)
      523 del Zt
      ........................................................................... C:\Users\user\Anaconda2\lib\site-packages\PyHum_pyhum_correct.py in correct_scans2(fp=memmap([[[ 28.47058868, 28.58823586, 28.588235... 0. , 0. ]]], dtype=float32), TL=memmap([[[ 0.02716745, 0.02690375, 0.0266379 ,... 0. , 0. ]]], dtype=float32)) 712 # ========================================================= 713 def correct_scans2(fp, TL): 714 if np.ndim(fp)==2: 715 return c_scans2(fp, TL) 716 else: --> 717 return Parallel(n_jobs = cpu_count(), verbose=0)(delayed(c_scans2)(fp[p], TL) for p in xrange(len(fp))) fp = memmap([[[ 28.47058868, 28.58823586, 28.58823586, ..., 26.58823586, 27.52941132, 26.47058868], [ 28.70588303, 28.58823586, 28.47058868, ..., 26.58823586, 27.64705849, 26.11764717], [ 28.58823586, 28.47058868, 28.47058868, ..., 26.8235302 , 25.88235283, 26.70588303], ..., [ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. , 0. , 0. , ..., 0. , 0. , 0. ], [ 0. , 0. , 0. , ..., 0. , 0. , 0. ]],

    [[ 28.35294151, 26.35294151, 28.11764717, ..., 27.52941132, 27.05882263, 26.8235302 ], [ 28.35294151, 27.41176414, 28.23529434, ..., 27.1764698 , 27.76470566, 26.58823586], [ 28.11764717, 28.11764717, 27.76470566, ..., 27.1764698 , 28.23529434, 24.8235302 ], ..., [ 0. , 0. , 0. , ..., 0. ,

  4. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
  5. , 0. ], [ 0. , 0. , 0. , ..., 0. ,

  6. , 0. ]]], dtype=float32) TL = memmap([[[ 0.02716745, 0.02690375, 0.0266379 , ..., 0.03162833, 0.0314817 , 0.03130488], [ 0.02969706, 0.029452 , 0.02920508, ..., 0.03414679, 0.03401045, 0.0338461 ], [ 0.03206417, 0.03183534, 0.03160489, ..., 0.03650322, 0.03637585, 0.03622236], ..., [ 0. , 0. , 0. , ..., 0. ,

    1. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    2. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    3. , 0. ]],

    [[ 0.0310825 , 0.03082864, 0.03057281, ..., 0.02253968, 0.02253968, 0.02253968], [ 0.03363949, 0.03340374, 0.03316629, ..., 0.02504963, 0.02504963, 0.02504963], [ 0.03602947, 0.03580946, 0.03558797, ..., 0.02740089, 0.02740089, 0.02740089], ..., [ 0. , 0. , 0. , ..., 0. ,

    1. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    2. , 0. ], [ 0. , 0. , 0. , ..., 0. ,
    3. , 0. ]]], dtype=float32) 718 719 # ========================================================= 720 def c_scans2(fp, TL): 721 #nodata = fp==0 ........................................................................... C:\Users\user\Anaconda2\lib\site-packages\joblib\parallel.py in call(self=Parallel(n_jobs=2), iterable=>) 514 self.n_dispatched = 0 515 try: 516 for function, args, kwargs in iterable: 517 self.dispatch(function, args, kwargs) 518 --> 519 self.retrieve() self.retrieve = 520 # Make sure that we get a last message telling us we are done 521 elapsed_time = time.time() - self._start_time 522 self._print('Done %3i out of %3i | elapsed: %s finished', 523 (len(self._output),

Sub-process traceback:

IndexError Mon Feb 01 16:34:31 2016 PID: 2180 Python 2.7.11: C:\Users\user\Anaconda2\python.exe ........................................................................... C:\Users\user\Anaconda2\lib\site-packages\PyHum_pyhum_correct.pyc in c_scans2(fp=memmap([[ 28.47058868, 28.58823586, 28.5882358... 0. , 0. ]], dtype=float32), TL=memmap([[[ 0.02716745, 0.02690375, 0.0266379 ,... 0. , 0. ]]], dtype=float32)) 722 try: 723 mg = 10np.log10(np.asarray(fp,'float32')+0.001 + TL) #[:,::2] ) 724 except: 725 mg = 10np.log10(np.asarray(fp,'float32')+0.001 ) 726 --> 727 mg[fp==0] = np.nan 728 mg[mg<0] = np.nan 729 #mg[nodata] = np.nan 730 return mg 731

IndexError: index 921 is out of bounds for axis 0 with size 2

dbuscombe-usgs commented 8 years ago

No idea. Windows issue only. Do you get similar errors with other data files, or just this one? @danhamill - any insight?

dbuscombe-usgs commented 8 years ago

I cannot reproduce this error. Please provide the data and your processing arguments, so I can investigate further. Thanks

danhamill commented 8 years ago

Nope, never seen anything like that before.

I noticed in your test script you are specifying chunk distance of:

chunk = '65'

From the test.py` script it should be something like:

chunk = 'd65'
LZarri commented 8 years ago

This error was caused by running PyHum on Spyder. I used cmd and the problem goes away. @dbuscombe-usgs Thread can be closed