hudon / spike

Brain Simulator Parallelization
http://nengo.ca/
1 stars 1 forks source link

master branch hangs if you run tests over and over #64

Closed RobertElder closed 10 years ago

RobertElder commented 10 years ago

For at least two of the tests on master, if you run the tests over and over in a loop, you will get a deadlock. I have replicated this with test_array and test_radius. One time, I ran test_array for 10 mins and got a deadlock. Then I ran it again for an hour and it did not deadlock. test_radius seems to deadlock more often.

You can use this command to do the same test as me

while true; do /usr/bin/python2 /home/robert/spike/test/nengo_tests/test_radius.py /home/robert/spike/test/../src; date; done;

If the directories in the above command are wrong, just do 'make test' and copy the command it spits out.

Note that the first part of the stack trace makes it look like it is sitting in zeromq error. Perhaps a synchronization problem in our code?

robert@robert-ubuntu:~/spike$ while true; do /usr/bin/python2 /home/robert/spike/test/nengo_tests/test_radius.py /home/robert/spike/test/../src; date; done; starting simulation Sun Sep 8 00:11:58 EDT 2013 starting simulation Sun Sep 8 00:12:10 EDT 2013 starting simulation Sun Sep 8 00:12:22 EDT 2013 starting simulation Sun Sep 8 00:12:34 EDT 2013 starting simulation Sun Sep 8 00:12:47 EDT 2013 starting simulation Sun Sep 8 00:13:00 EDT 2013 starting simulation Sun Sep 8 00:13:13 EDT 2013 starting simulation Sun Sep 8 00:13:25 EDT 2013 starting simulation Sun Sep 8 00:13:37 EDT 2013 starting simulation Sun Sep 8 00:13:50 EDT 2013 starting simulation Sun Sep 8 00:14:02 EDT 2013 starting simulation Sun Sep 8 00:14:14 EDT 2013 starting simulation Sun Sep 8 00:14:27 EDT 2013 starting simulation Sun Sep 8 00:14:39 EDT 2013 starting simulation Sun Sep 8 00:14:52 EDT 2013 starting simulation Sun Sep 8 00:15:04 EDT 2013 starting simulation Sun Sep 8 00:15:16 EDT 2013 starting simulation Sun Sep 8 00:15:29 EDT 2013 starting simulation Sun Sep 8 00:15:41 EDT 2013 starting simulation Sun Sep 8 00:15:53 EDT 2013 starting simulation Sun Sep 8 00:16:06 EDT 2013 starting simulation Sun Sep 8 00:16:18 EDT 2013 starting simulation Sun Sep 8 00:16:31 EDT 2013 starting simulation Sun Sep 8 00:16:43 EDT 2013 starting simulation Sun Sep 8 00:16:55 EDT 2013 starting simulation Sun Sep 8 00:17:08 EDT 2013 starting simulation Sun Sep 8 00:17:20 EDT 2013 starting simulation Sun Sep 8 00:17:33 EDT 2013 starting simulation Sun Sep 8 00:17:45 EDT 2013 starting simulation Sun Sep 8 00:17:57 EDT 2013 starting simulation Sun Sep 8 00:18:10 EDT 2013 starting simulation Sun Sep 8 00:18:22 EDT 2013 starting simulation Sun Sep 8 00:18:35 EDT 2013 starting simulation Sun Sep 8 00:18:47 EDT 2013 starting simulation Sun Sep 8 00:18:59 EDT 2013 starting simulation Sun Sep 8 00:19:11 EDT 2013 starting simulation Sun Sep 8 00:19:24 EDT 2013 starting simulation Sun Sep 8 00:19:36 EDT 2013 starting simulation Sun Sep 8 00:19:48 EDT 2013 starting simulation Sun Sep 8 00:20:00 EDT 2013 starting simulation Sun Sep 8 00:20:12 EDT 2013 starting simulation ^CProcess D: Process B: Traceback (most recent call last): Process C: Traceback (most recent call last): Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap Traceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 56, in Process A: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap Traceback (most recent call last): net.run(timesteps * dt_step) File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "/home/robert/spike/test/../src/nef_theano/network.py", line 504, in run self.run() File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 97, in run self.ticker_conn.recv() # wait for an ACK from main proc before exiting File "socket.pyx", line 616, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5961) if isinstance(p, ensemble.EnsembleProcess): conn.recv() self.run() File "socket.pyx", line 616, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5961) self.run() self.run() File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 97, in run File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 94, in run File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 97, in run self.ticker_conn.recv() # wait for an ACK from main proc before exiting self.tick() self.ticker_conn.recv() # wait for an ACK from main proc before exiting File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 74, in tick File "socket.pyx", line 616, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5961) File "socket.pyx", line 616, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5961) responses = dict(self.poller.poll(1)) File "poll.pyx", line 189, in zmq.core.poll.Poller.poll (zmq/core/poll.c:2189) File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "poll.pyx", line 101, in zmq.core.poll._poll (zmq/core/poll.c:1440) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) KeyboardInterrupt KeyboardInterrupt File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt Sun Sep 8 00:32:07 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 6, in import unittest_tools File "/usr/local/lib/python2.7/dist-packages/theano/tests/unittest_tools.py", line 8, in import theano.tensor as T File "/usr/local/lib/python2.7/dist-packages/theano/tensor/init.py", line 10, in import blas File "/usr/local/lib/python2.7/dist-packages/theano/tensor/blas.py", line 162, in numpy.dtype('float32'): fblas.sgemv, KeyboardInterrupt Sun Sep 8 00:32:08 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 215, in init low=max_rate[0], high=max_rate[1])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 779, in uniform node_rstate = shared(self.get_substream_rstates(nstreams)) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 696, in get_substream_rstates rval[i] = ff_2p72(rval[i-1]) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 86, in ff_2p72 return multMatVect(rstate, A1p72, M1, A2p72, M2) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 45, in multMatVect r[:3] = matVecModM(A, v[:3], m1) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 32, in matVecModM r = numpy.int32((numpy.int64(A[i][j]) * s[j] + x[i]) % m) KeyboardInterrupt Sun Sep 8 00:32:08 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 220, in init [], self.neurons.make_alpha_bias(max_rates, threshold))() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1478, in apply lopt_change = self.process_node(fgraph, node, lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1228, in process_node fgraph.replace_all_validate(repl_pairs, reason=lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/toolbox.py", line 206, in replace_all_validate fgraph.replace(r, new_r, reason=reason) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 430, in replace self.change_input(node, i, new_r, reason=reason) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 400, in change_input self.import_r([new_r]) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 224, in import_r self.import(apply_node) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 330, in import self.execute_callbacks('on_import', node) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 513, in execute_callbacks fn(self, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 1022, in on_import self.set_shape(r, s) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 830, in set_shape for i in range(r.ndim)]) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/basic.py", line 2013, in equals self.signature() == other.signature()) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/basic.py", line 1910, in eq self.no_nan # Ensure has_nan is computed. File "/usr/local/lib/python2.7/dist-packages/theano/tensor/basic.py", line 1966, in _get_no_nan return self._no_nan KeyboardInterrupt Sun Sep 8 00:32:09 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 220, in init [], self.neurons.make_alpha_bias(max_rates, threshold))() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1478, in apply lopt_change = self.process_node(fgraph, node, lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1228, in process_node fgraph.replace_all_validate(repl_pairs, reason=lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/toolbox.py", line 218, in replace_all_validate fgraph.validate() File "/usr/local/lib/python2.7/dist-packages/theano/gof/toolbox.py", line 157, in validate ret = fgraph.execute_callbacks('validate') File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 513, in execute_callbacks fn(self, _args, _kwargs) KeyboardInterrupt Sun Sep 8 00:32:10 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 220, in init [], self.neurons.make_alpha_bias(max_rates, threshold))() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1478, in apply lopt_change = self.process_node(fgraph, node, lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1205, in process_node replacements = lopt.transform(node) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 3898, in constant_folding no_recycling=[]) File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 580, in make_thunk output_storage=node_output_storage) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 913, in make_thunk keep_lock=keep_lock) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 856, in compile keep_lock=keep_lock) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1278, in cthunk_factory module = get_module_cache().module_from_key( File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 70, in get_module_cache return cmodule.get_module_cache(config.compiledir, init_args=init_args) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 1388, in get_module_cache _module_cache = ModuleCache(dirname, init_args) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 584, in init self.refresh() File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 653, in refresh key_data = cPickle.load(open(key_pkl, 'rb')) File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 529, in new def new(cls, _args, _kwargs): KeyboardInterrupt Sun Sep 8 00:32:10 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 6, in import unittest_tools File "/usr/local/lib/python2.7/dist-packages/theano/tests/unittest_tools.py", line 8, in import theano.tensor as T File "/usr/local/lib/python2.7/dist-packages/theano/tensor/init.py", line 10, in import blas File "/usr/local/lib/python2.7/dist-packages/theano/tensor/blas.py", line 152, in import scipy.linalg.blas File "/usr/lib/python2.7/dist-packages/scipy/linalg/init.py", line 15, in from decomp_schur import File "/usr/lib/python2.7/dist-packages/scipy/linalg/decomp_schur.py", line 82, in eps = numpy.finfo(float).eps File "/usr/lib/python2.7/dist-packages/numpy/core/getlimits.py", line 119, in new obj = object.new(cls)._init(dtype) File "/usr/lib/python2.7/dist-packages/numpy/core/getlimits.py", line 149, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib/python2.7/dist-packages/numpy/core/machar.py", line 110, in init self._do_init(float_conv, int_conv, float_to_float, float_to_str, title) File "/usr/lib/python2.7/dist-packages/numpy/core/machar.py", line 255, in _do_init if any(a+a != zero) and any(abs(y) < xmin): File "/usr/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 1576, in any return any(axis, out) KeyboardInterrupt Sun Sep 8 00:32:11 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 6, in import unittest_tools File "/usr/local/lib/python2.7/dist-packages/theano/tests/unittest_tools.py", line 8, in import theano.tensor as T File "/usr/local/lib/python2.7/dist-packages/theano/tensor/init.py", line 10, in import blas File "/usr/local/lib/python2.7/dist-packages/theano/tensor/blas.py", line 152, in import scipy.linalg.blas File "/usr/lib/python2.7/dist-packages/scipy/linalg/init.py", line 15, in from decomp_schur import File "/usr/lib/python2.7/dist-packages/scipy/linalg/decomp_schur.py", line 82, in eps = numpy.finfo(float).eps File "/usr/lib/python2.7/dist-packages/numpy/core/getlimits.py", line 119, in new obj = object.new(cls)._init(dtype) File "/usr/lib/python2.7/dist-packages/numpy/core/getlimits.py", line 149, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib/python2.7/dist-packages/numpy/core/machar.py", line 110, in init self._do_init(float_conv, int_conv, float_to_float, float_to_str, title) File "/usr/lib/python2.7/dist-packages/numpy/core/machar.py", line 253, in _do_init a = y \ one KeyboardInterrupt Sun Sep 8 00:32:11 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 6, in import unittest_tools File "/usr/local/lib/python2.7/dist-packages/theano/tests/unittest_tools.py", line 8, in import theano.tensor as T File "/usr/local/lib/python2.7/dist-packages/theano/tensor/init.py", line 52, in import nnet # used for softmax, sigmoid, etc. File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/init.py", line 2, in from conv import conv2d, ConvOp File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/conv.py", line 28, in from scipy.signal.signaltools import _valfrommode, _bvalfromboundary File "/usr/lib/python2.7/dist-packages/scipy/signal/init.py", line 14, in from signaltools import KeyboardInterrupt Sun Sep 8 00:32:12 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 226, in init self.encoders = self.make_encoders(encoders=encoders) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 446, in make_encoders return theano.function([], encoders)() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1436, in apply gopt.apply(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/nnet.py", line 1272, in crossentropy_to_crossentropy_with_softmax_with_bias while search_make_one_sub(): File "/usr/local/lib/python2.7/dist-packages/theano/tensor/nnet/nnet.py", line 1258, in search_make_one_sub for node in fgraph.toposort(): File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 561, in toposort order = graph.io_toposort(fg.inputs, fg.outputs, ords) File "/usr/local/lib/python2.7/dist-packages/theano/gof/graph.py", line 811, in io_toposort topo = general_toposort(outputs, deps) File "/usr/local/lib/python2.7/dist-packages/theano/gof/graph.py", line 752, in general_toposort reachable, clients = stack_search(deque(r_out), _deps, 'dfs', True) File "/usr/local/lib/python2.7/dist-packages/theano/gof/graph.py", line 523, in stack_search if id(l) not in rval_set: KeyboardInterrupt Sun Sep 8 00:32:13 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 215, in init low=max_rate[0], high=max_rate[1])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 779, in uniform node_rstate = shared(self.get_substream_rstates(nstreams)) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 696, in get_substream_rstates rval[i] = ff_2p72(rval[i-1]) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 86, in ff_2p72 return multMatVect(rstate, A1p72, M1, A2p72, M2) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 46, in multMatVect r[3:] = matVecModM(B, v[3:], m2) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 31, in matVecModM for j in xrange(len(s)): KeyboardInterrupt Sun Sep 8 00:32:13 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 6, in import unittest_tools File "/usr/local/lib/python2.7/dist-packages/theano/tests/unittest_tools.py", line 8, in import theano.tensor as T File "/usr/local/lib/python2.7/dist-packages/theano/tensor/init.py", line 10, in import blas File "/usr/local/lib/python2.7/dist-packages/theano/tensor/blas.py", line 152, in import scipy.linalg.blas File "/usr/lib/python2.7/dist-packages/scipy/linalg/init.py", line 15, in from decomp_schur import
File "/usr/lib/python2.7/dist-packages/scipy/linalg/decomp_schur.py", line 82, in eps = numpy.finfo(float).eps File "/usr/lib/python2.7/dist-packages/numpy/core/getlimits.py", line 119, in
new obj = object.new(cls)._init(dtype) File "/usr/lib/python2.7/dist-packages/numpy/core/getlimits.py", line 149, in _init 'numpy %s precision floating point number' % precname) File "/usr/lib/python2.7/dist-packages/numpy/core/machar.py", line 110, in init self._do_init(float_conv, int_conv, float_to_float, float_to_str, title) File "/usr/lib/python2.7/dist-packages/numpy/core/machar.py", line 258, in _do_init if any(temp1_beta == y) and any(temp != y): File "/usr/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 1508, in any def any(a,axis=None, out=None): KeyboardInterrupt Sun Sep 8 00:32:14 EDT 2013 ^C^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 220, in init [], self.neurons.make_alpha_bias(max_rates, threshold))() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 4640, in apply new_outputs = self.optimizer(node) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 4606, in local_fuse ret = local_fuse(n) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 4606, in local_fuse ret = local_fuse(n) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 4606, in local_fuse ret = local_fuse(n) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 4606, in local_fuse ret = local_fuse(n) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 4594, in local_fuse n = OP(C).make_node(_inputs) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/elemwise.py", line 529, in make_node [Scalar(dtype=i.type.dtype)() for i in inputs]) File "/usr/local/lib/python2.7/dist-packages/theano/gof/type.py", line 317, in call return utils.add_tag_trace(self.make_variable(name)) File "/usr/local/lib/python2.7/dist-packages/theano/gof/utils.py", line 15, in add_tag_trace thing.tag.trace = traceback.extract_stack(limit=limit)[:-1] File "/usr/lib/python2.7/traceback.py", line 304, in extract_stack linecache.checkcache(filename) File "/usr/lib/python2.7/linecache.py", line 60, in checkcache stat = os.stat(fullname) KeyboardInterrupt Sun Sep 8 00:32:15 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 9, in from . import ensemble File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 8, in from . import ensemble_origin File "/home/robert/spike/test/../src/nef_theano/ensemble_origin.py", line 10, in from . import cache File "/home/robert/spike/test/../src/nef_theano/cache.py", line 45, in 'nefpy_cache_gamma_inv')) File "/usr/lib/python2.7/shelve.py", line 239, in open return DbfilenameShelf(filename, flag, protocol, writeback) File "/usr/lib/python2.7/shelve.py", line 223, in init Shelf.init(self, anydbm.open(filename, flag), protocol, writeback) File "/usr/lib/python2.7/anydbm.py", line 85, in open return mod.open(file, flag, mode) File "/usr/lib/python2.7/dbhash.py", line 18, in open return bsddb.hashopen(file, flag, mode) File "/usr/lib/python2.7/bsddb/init.py", line 357, in hashopen e = _openDBEnv(cachesize) File "/usr/lib/python2.7/bsddb/init.py", line 414, in _openDBEnv e.open('.', db.DB_PRIVATE | db.DB_CREATE | db.DB_THREAD | db.DB_INIT_LOCK | db.DB_INIT_MPOOL) KeyboardInterrupt Sun Sep 8 00:32:15 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 215, in init low=max_rate[0], high=max_rate[1])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 735, in uniform high = as_tensor_variable(high) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/basic.py", line 193, in as_tensor_variable return constant(x, name=name, ndim=ndim) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/basic.py", line 401, in constant dtype=dtype) KeyboardInterrupt Sun Sep 8 00:32:16 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 3, in from main import main, TheanoNoseTester File "/usr/local/lib/python2.7/dist-packages/theano/tests/main.py", line 2, in import nose.plugins.builtin File "/usr/lib/python2.7/dist-packages/nose/init.py", line 1, in from nose.core import collector, main, run, run_exit, runmodule File "/usr/lib/python2.7/dist-packages/nose/core.py", line 11, in from nose.config import Config, all_config_files File "/usr/lib/python2.7/dist-packages/nose/config.py", line 9, in from nose.plugins.manager import NoPlugins File "/usr/lib/python2.7/dist-packages/nose/plugins/init.py", line 185, in from nose.plugins.manager import
File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 405, in import pkg_resources File "/usr/lib/python2.7/dist-packages/pkgresources.py", line 1901, in CBRACKET = re.compile(r"\s]").match File "/usr/lib/python2.7/re.py", line 190, in compile return _compile(pattern, flags) File "/usr/lib/python2.7/re.py", line 240, in _compile p = sre_compile.compile(pattern, flags) File "/usr/lib/python2.7/sre_compile.py", line 504, in compile code = _code(p, flags) File "/usr/lib/python2.7/sre_compile.py", line 489, in _code _compile(code, p.data, flags) File "/usr/lib/python2.7/sre_compile.py", line 82, in _compile _compile(code, av[2], flags) File "/usr/lib/python2.7/sre_compile.py", line 38, in _compile SUCCESS_CODES = _SUCCESS_CODES KeyboardInterrupt Sun Sep 8 00:32:16 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/
init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 3, in from main import main, TheanoNoseTester File "/usr/local/lib/python2.7/dist-packages/theano/tests/main.py", line 2, in import nose.plugins.builtin File "/usr/lib/python2.7/dist-packages/nose/init.py", line 1, in from nose.core import collector, main, run, run_exit, runmodule File "/usr/lib/python2.7/dist-packages/nose/core.py", line 11, in from nose.config import Config, all_config_files File "/usr/lib/python2.7/dist-packages/nose/config.py", line 2, in import optparse File "/usr/lib/python2.7/optparse.py", line 90, in from gettext import gettext File "/usr/lib/python2.7/gettext.py", line 49, in import locale, copy, os, re, struct, sys File "/usr/lib/python2.7/locale.py", line 174, in _percentre = re.compile(r'%(?:((?P.?)))?' File "/usr/lib/python2.7/re.py", line 190, in compile return _compile(pattern, flags) File "/usr/lib/python2.7/re.py", line 240, in _compile p = sre_compile.compile(pattern, flags) File "/usr/lib/python2.7/sre_compile.py", line 504, in compile code = _code(p, flags) File "/usr/lib/python2.7/sre_compile.py", line 489, in _code _compile(code, p.data, flags) File "/usr/lib/python2.7/sre_compile.py", line 57, in _compile _compile_charset(av, flags, code, fixup) File "/usr/lib/python2.7/sre_compile.py", line 183, in _compile_charset for op, av in _optimize_charset(charset, fixup): File "/usr/lib/python2.7/sre_compile.py", line 253, in _optimize_charset data = _mk_bitmap(charmap) File "/usr/lib/python2.7/sre_compile.py", line 270, in _mk_bitmap if m > MAXCODE: KeyboardInterrupt Sun Sep 8 00:32:16 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 3, in from main import main, TheanoNoseTester File "/usr/local/lib/python2.7/dist-packages/theano/tests/main.py", line 2, in import nose.plugins.builtin File "/usr/lib/python2.7/dist-packages/nose/init.py", line 1, in from nose.core import collector, main, run, run_exit, runmodule File "/usr/lib/python2.7/dist-packages/nose/core.py", line 11, in from nose.config import Config, all_config_files File "/usr/lib/python2.7/dist-packages/nose/config.py", line 9, in from nose.plugins.manager import NoPlugins File "/usr/lib/python2.7/dist-packages/nose/plugins/init.py", line 185, in from nose.plugins.manager import * File "/usr/lib/python2.7/dist-packages/nose/plugins/manager.py", line 405, in import pkg_resources File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2728, in working_set.entries=[]; map(working_set.add_entry,sys.path) # match order File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 450, in add_entry for dist in find_distributions(entry, True): File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1745, in find_on_path path_item,entry,metadata,precedence=DEVELOP_DIST File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2119, in from_location py_version=py_version, platform=platform, kw File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2105, in init self._provider = metadata or empty_provider KeyboardInterrupt Sun Sep 8 00:32:17 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 9, in from . import ensemble File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 2, in from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 19, in import multinomial File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/multinomial.py", line 9, in from theano.sandbox.cuda import cuda_available, GpuOp KeyboardInterrupt Sun Sep 8 00:32:17 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 48, in import gof File "/usr/local/lib/python2.7/dist-packages/theano/gof/init.py", line 40, in from cc import \ File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 47, in import graph File "/usr/local/lib/python2.7/dist-packages/theano/gof/graph.py", line 235, in class Variable(Node): File "/usr/local/lib/python2.7/dist-packages/theano/gof/graph.py", line 235, in Variable class Variable(Node): KeyboardInterrupt Sun Sep 8 00:32:17 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 76, in import theano.tests File "/usr/local/lib/python2.7/dist-packages/theano/tests/init.py", line 3, in from main import main, TheanoNoseTester File "/usr/local/lib/python2.7/dist-packages/theano/tests/main.py", line 2, in import nose.plugins.builtin File "/usr/lib/python2.7/dist-packages/nose/plugins/builtin.py", line 26, in plugmod = import(module, globals(), locals(), [cls]) File "/usr/lib/python2.7/dist-packages/nose/plugins/doctests.py", line 71, in import doctest File "/usr/lib/python2.7/doctest.py", line 549, in class DocTestParser: File "/usr/lib/python2.7/doctest.py", line 569, in DocTestParser ''', re.MULTILINE | re.VERBOSE) File "/usr/lib/python2.7/re.py", line 190, in compile return _compile(pattern, flags) File "/usr/lib/python2.7/re.py", line 240, in _compile p = sre_compile.compile(pattern, flags) File "/usr/lib/python2.7/sre_compile.py", line 500, in compile p = sre_parse.parse(p, flags) File "/usr/lib/python2.7/sre_parse.py", line 673, in parse p = _parse_sub(source, pattern, 0) File "/usr/lib/python2.7/sre_parse.py", line 308, in _parse_sub itemsappend(_parse(source, state)) File "/usr/lib/python2.7/sre_parse.py", line 633, in _parse p = _parse_sub(source, state) File "/usr/lib/python2.7/sre_parse.py", line 308, in _parse_sub itemsappend(_parse(source, state)) File "/usr/lib/python2.7/sre_parse.py", line 633, in _parse p = _parse_sub(source, state) File "/usr/lib/python2.7/sre_parse.py", line 308, in _parse_sub itemsappend(_parse(source, state)) File "/usr/lib/python2.7/sre_parse.py", line 397, in _parse this = sourceget() File "/usr/lib/python2.7/sre_parse.py", line 203, in get self.__next() File "/usr/lib/python2.7/sre_parse.py", line 186, in next char = self.string[self.index] KeyboardInterrupt Sun Sep 8 00:32:18 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init__.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 5, in import theano File "/usr/local/lib/python2.7/dist-packages/theano/init.py", line 48, in import gof File "/usr/local/lib/python2.7/dist-packages/theano/gof/init*.py", line 40, in from cc import \ File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 53, in import cmodule File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 107, in class ExtFunction(object): KeyboardInterrupt Sun Sep 8 00:32:18 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 220, in init [], self.neurons.make_alpha_bias(max_rates, threshold))() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, *_kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1478, in apply lopt_change = self.process_node(fgraph, node, lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1205, in process_node replacements = lopt.transform(node) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 3898, in constant_folding no_recycling=[]) File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 580, in make_thunk output_storage=node_output_storage) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 913, in make_thunk keep_lock=keep_lock) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 856, in compile keep_lock=keep_lock) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1278, in cthunk_factory module = get_module_cache().module_from_key( File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 70, in get_module_cache return cmodule.get_module_cache(config.compiledir, init_args=init_args) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 1388, in get_module_cache _module_cache = ModuleCache(dirname, init_args) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 584, in init self.refresh() File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 766, in refresh self.entry_from_key[key] = entry KeyboardInterrupt Sun Sep 8 00:32:19 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 3, in import quantities File "/usr/local/lib/python2.7/dist-packages/quantities/init.py", line 285, in from . import constants File "/usr/local/lib/python2.7/dist-packages/quantities/constants/init*.py", line 15, in from .quantum import File "/usr/local/lib/python2.7/dist-packages/quantities/constants/quantum.py", line 106, in u_symbol='(G_F/(ħ·c)³)' File "/usr/local/lib/python2.7/dist-packages/quantities/unitquantity.py", line 62, in new ret._conv_ref = definition._reference File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 137, in _reference rq = 1unit_registry['dimensionless'] File "/usr/local/lib/python2.7/dist-packages/quantities/registry.py", line 54, in getitem r"\g<1>\g<2>", label.replace('^', '').replace('·', '_')) KeyboardInterrupt Sun Sep 8 00:32:19 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 220, in init [], self.neurons.make_alpha_bias(max_rates, threshold))() File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 221, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 506, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1338, in orig_function on_unused_input=on_unused_input).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1008, in init optimizer_profile = optimizer(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 75, in call return self.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, _kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 161, in apply sub_prof = optimizer.optimize(fgraph) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 69, in optimize return self.apply(fgraph, _args, *_kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1478, in apply lopt_change = self.process_node(fgraph, node, lopt) File "/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py", line 1205, in process_node replacements = lopt.transform(node) File "/usr/local/lib/python2.7/dist-packages/theano/tensor/opt.py", line 3898, in constant_folding no_recycling=[]) File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 580, in make_thunk output_storage=node_output_storage) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 913, in make_thunk keep_lock=keep_lock) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 856, in compile keep_lock=keep_lock) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1278, in cthunk_factory module = get_module_cache().module_from_key( File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 70, in get_module_cache return cmodule.get_module_cache(config.compiledir, init_args=init_args) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 1388, in get_module_cache _module_cache = ModuleCache(dirname, init_args) File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 584, in init self.refresh() File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 653, in refresh key_data = cPickle.load(open(key_pkl, 'rb')) KeyboardInterrupt Sun Sep 8 00:32:20 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 3, in import quantities File "/usr/local/lib/python2.7/dist-packages/quantities/init.py", line 285, in from . import constants File "/usr/local/lib/python2.7/dist-packages/quantities/constants/init.py", line 5, in from .atomicunits import File "/usr/local/lib/python2.7/dist-packages/quantities/constants/atomicunits.py", line 84, in u_symbol='(e·a₀²)' File "/usr/local/lib/python2.7/dist-packages/quantities/unitquantity.py", line 62, in new ret._conv_ref = definition._reference File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 140, in _reference return rq * self.magnitude File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 231, in __array_prepare__ res._dimensionality = p_dict[uf](objs) File "/usr/local/lib/python2.7/dist-packages/quantities/dimensionality.py", line 236, in _d_multiply return q2.dimensionality AttributeError: 'numpy.ndarray' object has no attribute 'dimensionality' Sun Sep 8 00:32:20 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 17, in import nef_theano as nef File "/home/robert/spike/test/../src/nef_theano/init.py", line 1, in from .network import Network File "/home/robert/spike/test/../src/nef_theano/network.py", line 3, in import quantities File "/usr/local/lib/python2.7/dist-packages/quantities/init.py", line 285, in from . import constants File "/usr/local/lib/python2.7/dist-packages/quantities/constants/init.py", line 5, in from .atomicunits import * File "/usr/local/lib/python2.7/dist-packages/quantities/constants/atomicunits.py", line 202, in _cd('atomic mass unit-inverse meter relationship') File "/usr/local/lib/python2.7/dist-packages/quantities/constants/_utils.py", line 15, in _cd return Quantity(entry['value'], entry['units']) File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 127, in new ret._dimensionality.update(validate_dimensionality(units)) File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 34, in validate_dimensionality return unit_registry[value].dimensionality File "/usr/local/lib/python2.7/dist-packages/quantities/registry.py", line 61, in getitem return self.registry[label] File "/usr/local/lib/python2.7/dist-packages/quantities/registry.py", line 21, in getitem return eval(string, self.context) File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/quantities/unitquantity.py", line 230, in pow return self.view(Quantity).pow(other) File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 215, in array_finalize self._dimensionality = getattr(obj, 'dimensionality', Dimensionality()) File "/usr/local/lib/python2.7/dist-packages/quantities/quantity.py", line 132, in dimensionality return self._dimensionality.copy() File "/usr/local/lib/python2.7/dist-packages/quantities/unitquantity.py", line 99, in _dimensionality return Dimensionality({self:1}) KeyboardInterrupt Sun Sep 8 00:32:20 EDT 2013 ^CTraceback (most recent call last): File "/home/robert/spike/test/nengo_tests/test_radius.py", line 33, in net.make('A', 1000, 1, radius=5) File "/home/robert/spike/test/../src/nef_theano/network.py", line 381, in make ep = ensemble.EnsembleProcess(name, node_socket, _args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 33, in init self.ensemble = Ensemble(_args, _kwargs) File "/home/robert/spike/test/../src/nef_theano/ensemble.py", line 215, in init** low=max_rate[0], high=max_rate[1])
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 779, in uniform node_rstate = shared(self.get_substream_rstates(nstreams)) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 696, in get_substream_rstates rval[i] = ff_2p72(rval[i-1]) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 86, in ff_2p72 return multMatVect(rstate, A1p72, M1, A2p72, M2) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 45, in multMatVect r[:3] = matVecModM(A, v[:3], m1) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/rng_mrg.py", line 33, in matVecModM if r >= 0: KeyboardInterrupt

I killed the simulation at Sun Sep 8 00:32:05 EDT 2013.

hudon commented 10 years ago

Can you run what you ran ( /usr/bin/python2 /home/robert/spike/test/nengo_tests/test_radius.py) with their code too so that we're sure it's us?

RobertElder commented 10 years ago

Ran for about 8 hours with their code and did not encounter deadlocks. See branch all-night-deadlock-test-with-their-code for details.

RobertElder commented 10 years ago

Added some debug output, and got it to (appear to) deadlock after running for quite a while.

Output was:

.........(snip).... In ensemble.py EnsembleProcess tick, before recv_pyobj 24542 D In ensemble.py Ensemble tick function before theano_tick 24536
In ensemble.py EnsembleProcess start of tick function 24533 A In ensemble.py EnsembleProcess tick, before recv_pyobj 24539 C In ensemble.py EnsembleProcess tick, after recv_pyobj 24542 D In ensemble.py EnsembleProcess tick, before recv_pyobj 24533 A In ensemble.py EnsembleProcess tick, after recv_pyobj 24539 C In ensemble.py Ensemble tick function before theano_tick 24542
In ensemble.py EnsembleProcess tick, after recv_pyobj 24533 A In ensemble.py Ensemble tick function before theano_tick 24539
In ensemble.py Ensemble tick function before theano_tick 24533
In ensemble.py Ensemble tick function after theano_tick 24536
In ensemble.py EnsembleProcess end of tick function 24536 B In ensemble.py EnsembleProcess start of tick function 24536 B In ensemble.py Ensemble tick function after theano_tick 24539
In ensemble.py EnsembleProcess end of tick function 24539 C In ensemble.py EnsembleProcess start of tick function 24539 C In ensemble.py Ensemble tick function after theano_tick 24533
In ensemble.py Ensemble tick function after theano_tick 24542
In ensemble.py EnsembleProcess end of tick function 24542 D In ensemble.py EnsembleProcess start of tick function 24542 D In ensemble.py EnsembleProcess tick, before recv_pyobj 24536 B In ensemble.py EnsembleProcess tick, after recv_pyobj 24536 B In ensemble.py EnsembleProcess end of tick function 24533 A In ensemble.py EnsembleProcess tick, before recv_pyobj 24542 D In ensemble.py EnsembleProcess start of tick function 24533 A In ensemble.py Ensemble tick function before theano_tick 24536
In ensemble.py EnsembleProcess tick, before recv_pyobj 24539 C In ensemble.py EnsembleProcess tick, before recv_pyobj 24533 A In ensemble.py EnsembleProcess tick, after recv_pyobj 24542 D In ensemble.py EnsembleProcess tick, after recv_pyobj 24539 C In ensemble.py EnsembleProcess tick, after recv_pyobj 24533 A In ensemble.py Ensemble tick function before theano_tick 24539
In ensemble.py Ensemble tick function before theano_tick 24542
In ensemble.py Ensemble tick function before theano_tick 24533
In ensemble.py Ensemble tick function after theano_tick 24536
In ensemble.py EnsembleProcess end of tick function 24536 B In ensemble.py EnsembleProcess start of tick function 24536 B In ensemble.py Ensemble tick function after theano_tick 24539
In ensemble.py EnsembleProcess end of tick function 24539 C In ensemble.py EnsembleProcess start of tick function 24539 C In ensemble.py Ensemble tick function after theano_tick 24533
In ensemble.py Ensemble tick function after theano_tick 24542
In ensemble.py EnsembleProcess end of tick function 24542 D In ensemble.py EnsembleProcess start of tick function 24542 D In ensemble.py EnsembleProcess tick, before recv_pyobj 24536 B In ensemble.py EnsembleProcess end of tick function 24533 A In ensemble.py EnsembleProcess tick, before recv_pyobj 24542 D In ensemble.py EnsembleProcess start of tick function 24533 A In ensemble.py EnsembleProcess tick, after recv_pyobj 24536 B In ensemble.py EnsembleProcess tick, after recv_pyobj 24542 D In ensemble.py Ensemble tick function before theano_tick 24536
In ensemble.py EnsembleProcess tick, before recv_pyobj 24539 C In ensemble.py Ensemble tick function before theano_tick 24542
In ensemble.py EnsembleProcess tick, after recv_pyobj 24539 C In ensemble.py Ensemble tick function before theano_tick 24539
In ensemble.py Ensemble tick function after theano_tick 24536
In ensemble.py EnsembleProcess end of tick function 24536 B In ensemble.py EnsembleProcess start of tick function 24536 B In ensemble.py Ensemble tick function after theano_tick 24542
In ensemble.py EnsembleProcess end of tick function 24542 D In ensemble.py Ensemble tick function after theano_tick 24539
In ensemble.py EnsembleProcess start of tick function 24542 D In ensemble.py EnsembleProcess end of tick function 24539 C In ensemble.py EnsembleProcess start of tick function 24539 C

Will Include better output later.

RobertElder commented 10 years ago

Was able to get into deadlock looking state with gdb. Use command:

while true; do gdb -x /tmp/cmd --args make test; date; done;

file at /tmp/cmd contained run command

Computing output from command '/usr/bin/python2 /home/robert/spike/test/nengo_tests/test_decoded_weight_matrix.py /home/robert/spike/test/../src'...^CProcess Probe2: Traceback (most recent call last): Process Probe1: File "/home/robert/spike/test/nengo_tests/test_decoded_weight_matrix.py", line 67, in Traceback (most recent call last): Traceback (most recent call last): net.run(timesteps * dt_step) File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "/home/robert/spike/test/../src/nef_theano/network.py", line 512, in run self.probes[probe]["data"] = ticker_conn.recv_pyobj() File "socket.pyx", line 800, in zmq.core.socket.Socket.recv_pyobj (zmq/core/socket.c:7090) self.run() File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(_self._args, _self._kwargs) File "/home/robert/spike/test/../src/nef_theano/probe.py", line 92, in run File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) self.tick() File "/home/robert/spike/test/../src/nef_theano/probe.py", line 61, in tick self.run() val = self.input_socket.get_instance().recv_pyobj() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run File "socket.pyx", line 800, in zmq.core.socket.Socket.recv_pyobj (zmq/core/socket.c:7090) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) self._target(_self._args, _self._kwargs) File "/home/robert/spike/test/../src/nef_theano/probe.py", line 99, in run KeyboardInterrupt ticker_conn.recv() # want an ack of receiving the data File "socket.pyx", line 616, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5961) File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832)

Program received signal SIGINT, Interrupt. File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) KeyboardInterrupt File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) Process Probe3: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(_self._args, *_self._kwargs) File "/home/robert/spike/test/../src/nef_theano/probe.py", line 99, in run ticker_conn.recv() # want an ack of receiving the data File "socket.pyx", line 616, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5961) KeyboardInterrupt File "socket.pyx", line 650, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5832) File "socket.pyx", line 119, in zmq.core.socket._recv_copy (zmq/core/socket.c:1664) File "error.pyx", line 92, in zmq.core.error.ZMQError.init (zmq/core/error.c:936) KeyboardInterrupt 0x00007ffff78d23c4 in __libc_wait (stat_loc=0x7fffffffc760) at ../sysdeps/unix/sysv/linux/wait.c:32 32 ../sysdeps/unix/sysv/linux/wait.c: No such file or directory.

(gdb) info threads Id Target Id Frame

(gdb) bt full

0 0x00007ffff78d23c4 in __libc_wait (stat_loc=0x7fffffffc760) at ../sysdeps/unix/sysv/linux/wait.c:32

    resultvar = 18446744073709551104
    oldtype = <optimized out>
    result = <optimized out>

1 0x000000000040febe in ?? ()

No symbol table info available.

2 0x00000000004102fc in ?? ()

No symbol table info available.

3 0x00000000004190a2 in ?? ()

No symbol table info available.

4 0x00000000004198b7 in ?? ()

No symbol table info available.

5 0x0000000000403d88 in ?? ()

No symbol table info available.

6 0x00007ffff783476d in __libc_start_main (main=0x402590, argc=2, ubp_av=0x7fffffffe2b8, init=, fini=,

rtld_fini=<optimized out>, stack_end=0x7fffffffe2a8) at libc-start.c:226
    result = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {0, -7552811529027350682, 4210568, 140737488347824, 0, 0, 7552811528422526822, 
            7552794860517429094}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x41e200, 0x7fffffffe2b8}, data = {prev = 0x0, 
          cleanup = 0x0, canceltype = 4317696}}}
    not_first_call = <optimized out>

7 0x0000000000403fb1 in ?? ()

No symbol table info available.

8 0x00007fffffffe2a8 in ?? ()

No symbol table info available.

9 0x000000000000001c in ?? ()

No symbol table info available.

10 0x0000000000000002 in ?? ()

No symbol table info available.

11 0x00007fffffffe561 in ?? ()

No symbol table info available.

12 0x00007fffffffe56f in ?? ()

No symbol table info available.

13 0x0000000000000000 in ?? ()

No symbol table info available.

RobertElder commented 10 years ago

Later I realized you can just wait for the program to deadlock, then use

ps -ef | grep python

to find all related processes.

You can then attach gdb to each of the running processes and inspect the state of the threads. The threads are all waiting on synchronization points in our code.

Added better debug output:

Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4991 ----- After recv_pyobj. B EnsembleProcess tick function. ----- 4991 ----- After recv_pyobj. D Ensemble tick function. Before theano_tick. 28899 Origin tick function. After send_pyobj. 28896 Ensemble tick function. Before theano_tick. 28905 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. After self.ensemble.tick. 28896 A EnsembleProcess tick function. ----- 4991 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 4991 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4991 ----- After recv_pyobj. C EnsembleProcess tick function. ----- 4991 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. Before theano_tick. 28896 Ensemble tick function. After theano_tick. 28899 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After theano_tick. 28905 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. Completed o.ticks. 28899 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28899 B EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4992 ----- Before recv_pyobj. B EnsembleProcess tick function. ----- 4992 ----- After recv_pyobj. B Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Ensemble tick function. Before theano_tick. 28899 EnsembleProcess tick function. ----- 4992 ----- Before recv_pyobj. D EnsembleProcess tick function. ----- 4992 ----- After recv_pyobj. D Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before theano_tick. 28905 Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. After self.ensemble.tick. 28896 A EnsembleProcess tick function. ----- 4992 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4992 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 28896 Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28902 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. After o.tick. 28902 Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C EnsembleProcess tick function. ----- 4992 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 4992 ----- After recv_pyobj. C Ensemble tick function. After theano_tick. 28899 Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Completed o.ticks. 28899 EnsembleProcess tick function. After self.ensemble.tick. 28899 B Ensemble tick function. After theano_tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 EnsembleProcess tick function. ----- 4993 ----- Before recv_pyobj. B Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4993 ----- After recv_pyobj. B Ensemble tick function. Before theano_tick. 28899 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After theano_tick. 28902 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. Completed o.ticks. 28896 Ensemble tick function. After o.tick. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28896 A Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C EnsembleProcess tick function. ----- 4993 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4993 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 4993 ----- After recv_pyobj. A Ensemble tick function. After theano_tick. 28899 EnsembleProcess tick function. ----- 4993 ----- After recv_pyobj. C Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Completed o.ticks. 28899 EnsembleProcess tick function. ----- 4993 ----- Before recv_pyobj. D EnsembleProcess tick function. After self.ensemble.tick. 28899 B Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. Before theano_tick. 28896 EnsembleProcess tick function. ----- 4993 ----- After recv_pyobj. D Ensemble tick function. Before theano_tick. 28905 Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28902 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. After o.tick. 28902 Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C Ensemble tick function. After theano_tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4994 ----- Before recv_pyobj. B Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4994 ----- After recv_pyobj. B EnsembleProcess tick function. ----- 4994 ----- Before recv_pyobj. D Origin tick function. After send_pyobj. 28896 Ensemble tick function. Before theano_tick. 28899 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. ----- 4994 ----- After recv_pyobj. D EnsembleProcess tick function. After self.ensemble.tick. 28896 A EnsembleProcess tick function. ----- 4994 ----- Before recv_pyobj. C Ensemble tick function. Before theano_tick. 28905 EnsembleProcess tick function. ----- 4994 ----- After recv_pyobj. C EnsembleProcess tick function. ----- 4994 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4994 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. Before theano_tick. 28896 Ensemble tick function. After theano_tick. 28899 Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After o.tick. 28899 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28899 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28899 B EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28902 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. After o.tick. 28902 Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C Ensemble tick function. After theano_tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4995 ----- Before recv_pyobj. B Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4995 ----- After recv_pyobj. B EnsembleProcess tick function. ----- 4995 ----- Before recv_pyobj. D Origin tick function. After send_pyobj. 28896 Ensemble tick function. Before theano_tick. 28899 Ensemble tick function. After o.tick. 28896 EnsembleProcess tick function. ----- 4995 ----- After recv_pyobj. D Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. After self.ensemble.tick. 28896 A Ensemble tick function. Before theano_tick. 28905 EnsembleProcess tick function. ----- 4995 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 4995 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4995 ----- After recv_pyobj. C EnsembleProcess tick function. ----- 4995 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. Before theano_tick. 28896 Ensemble tick function. After theano_tick. 28899 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Completed o.ticks. 28899 EnsembleProcess tick function. After self.ensemble.tick. 28899 B Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28902 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. After o.tick. 28902 Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C Ensemble tick function. After theano_tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4996 ----- Before recv_pyobj. B Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 EnsembleProcess tick function. ----- 4996 ----- After recv_pyobj. B Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4996 ----- Before recv_pyobj. D Ensemble tick function. Before theano_tick. 28899 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. ----- 4996 ----- After recv_pyobj. D EnsembleProcess tick function. After self.ensemble.tick. 28896 A EnsembleProcess tick function. ----- 4996 ----- Before recv_pyobj. C Ensemble tick function. Before theano_tick. 28905 EnsembleProcess tick function. ----- 4996 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4996 ----- After recv_pyobj. C EnsembleProcess tick function. ----- 4996 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. Before theano_tick. 28896 Ensemble tick function. After theano_tick. 28899 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Completed o.ticks. 28899 EnsembleProcess tick function. After self.ensemble.tick. 28899 B Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28902 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. After o.tick. 28902 Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C Ensemble tick function. After theano_tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4997 ----- Before recv_pyobj. B Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 EnsembleProcess tick function. ----- 4997 ----- After recv_pyobj. B Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4997 ----- Before recv_pyobj. D Ensemble tick function. Before theano_tick. 28899 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. After self.ensemble.tick. 28896 A EnsembleProcess tick function. ----- 4997 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 4997 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 4997 ----- After recv_pyobj. C EnsembleProcess tick function. ----- 4997 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 28902 EnsembleProcess tick function. ----- 4997 ----- After recv_pyobj. D Ensemble tick function. Before theano_tick. 28896 Ensemble tick function. Before theano_tick. 28905 Ensemble tick function. After theano_tick. 28899 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Completed o.ticks. 28899 EnsembleProcess tick function. After self.ensemble.tick. 28899 B Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After theano_tick. 28902 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. Completed o.ticks. 28905 Ensemble tick function. After o.tick. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. Completed o.ticks. 28902 Ensemble tick function. After theano_tick. 28896 EnsembleProcess tick function. After self.ensemble.tick. 28902 C Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 Origin tick function. After send_pyobj. 28896 EnsembleProcess tick function. ----- 4998 ----- Before recv_pyobj. B Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4998 ----- After recv_pyobj. B Ensemble tick function. Before theano_tick. 28899 Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Before o.tick. 28896 Origin tick function. Before send_pyobj. 28896 EnsembleProcess tick function. ----- 4998 ----- Before recv_pyobj. D Origin tick function. After send_pyobj. 28896 Ensemble tick function. After o.tick. 28896 Ensemble tick function. Completed o.ticks. 28896 EnsembleProcess tick function. After self.ensemble.tick. 28896 A EnsembleProcess tick function. ----- 4998 ----- After recv_pyobj. D EnsembleProcess tick function. ----- 4998 ----- Before recv_pyobj. C Ensemble tick function. Before theano_tick. 28905 EnsembleProcess tick function. ----- 4998 ----- After recv_pyobj. C Ensemble tick function. Before theano_tick. 28902 Ensemble tick function. After theano_tick. 28899 Ensemble tick function. Before o.tick. 28899 Ensemble tick function. After o.tick. 28899 Ensemble tick function. Completed o.ticks. 28899 EnsembleProcess tick function. After self.ensemble.tick. 28899 B Ensemble tick function. After theano_tick. 28905 Ensemble tick function. Before o.tick. 28905 Ensemble tick function. After o.tick. 28905 Ensemble tick function. Completed o.ticks. 28905 EnsembleProcess tick function. After self.ensemble.tick. 28905 D Ensemble tick function. After theano_tick. 28902 Ensemble tick function. Before o.tick. 28902 Ensemble tick function. After o.tick. 28902 Ensemble tick function. Completed o.ticks. 28902 EnsembleProcess tick function. After self.ensemble.tick. 28902 C Over 3000 times, hang forever. 28899 B Over 3000 times, hang forever. 28896 A Over 3000 times, hang forever. 28902 C Over 3000 times, hang forever. 28905 D

This output was achieved on branch nondeterministic-hang

using this command:

while true; do /usr/bin/python2 /home/robert/spike/test/nengo_tests/test_radius.py /home/robert/spike/test/../src; date; done;

Running for an indeteminate amount of time ( Possibly up to an hour or more)

It looks like the program is one tick away from wrapping up, but the inputs are not available for the last tick for some reason.

Still Investigating.

RobertElder commented 10 years ago

The error is re-producible with only 2 ticks:

Sun Sep 29 00:37:54 EDT 2013 starting simulation Input run function getting sim_time. Before recv. 1598 Network run function sending str time. Before send. 1589 Encoder Test Network run function sending str time. After send. 1589 Encoder Test Network run function sending str time. Before send. 1589 Encoder Test Input run function getting sim_time. After recv. 1598 Network run function sending str time. After send. 1589 Encoder Test Network run function sending str time. Before send. 1589 Encoder Test Input run -------------------- 0 . Before * self.tick. 1598 Network run function sending str time. After send. 1589 Encoder Test Network run function sending str time. Before send. 1589 Encoder Test Network run function sending str time. After send. 1589 Encoder Test Network run function sending str time. Before send. 1589 Encoder Test Origin tick function. Before send_pyobj. 1598 Origin tick function. After send_pyobj. 1598 Input run -------------------- 0 . After * self.tick. 1598 Input run -------------------- 1 . Before * self.tick. 1598 Origin tick function. Before send_pyobj. 1598 Origin tick function. After send_pyobj. 1598 Input run -------------------- 1 . After * self.tick. 1598 Network run function sending str time. After send. 1589 Encoder Test Network run function. Before recv. 1589 Encoder Test EnsembleProcess run function getting time. Before recv. 1610 D EnsembleProcess run function getting time. After recv. 1610 D EnsembleProcess run function getting time. Before recv. 1604 B EnsembleProcess run function getting time. After recv. 1604 B EnsembleProcess run function getting time. Before recv. 1607 C EnsembleProcess run function getting time. After recv. 1607 C Origin tick function. Before send_pyobj. 1601 Origin tick function. After send_pyobj. 1601 Origin tick function. Before send_pyobj. 1601 Origin tick function. After send_pyobj. 1601 EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. B Origin tick function. Before send_pyobj. 1601 EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. B EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. D Origin tick function. After send_pyobj. 1601 EnsembleProcess run function getting time. Before recv. 1601 A Ensemble tick function. Before theano_tick. 1604 EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. D EnsembleProcess run function getting time. After recv. 1601 A EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. C Ensemble tick function. Before theano_tick. 1610 EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. A EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. A Ensemble tick function. Before theano_tick. 1607 Ensemble tick function. Before theano_tick. 1601 Ensemble tick function. After theano_tick. 1604 Ensemble tick function. Before o.tick. 1604 Ensemble tick function. After o.tick. 1604 Ensemble tick function. Completed o.ticks. 1604 EnsembleProcess tick function. After self.ensemble.tick. 1604 B Ensemble tick function. After theano_tick. 1610 Ensemble tick function. Before o.tick. 1610 Ensemble tick function. After o.tick. 1610 Ensemble tick function. Completed o.ticks. 1610 EnsembleProcess tick function. After self.ensemble.tick. 1610 D Ensemble tick function. After theano_tick. 1607 Ensemble tick function. Before o.tick. 1607 Ensemble tick function. After o.tick. 1607 Ensemble tick function. Completed o.ticks. 1607 EnsembleProcess tick function. After self.ensemble.tick. 1607 C Ensemble tick function. After theano_tick. 1601 Ensemble tick function. Before o.tick. 1601 Origin tick function. Before send_pyobj. 1601 Origin tick function. After send_pyobj. 1601 Ensemble tick function. After o.tick. 1601 Ensemble tick function. Before o.tick. 1601 Origin tick function. Before send_pyobj. 1601 EnsembleProcess tick function. ----- 1 ----- Before recv_pyobj. B Origin tick function. After send_pyobj. 1601 Ensemble tick function. After o.tick. 1601 EnsembleProcess tick function. ----- 1 ----- After recv_pyobj. B Ensemble tick function. Before o.tick. 1601 Origin tick function. Before send_pyobj. 1601 Ensemble tick function. Before theano_tick. 1604 EnsembleProcess tick function. ----- 1 ----- Before recv_pyobj. D EnsembleProcess tick function. ----- 1 ----- After recv_pyobj. D Origin tick function. After send_pyobj. 1601 Ensemble tick function. After o.tick. 1601 Ensemble tick function. Completed o.ticks. 1601 Ensemble tick function. Before theano_tick. 1610 EnsembleProcess tick function. After self.ensemble.tick. 1601 A EnsembleProcess tick function. ----- 1 ----- Before recv_pyobj. C EnsembleProcess tick function. ----- 1 ----- After recv_pyobj. C Ensemble tick function. Before theano_tick. 1607 Ensemble tick function. After theano_tick. 1604 Ensemble tick function. Before o.tick. 1604 Ensemble tick function. After o.tick. 1604 Ensemble tick function. Completed o.ticks. 1604 EnsembleProcess tick function. After self.ensemble.tick. 1604 B EnsembleProcess run function sending FIN. Before send. 1604 B EnsembleProcess run function sending FIN. After send. 1604 B EnsembleProcess run function getting ACK. Before recv. 1604 B Ensemble tick function. After theano_tick. 1610 Ensemble tick function. Before o.tick. 1610 Ensemble tick function. After o.tick. 1610 Ensemble tick function. Completed o.ticks. 1610 EnsembleProcess tick function. After self.ensemble.tick. 1610 D EnsembleProcess run function sending FIN. Before send. 1610 D EnsembleProcess run function sending FIN. After send. 1610 D EnsembleProcess run function getting ACK. Before recv. 1610 D Ensemble tick function. After theano_tick. 1607 Ensemble tick function. Before o.tick. 1607 Ensemble tick function. After o.tick. 1607 Ensemble tick function. Completed o.ticks. 1607 EnsembleProcess tick function. After self.ensemble.tick. 1607 C EnsembleProcess run function sending FIN. Before send. 1607 C EnsembleProcess run function sending FIN. After send. 1607 C EnsembleProcess run function getting ACK. Before recv. 1607 C Over 3000 times, hang forever. 1601 A

I believe what is happening is related to the problem that we saw previously with the one tick delay introducing deadlocks because of synchronization problems.

In this topology, the nodes are connected this way:

in -> A

A -> B A -> C A -> D

When we want to run the simulation for 2 ticks, the input will send two ticks to A. Because of the one tick delay, A, starts by sending an extra tick to B, C and D. At this point A will process one of the two inputs it has, and generate output for B, C and D. This output for B, C, and D will complete the simulation, but A still has one input to process. I believe that due to race conditions, sometimes A attempts to process this remaining input, but since the simulation is over and things are shutting down, it waits indefinitely for some closed resource to be available.

The correct behavior would be to send ticks -1 inputs at the beginning, and for any non-terminal node (any node that sends one time step delays) to iterate ticks -1 iterations, while terminal nodes iterate for ticks ticks (because it got 1 from the time step delay and (ticks -1) from the input source).

RobertElder commented 10 years ago

Sun Sep 29 16:00:07 EDT 2013 starting simulation Input run function getting sim_time. Before recv. 13200 Network run function sending str time. Before send. 13191 Encoder Test Network run function sending str time. After send. 13191 Encoder Test Network run function sending str time. Before send. 13191 Encoder Test Input run function getting sim_time. After recv. 13200 Network run function sending str time. After send. 13191 Encoder Test Network run function sending str time. Before send. 13191 Encoder Test Input run -------------------- 0 . Before * self.tick. 13200 Network run function sending str time. After send. 13191 Encoder Test Network run function sending str time. Before send. 13191 Encoder Test Network run function sending str time. After send. 13191 Encoder Test Network run function sending str time. Before send. 13191 Encoder Test Origin tick function. Before send_pyobj. 13200 Origin tick function. After send_pyobj. 13200 Input run -------------------- 0 . After * self.tick. 13200 Input run -------------------- 1 . Before * self.tick. 13200 Origin tick function. Before send_pyobj. 13200 Origin tick function. After send_pyobj. 13200 Input run -------------------- 1 . After * self.tick. 13200 Network run function sending str time. After send. 13191 Encoder Test Network run function. Before recv. 13191 Encoder Test EnsembleProcess run function getting time. Before recv. 13209 C EnsembleProcess run function getting time. After recv. 13209 C EnsembleProcess run function getting time. Before recv. 13212 D EnsembleProcess run function getting time. After recv. 13212 D EnsembleProcess run function getting time. Before recv. 13206 B EnsembleProcess run function getting time. After recv. 13206 B Origin tick function. Before send_pyobj. 13203 Origin tick function. After send_pyobj. 13203 Origin tick function. Before send_pyobj. 13203 Origin tick function. After send_pyobj. 13203 EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. B Origin tick function. Before send_pyobj. 13203 EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. B EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. D Ensemble tick function. Before theano_tick. 13206 Origin tick function. After send_pyobj. 13203 EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. D EnsembleProcess run function getting time. Before recv. 13203 A EnsembleProcess run function getting time. After recv. 13203 A EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. C Ensemble tick function. Before theano_tick. 13212 EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. C Ensemble tick function. Before theano_tick. 13209 Ensemble tick function. After theano_tick. 13206 Ensemble tick function. Before o.tick. 13206 Ensemble tick function. After o.tick. 13206 Ensemble tick function. Completed o.ticks. 13206 EnsembleProcess tick function. After self.ensemble.tick. 13206 B Ensemble tick function. After theano_tick. 13212 Ensemble tick function. Before o.tick. 13212 Ensemble tick function. After o.tick. 13212 Ensemble tick function. Completed o.ticks. 13212 EnsembleProcess tick function. After self.ensemble.tick. 13212 D Ensemble tick function. After theano_tick. 13209 Ensemble tick function. Before o.tick. 13209 Ensemble tick function. After o.tick. 13209 Ensemble tick function. Completed o.ticks. 13209 EnsembleProcess tick function. After self.ensemble.tick. 13209 C Over 3000 times, hang forever. 13203 A in_0 not in responses. 13203 A Over 3000 times, hang forever. 13212 D A_0 not in responses. 13212 D Over 3000 times, hang forever. 13209 C A_0 not in responses. 13209 C Over 3000 times, hang forever. 13206 B A_0 not in responses. 13206 B

Looks like for some reason, A should have a piece of input available for it, but it doesn't take it? Still investigating...

RobertElder commented 10 years ago

I went with the theory that A is binding sockets after the input has sent to it, but it looks like in the error case it binds after (assuming we can trust the ordering on printed statements and there is no nasty race conditions in buffering output.).

starting simulation Input run function getting sim_time. Before recv. 561 EnsembleProcess run Before bind_sockets. 564 A EnsembleProcess run After bind_sockets. 564 A EnsembleProcess run Before bind_sockets. 567 B EnsembleProcess run After bind_sockets. 567 B EnsembleProcess run Before bind_sockets. 570 C EnsembleProcess run After bind_sockets. 570 C Network run function sending str time. Before send. 552 Encoder Test Network run function sending str time. After send. 552 Encoder Test Network run function sending str time. Before send. 552 Encoder Test Network run function sending str time. After send. 552 Encoder Test Input run function getting sim_time. After recv. 561 Network run function sending str time. Before send. 552 Encoder Test Input run -------------------- 0 . Before * self.tick. 561 Network run function sending str time. After send. 552 Encoder Test Network run function sending str time. Before send. 552 Encoder Test Network run function sending str time. After send. 552 Encoder Test Network run function sending str time. Before send. 552 Encoder Test Input tick function Before setting float. 561 EnsembleProcess run Before bind_sockets. 573 D Input tick function After settingfloat. 561 Input tick function Before o.tick. 561 Origin tick function. Before send_pyobj. 561 EnsembleProcess run After bind_sockets. 573 D send_pyobj sending : [ 0.] Origin tick function. After send_pyobj. 561 Input tick function After o.tick. 561 Input run -------------------- 0 . After * self.tick. 561 Network run function sending str time. After send. 552 Encoder Test Network run function. Before recv. 552 Encoder Test EnsembleProcess run function getting time. Before recv. 570 C EnsembleProcess run function getting time. After recv. 570 C EnsembleProcess run function getting time. Before recv. 573 D EnsembleProcess run function getting time. After recv. 573 D Origin tick function. Before send_pyobj. 564 send_pyobj sending : [ 0.] Origin tick function. After send_pyobj. 564 Origin tick function. Before send_pyobj. 564 send_pyobj sending : [ 0.] Origin tick function. After send_pyobj. 564 Origin tick function. Before send_pyobj. 564 EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. C send_pyobj sending : [ 0.] EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. C Origin tick function. After send_pyobj. 564 EnsembleProcess run function getting time. Before recv. 564 A EnsembleProcess run function getting time. After recv. 564 A Ensemble tick function. Before theano_tick. 570 EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. D EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. D Ensemble tick function. Before theano_tick. 573 Ensemble tick function. After theano_tick. 570 Ensemble tick function. Before o.tick. 570 Ensemble tick function. After theano_tick. 573 Ensemble tick function. After o.tick. 570 Ensemble tick function. Before o.tick. 573 Ensemble tick function. Completed o.ticks. 570 Ensemble tick function. After o.tick. 573 Ensemble tick function. Completed o.ticks. 573 EnsembleProcess tick function. After self.ensemble.tick. 570 C EnsembleProcess tick function. After self.ensemble.tick. 573 D EnsembleProcess run function sending FIN. Before send. 573 D EnsembleProcess run function sending FIN. Before send. 570 C EnsembleProcess run function sending FIN. After send. 573 D EnsembleProcess run function sending FIN. After send. 570 C EnsembleProcess run function getting ACK. Before recv. 573 D EnsembleProcess run function getting ACK. Before recv. 570 C EnsembleProcess run function getting time. Before recv. 567 B EnsembleProcess run function getting time. After recv. 567 B EnsembleProcess tick function. ----- 0 ----- Before recv_pyobj. B EnsembleProcess tick function. ----- 0 ----- After recv_pyobj. B Ensemble tick function. Before theano_tick. 567 Ensemble tick function. After theano_tick. 567 Ensemble tick function. Before o.tick. 567 Ensemble tick function. After o.tick. 567 Ensemble tick function. Completed o.ticks. 567 EnsembleProcess tick function. After self.ensemble.tick. 567 B EnsembleProcess run function sending FIN. Before send. 567 B EnsembleProcess run function sending FIN. After send. 567 B EnsembleProcess run function getting ACK. Before recv. 567 B Over 3000 times, hang forever. 564 A in_0 not in responses. 564 A

RobertElder commented 10 years ago

while true; do /usr/bin/python2 /home/robert/spike/test/nengo_tests/test_radius.py /home/robert/spike/test/../src; date; done;

RobertElder commented 10 years ago

Fixed.