Our ensembles are using the data from the current tick (no delay), this causes undefined behavior (or deadlock, in our case), for recurrent networks (A->B->C, C->B).
See Terry's email:
Interesting. That sounds like it might be a bug in the Theano code.
The general principle should be that any computing for any component
is done on the data coming out of the parts that feed into it on the
previous time step (another way of thinking of this is that there's a
1-time-step time delay on every transmission line). We can't go with
a zero-time step delay, as that would make the performance of networks
with recurrent connections undefined. But it sounds like you're
describing a 2-time-step delay. Is that right? Or am I confused?
Basically, on the very first time step, every component should be
seeing a zero input, but after that time step they should be seeing
whatever the output was of the other components. That's the intent,
anyway -- I don't think the code has been rigorously tested for that.
In any case, it would be best if we can get it to be consistent
between your version and the original version, but as to which way the
consistency should be, I'm open for debate. As long as it's
consistent, and works for recurrent connections.
As for the run(0.001), that should just be there to simplify the
timing -- Theano does lots of setup the first time run() is called, so
just I didn't want to include that in the timing calculations.
Solution: force all the ensembles to run on zeroes on first tick
Our ensembles are using the data from the current tick (no delay), this causes undefined behavior (or deadlock, in our case), for recurrent networks (A->B->C, C->B).
See Terry's email:
Solution: force all the ensembles to run on zeroes on first tick