no 1-time-step delay in distribute and subensembles

[hudon]

Our ensembles are using the data from the current tick (no delay), this causes undefined behavior (or deadlock, in our case), for recurrent networks (A->B->C, C->B).

See Terry's email:

Interesting. That sounds like it might be a bug in the Theano code. The general principle should be that any computing for any component is done on the data coming out of the parts that feed into it on the previous time step (another way of thinking of this is that there's a 1-time-step time delay on every transmission line). We can't go with a zero-time step delay, as that would make the performance of networks with recurrent connections undefined. But it sounds like you're describing a 2-time-step delay. Is that right? Or am I confused? Basically, on the very first time step, every component should be seeing a zero input, but after that time step they should be seeing whatever the output was of the other components. That's the intent, anyway -- I don't think the code has been rigorously tested for that.

In any case, it would be best if we can get it to be consistent between your version and the original version, but as to which way the consistency should be, I'm open for debate. As long as it's consistent, and works for recurrent connections.

As for the run(0.001), that should just be there to simplify the timing -- Theano does lots of setup the first time run() is called, so just I didn't want to include that in the timing calculations.

Solution: force all the ensembles to run on zeroes on first tick

[gretac]

Fixed in 84330484bfd239a65642941202bbd2961e6d3567