FNNs + RNNs (Echo State Networks) - implementation

[f4bD3v]

Review of model and training method. Design models.

[mstefanro]

I've read the paper from 2010 and given this some thought. We will use a feed-forward NN to learn some neurons and then use them in the ESRNN. We will probably not predict prices separately/individually, but all of them at once (we will see whether we can do per-region prediction or just per-country averages).

[mstefanro]

We will probably use Numba (http://numba.pydata.org/) to have JIT on our python code (great speed improvements). We might benefit from Numexpr (https://github.com/pydata/numexpr) at one point as well.

[f4bD3v]

Let's finalize the networks on tuesday so we can start implementation

[f4bD3v]

Feedforward implementation: In Pybrain there are different kinds of layers. The important ones for the feedforward implementation are:

• LinearLayer inputL = TanhLayer(h1, name="input layer")
• TanhLayer - _hiddenL_1 = TanhLayer(h1, name="hidden layer 1 - R_iPj")

The layers are added to the network as modules:

• ffn.addInputModule(...)
• ffn.addOutputModule(..)
• ffn.addModule(..) - hidden layers

Each module has an input and an output buffer in which each neuron/unit represents an index. These index ranges can be sliced as to allow custom connections between different ranges of neurons/units of two layers. The most simple connection between two modules(layers) is of course the FullConnection - weights between all combinations of neurons in these two layers.

Current PROBLEM: Not 100% clear how the indexing works. Have sliced hidden layer 2 according to the conception of the feedforward network and have also created the connections between layer 1 and 2 with the proper indices connected, however the resulting connections have only 3 instead of 4*2=8 parameters/weights h2_inSlices[i] = ModuleSlice(hiddenL_2, inSliceFrom=h2_inIndices[i][0], inSliceTo=h2_inIndices[i][1])

[mstefanro]

I will look into this and assist soon. The first hidden layer should be linear as well, I think (because it should do some weighted-average kind of work).

On 04/24/2014 08:46 PM, Fabian Brix wrote:

In Pybrain there are different kinds of layers. The important ones for the feedforward implementation are:

• LinearLayer /inputL = TanhLayer(h1, name="input layer")/
• TanhLayer - /hiddenL_1 = TanhLayer(h1, name="hidden layer 1 - R_iP_j")/

The layers are added to the network as modules:

• ffn.addInputModule(...)
• ffn.addOutputModule(..)
• ffn.addModule(..) - hidden layers

Each module has an input and an output buffer in which each neuron/unit represents an index. These index ranges can be sliced as to allow custom connections between different ranges of neurons/units of two layers. The most simple connection between two modules(layers) is of course the FullConnection - weights between all combinations of neurons in these two layers.

Current PROBLEM: Not 100% clear how the indexing works. Have sliced hidden layer 2 according to the conception of the feedforward network and have also created the connections between layer 1 and 2 with the proper indices connected, however the resulting connections have only 3 instead of 4*2=8 parameters/weights h2_inSlices[i] = ModuleSlice(hiddenL_2, inSliceFrom=h2_inIndices[i][0], inSliceTo=h2_inIndices[i][1])

— Reply to this email directly or view it on GitHub https://github.com/fabbrix/humanitas/issues/8#issuecomment-41317197.

[f4bD3v]

ok, fixed the layer.

There's an alternative to ESNs that might be easier to implement with pybrain, because the method was developed by the creators of pybrain.

In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for training RNNs, due to numer- ous local minima. For such cases, we present a novel method: EVOlution of systems with LINear Outputs (Evolino). Evolino evolves weights to the nonlinear, hidden nodes of RNNs while computing optimal linear mappings from hidden state to output, using methods such as pseudo- inverse-based linear regression. If we instead use quadratic programming to maximize the margin, we obtain the first evolutionary recurrent sup- port vector machines. We show that Evolino-based LSTM can solve tasks that Echo State nets (Jaeger, 2004a) cannot and achieves higher accuracy in certain continuous function generation tasks than conventional gradient descent RNNs, including gradient-based LSTM.

Training recurrent networks by Evolino

[f4bD3v]

The ESN can be either implemented in plain scientific python or using the reservoir computing package Oger