Increase modelling power by increasing network size (constrained by number of training points!)
or easier by applying an additional nonlinear transformation to the network states x(n) to output: E.g. use squared version of the network states and expand Wout to 2N+2 entries and xsquares=(u(n),x(n),u2(n),x2(n)).
Insert noise into state update (look this up)
Online learning methods: recompute W^{out} after every update of the reservoir states x(t) which might give better performance
RLS(Recursive Least Squares)-based ESN online learning algorithm
the RLS algorithm minimizes the exponentially discounted square "pre-error":
yn(k) is the model output that would be obtained at time k if the current output matrix would be employed at all times k=0,...,n
Fully-connected Reservoir Network with BPDC algorithm
better implement the backpropagation-decorrelation algorithm with fully connected reservoir, which according to the paper is better than RLS as it does not conditions on the matrix W to apply as in an Echo State Network.
No matter what training algorithm we used, we should implement a sort of 'bagging' by averaging over the results obtained by using multiple different reservoir initializations.
Bootstrapping vs Cross-validation
Cross-validation and bootstrapping are both methods for estimating generalization error based on "resampling". The resulting estimates of generalization error are often used for choosing among various models, such as different network architectures.
Bootstrapping seems to work better than cross-validation in many cases (Efron, 1983). In the simplest form of bootstrapping, instead of repeatedly analyzing subsets of the data, you repeatedly analyze subsamples of the data. Each subsample is a random sample with replacement from the full
sample. Depending on what you want to do, anywhere from 50 to 2000 subsamples might be used. There are many more sophisticated bootstrap methods that can be used not only for estimating generalization error but also for estimating confidence bounds for network outputs. There is more on this topic on this website.
Echo State Networks
Optimization of the network size
shown in this paper
Fully-connected Reservoir Network with BPDC algorithm
No matter what training algorithm we used, we should implement a sort of 'bagging' by averaging over the results obtained by using multiple different reservoir initializations.
Bootstrapping vs Cross-validation
Cross-validation and bootstrapping are both methods for estimating generalization error based on "resampling". The resulting estimates of generalization error are often used for choosing among various models, such as different network architectures. Bootstrapping seems to work better than cross-validation in many cases (Efron, 1983). In the simplest form of bootstrapping, instead of repeatedly analyzing subsets of the data, you repeatedly analyze subsamples of the data. Each subsample is a random sample with replacement from the full sample. Depending on what you want to do, anywhere from 50 to 2000 subsamples might be used. There are many more sophisticated bootstrap methods that can be used not only for estimating generalization error but also for estimating confidence bounds for network outputs. There is more on this topic on this website.