An open-source implementation of the Hessian-free optimization algorithm, based on the "Deep Learning via Hessian-free Optimization" paper by James Martens (ICML 2010, http://www.cs.toronto.edu/~jmartens/docs/Deep_HessianFree.pdf). The implementation is in Theano (http://deeplearning.net/software/theano), a highly efficient framework for defining and optimizing expressions with multi-dimensional arrays, and re-uses a lot of code from the Deep Learning Tutorials (http://www.deeplearning.net/tutorial).

The prerequisites for using the Hessian-free code are:

• a working installation of Theano in the python path
• a copy of the Deep Learning tutorials, which can be checked out from https://github.com/lisa-lab/DeepLearningTutorials (simply add the code/ directory to PYTHONPATH)
• a one-time execution of the download.sh script (sh download.sh)

Once these pre-requisities are satisfied, you can run the code as follows:

python run_autoencoder.py -p jacobi -b 500 -e 50 -d mnist -o hf

this will load the MNIST dataset, perform 50 iterations of layer-wise RBM pre-training, followed by Hessian-free optimization of the global reconstruction cost using mini-batches of size 5000 and an (approximate) Jacobi pre-conditioner. Supervised learning follows.

The code is companion to the following paper:

Olivier Chapelle and Dumitru Erhan. Improved preconditioner for hessian free optimization. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. http://olivier.chapelle.cc/pub/precond.pdf

For questions and/or comments, contact Olivier Chapelle and Dumitru Erhan (usernames chap and dumitru on the yahoo-inc.com domain).