Add training tool

[MartinThoma]

A tool for MLP training should be added.

Before this can be tackled, issue #3 has to be solved.

[MartinThoma]

From a users perspective I want something like

$ nntoolkit train -m model.tar -i data.tar

We have to think about which input format we want. I guess we could use a .tar archive with two HDF5 files: x.hdf5 and y.hdf5, where x.hdf5 contains the features and y contains the labels. For example, x could contain a (10,000,000 x 160) matrix. That means we have 10,000,000 training examples and each training example has 160 features. Then y.hdf5 will contain a (10,000,000 x 1) matrix.

I think using two files (x.hdf5 and y.hdf5) is better than using simply one, because that makes the difference between feature vectors and labels clearer.

HDF5 can store floats and strings, so it seems to suit our needs.

We should implement mini-batch gradient descent with classification error first. On the long term, I would like to add things like momentum or alternative error measures (e.g. cross-entropy).

[sadaf2605]

I also think it would be easy to make it that way, at least initially it would be fine. if we keep separate class/method for retrieving from dataset then it won't be difficult to change them if we want to.

What do you think about implementation? Should we implement with theano from the beginning or first without theano then we can change it for theano?

[MartinThoma]

For training, I think it makes sense to directly implement it with Theano. But if you think it would be easier to make a non-Theano version first and then change it to use Theano, feel free to do so.

[MartinThoma]

Here is an implementation which is pretty much what I want: http://deeplearning.net/tutorial/mlp.html

To test if it worked, try:

nntookit/train/misc$ nntoolkit train -m model.tar -i testdata.tar -o trained.tar

(after you have installed the latest development version of nntoolkit, e.g. with sudo python setup.py install)

[MartinThoma]

The training tool still does not work at all. @sadaf2605 I would appreciate your help here.

This is how I test if the training works:

tests/mnist$ ./get_data.py
[...]

tests/mnist$ nntoolkit create -t mlp -a 784:100:10 -f mnist_classifier.tar
2015-02-05 21:11:09,868 INFO Create mlp with a 784:100:10 architecture...

tests/mnist$ nntoolkit test -m mnist_classifier.tar -i mnist_testdata.tar
Correct: 1234/10000 = 0.12 of total correct

tests/mnist$ nntoolkit train -m mnist_classifier.tar -i mnist_traindata.tar -o mnist_classifier.tar --epochs 1 -lr 10 --batchsize 1
[...]

tests/mnist$ nntoolkit test -m mnist_classifier.tar -i mnist_testdata.tar
Correct: 2345/10000 = 0.23 of total correct

It should especially work with batchsizes greater than 1 and after training it should perform better (not necessarily with a lr of 10, but rather a smaller one).

[sadaf2605]

@MartinThoma, Alright when I could get free time I'll definitely try to do that.