How does the script calculate the train error

[johnny5550822]

When I run mnist example, I understand that how to calculate the val error (which is go through all the testing examples). What about the train error, how does the we obtain that? Is that by simple validation in training?

This image is generated by some other dataset, not mnist

[lenck]

Hi, yes, function computes error rates on both train (imdb.images.set==1) and validation (imdb.images.set==2) data, but not on test data - as in most of the recent datasets (like ILSVRC) you do not have access to the ground truth label in order to prevent overfitting on the test set (and the performance on the test data is usually measured on some remote server). The nn_train should be computing errors on both train and val subsets, so what it seems from the figure you sent is that you don't have any val data in the imdb structure (those with imdb.images.set==2).

[johnny5550822]

Thanks for the reply. So, how exactly the training error being calculated during batching?

[lenck]

In updateError function on line 269, the error is accummulated and then e.g. on line 172 it is divided by the number of already processed batches. So you are basically averaging the errors per each batch as you go...

[johnny5550822]

so that mean I use the training data to train the network and at the same time pass on the network and see the classification result?

[felixachilles]

As Karel explained, you train on the training-part of your data (set=1). The error-plot that is updated after each epoch shows you the current error on this training-part as well as on the validation-part (set=2). If you did not define a validation-set, you will only see one curve (plus the top-5-error curve, which might be confusing at first). In the updateError() function you can define how exactly the forward-pass of the current (train or val)-batch happens and how the error is calculated from that subsequently. Adaptions have to be made for different outputs (binary/multiclass/scene labelling/regression).

2015-01-23 21:03 GMT+01:00 johnny5550822 notifications@github.com:

so that mean I use the training data to train the network and at the same time pass on the network and see the classification result?

— Reply to this email directly or view it on GitHub https://github.com/vlfeat/matconvnet/issues/43#issuecomment-71256289.

---

---

_Technische Universität München_Felix Achilles, M.Sc.* · Faculty of Informatics · Chair for Computer Aided Medical Procedures Boltzmannstr. 3 · *85748 Garching b. München · +49 (0)176 623 18 312

achilles@in.tum.de · campar.in.tum.de

---

[johnny5550822]

Great, I got it.

And there is no cross-validation being implemented in the lib, right?

[vedaldi]

Hi, yes, no cross-validation. It takes days to train a single large-scale model unfortunately with ours or other toolkits, so automatic cross-validation did not seem a priority.

I should remark that the cnn_train function is just an example (although probably just as efficient as any standard training code). The training error is a bit of a hack as it is calculated while the network is being updated, on a "rolling" basis. The proper way would be to freeze the model and evaluate it on the whole training set, but that would be an overkill. Ultimately when the model does not change much it will give a fairly accurate idea of the actual training error in any case.