search

Neuroglycerin / neukrill-net-tools

Tools coded as part of the NDSB competition.
MIT License
0 stars 0 forks source link

Making predictions on models with augmentation #105

Closed gngdb closed 9 years ago

gngdb commented 9 years ago

Currently not supported in test.py. Should probably implement two ways of doing this. One would be to ignore the augmentations and just predict using the network trained with augmentations. The second is to predict on all the different augmentations and average the results. Second will run into memory problems unless done sequentially.

gngdb commented 9 years ago

Wrote a really stupid way of doing this loading everything into RAM. Ran, and couldn't see any problems but then on submission was not good. Probably bugs.

gngdb commented 9 years ago

Either the submission csv is being written wrong, or our estimate of the performance from the validation score is massively wrong. Best way to get to the bottom of this would be to run the model over the 10% holdout set and see if it's as bad as on the Kaggle leaderboard. If it is, then either we're combining the results badly or the model is massively overfitting (it would have to be really bad to get these results though).

This relates to this issue, and repeating what I wrote there: surely Pylearn2 has some way of running tests like this on stored model files?

gngdb commented 9 years ago

Turns out I forgot to put in the code to apply normalisation to the test data. Fixed that, and submitted again and we got the score we were expecting: 0.8