search

Neuroglycerin / neukrill-net-tools

Tools coded as part of the NDSB competition.
MIT License
0 stars 0 forks source link

Use test set #106

Closed gngdb closed 9 years ago

gngdb commented 9 years ago

At the moment we track progress of train and validation scores but for some reason don't seem to look at the score on the holdout set called "test". Seems like it might be useful to evaluate different models on this set and seems like something so obvious it's likely implemented somewhere in Pylearn2.

gngdb commented 9 years ago

Can't find a script that does this. Suspect you're just supposed to add the test set to monitored parameters as you run the model, then you'll have access to it's stored values. Then, there's the script print_monitor.py and you can just run that on multiple pickle files and read off the test set scores.

gngdb commented 9 years ago

Script for this now exists called check_test_score.py in work repo. Can check the expected score of any Pylearn2 model on holdout set. Most of the code, and some notes on how it works, can be found in the notebook called Holdout testing for Pylearn2 pickles.