search

allenai / deep_qa

A deep NLP library, based on Keras / tf, focused on question answering (but useful for other NLP too)
Apache License 2.0
404 stars 133 forks source link

Enable evaluation on epoch with max validation acc. #189

Closed nelson-liu closed 7 years ago

nelson-liu commented 7 years ago

I think model loading is broken right now due to custom layers, but I would need to write a test and check. Until we fix model loading, we can't evaluate on test set with the epoch of a model that had the max validation accuracy (done by serializing then reloading said epoch).

We also need hooks in the scala code to use these in experiments, and make sure the evaluation output is correct with it.

nelson-liu commented 7 years ago

Model loading was recently fixed, so updating this issue. #181 has some work in evaluating on test et data in the python code, but it'd be good to be able to evaluate with the highest performing epoch (as opposed to the last epoch, which it's doing now). In addition, it'd be nice to be able to specify a test file in the scala experiments code. This would also require rewriting the functions that parse the output from an experiment (i.e. accuracy and such).

matt-gardner commented 7 years ago

Fixed by #302