How to save trained multi-task model & evaluate on test data

[lashmore]

Jiant's documentation ends at training, which evaluates on the "eval" data partition. However, what if I want to then:

1. Save the model that was learned on multiple tasks (shared encoder + each task head)
2. Use this model to evaluate on a test dataset (withheld from training).

It's not clear to me which code in jiant's codebase would help me accomplish these aims.

UPDATE:: Note, I saw this and originally I wasn't getting the files that have the trained model's parameters.

https://github.com/nyu-mll/jiant/blob/master/guides/experiments/my_experiment_and_me.md

But then I set "do_save=True" in the main_runscript.RunConfiguration() call and now I am.

So now all I'm needing help with is how to evaluate on the test data partition using the model file saved as best_model.p

[zphang]

jiant doesn't currently support evaluation for test splits, given that test labels are usually not available. Instead, there is a write_test_preds argument that writes out test predictions to torch pickles. I recommend writing out the test preds and then separately using your own evaluation code to evaluate against your test labels.