search

nyu-mll / jiant-v1-legacy

The jiant toolkit for general-purpose text understanding models
MIT License
21 stars 9 forks source link

Speculative: Add support for automatic early stopping of poor runs #1055

Open jeswan opened 4 years ago

jeswan commented 4 years ago

Issue by sleepinyourhat Thursday Apr 09, 2020 at 21:08 GMT Originally opened as https://github.com/nyu-mll/jiant/issues/1055

Inspired by: https://arxiv.org/abs/2002.06305

Minimally: End training if a certain target accuracy isn't reached by some (early-ish) number of steps. This can be set manually to automatically catch jobs that fail due to a poor random seed.

Ideally: Integrate with some kind of experiment manager for hyperparameter tuning, so as to kill jobs that perform well below average early in tuning.