Add a warning in the defaults.conf for BERTAdam, and remove it as a default from all scripts

[jeswan]

Issue by sleepinyourhat Thursday Jun 11, 2020 at 21:52 GMT Originally opened as https://github.com/nyu-mll/jiant/issues/1096

---

BERTAdam considered harmful in fine-tuning:

https://arxiv.org/abs/2006.05987 https://arxiv.org/abs/2006.04884

In the spirit of Reasonable Defaults:

• Add a warning in the defaults.conf where BERTAdam is mentioned
• Remove it as a default from all example scripts
• Make sure that our support for conventional Adam is working, and use it as the replacement

(Relevant to @claravania)