search

nyu-mll / jiant-v1-legacy

The jiant toolkit for general-purpose text understanding models
MIT License
21 stars 9 forks source link

Consider QHAdam #687

Open jeswan opened 4 years ago

jeswan commented 4 years ago

Issue by sleepinyourhat Sunday May 12, 2019 at 17:18 GMT Originally opened as https://github.com/nyu-mll/jiant/issues/687

This seems to be the Fancy New Optimizer Presented at ICLR for this year, with improvements over Adam:

https://openreview.net/forum?id=S1fUpoR5FQ

I don't follow optimization closely, though, so there may be better alternatives out there, or reasons to doubt that these improvements will show up for what we're doing.

jeswan commented 4 years ago

Comment by sleepinyourhat Sunday May 12, 2019 at 17:22 GMT

Though here's a competing FNOPaI: https://openreview.net/forum?id=Bkg3g2R9FX