[13/06/2020] Saturday 9:00 PM GMT+2 An overview of gradient descent optimization algorithms

[Omarito2412]

Please join us in our discussion of "An overview of gradient descent optimization algorithms" on Saturday at 9:00 pm GMT+2. Blog post: https://ruder.io/optimizing-gradient-descent/index.html

[Ragabov]

Techniques mentioned during the discussion:

1- Learning Rate Finder https://www.pyimagesearch.com/2019/08/05/keras-learning-rate-finder/

2- Warm-ups and restarts https://ruder.io/deep-learning-optimization-2017/index.html#warmrestarts https://openreview.net/forum?id=r14EOsCqKX

3- Curriculum Learning for NLP https://www.aclweb.org/anthology/N19-1119.pdf

[ibrahimsharaf]

• ADAM is still the de-facto optimizer in most cases.
• Most frameworks now support distributed training out-of-the-box.
• @KhaledEssam mentioned that they have used Flesch–Kincaid readability tests for sorting datasets for curriculum learning for a NLU project at Microsoft.
• @EngSalem has pointed out to this reading group which contains some resources for optimization.