RFC: Keras Mixed Precision API

[reedwm]

This RFC will be open for comment until October 14, 2020.

Status	Proposed
RFC #	293
Author(s)	Reed Wanderman-Milne (reedwm@google.com
Sponsor	Francois Chollet (fchollet@google.com)
Updated	2020-09-29

Objective

Make mixed precision easy to use in Keras.

[fchollet]

We did the internal design review for this API today. Since there were no outstanding issues, it was very quick.

Notes:

• Reed: Let's present the RFC and go over the changes that happened since we sent it out.
• Before the changes: there were 3 dynamic loss scale classes. They were glorified named tuples
• Now: everything related to loss scaling moved to LossScaleOptimizer
  • There's a dynamic boolean argument to use dynamic scaling or not
  • if true, you can use initial_scale and dynamic_growth_steps
  • It's a simple API
• Q from Tomer: do you always have to use loss scale optimizer when using mixed precision?
  • Reed: it's automated with compile/fit, but with a custom training loop you need to use it
  • This is well documented

Status:

Design approved after the minor changes described during the meeting.