search

tensorflow / models

Models and examples built with TensorFlow
Other
77.04k stars 45.77k forks source link

floating precision & bfloat on GPU #9708

Open Chop1 opened 3 years ago

Chop1 commented 3 years ago

2. Describe the issue

When constructing a model with the tensorflow 2 object detection API, what is the floating precision used during training ?

Also, the pipeline.config used to configurate a model offer the possibility to use "bfloat". Is this possible on GPU or is it limited to TPU ?

saberkun commented 3 years ago

In theory, this should be easy to configure: https://www.tensorflow.org/guide/mixed_precision I don't know much of the OD API code base. Need the OD API team to answer where to add the mixed precision call.

purvang3 commented 2 years ago

following this question, I am able to train object detection efficientdet model using mixed precision training (https://www.tensorflow.org/guide/mixed_precision#summary). As I am casting prediction output and target tensor both to "float32" before calculation of the loss, Is it fine if I don't use LossScaleOptimizer? Also, when I analyze the size of saved checkpoints while training using mixed precision and w/o mixed precision, they both are same. Along with training benefits, does saved model also have smaller disk footprint compared to one trained not using mixed precision or they save all weights in float32?