Open Chop1 opened 3 years ago
In theory, this should be easy to configure: https://www.tensorflow.org/guide/mixed_precision I don't know much of the OD API code base. Need the OD API team to answer where to add the mixed precision call.
following this question, I am able to train object detection efficientdet model using mixed precision training (https://www.tensorflow.org/guide/mixed_precision#summary). As I am casting prediction output and target tensor both to "float32" before calculation of the loss, Is it fine if I don't use LossScaleOptimizer? Also, when I analyze the size of saved checkpoints while training using mixed precision and w/o mixed precision, they both are same. Along with training benefits, does saved model also have smaller disk footprint compared to one trained not using mixed precision or they save all weights in float32?
2. Describe the issue
When constructing a model with the tensorflow 2 object detection API, what is the floating precision used during training ?
Also, the pipeline.config used to configurate a model offer the possibility to use "bfloat". Is this possible on GPU or is it limited to TPU ?