Question about FP16 training

mit-han-lab / efficientvit

EfficientViT is a new family of vision models for efficient high-resolution vision.

Apache License 2.0

1.6k stars 142 forks source link

Question about FP16 training #59

Closed yvielcastillejos closed 5 months ago

yvielcastillejos commented 5 months ago

In line 399-400 of https://github.com/mit-han-lab/efficientvit/blob/master/efficientvit/models/nn/ops.py ,

it seems that we convert back to FP32 during training. Why would this be the case? Does training that part with FP16 significantly cause low accuracy?

MoritzMo123 commented 5 months ago

checkout #15