search

NVIDIA / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
672 stars 263 forks source link

Mixed Precision and FP16 #556

Closed cathy-kim closed 5 years ago

cathy-kim commented 5 years ago

Hi, I'm still trying to train SSD using mixed precision. What is the difference between the mixed precision training and FP16 training?

At @birusgin 's presentation(http://on-demand.gputechconf.com/gtc-il/2017/presentation/sil7116-boris-ginsburg-mixed-precision-training-of-deep-nn-with-volta.pdf) , FP16 training is different from Mixed Precision Training. And SSD's mAP of FP16 is 77.1%, higher than FP32. (http://on-demand.gputechconf.com/gtc/2017/presentation/s7218-training-with-mixed-precision-boris-ginsburg.pdf)

However, SSD's mAP of the Mixed precision is also 77.1% at other NVIDIA presentation and papers. (http://on-demand.gputechconf.com/gtc/2018/presentation/s8923-training-neural-networks-with-mixed-precision-theory-and-practice.pdf)

Are the mixed precision and FP16 training same? Even when I trained SSD in FP16, mAP didn't outperform that of FP32 training.
It was worse.

Please make it clear.

Hi @Seojin24 could you try this please:

#global_grad_scale: 1000.
global_grad_scale_adaptive: true

Thank you for answer. The problem was solved when I added "default_forward_math: FLOAT16 default_backward_math: FLOAT16"
and "Foward_type: FLOAT, Backward_type:FLOAT" to every PriorBox layer.

However, the pretrained model issue was not solved. Is a pretrained model also should have been trained as FP16 when I want to use it in FP16 training mode?

Originally posted by @ginn24 in https://github.com/NVIDIA/caffe/issues/551#issuecomment-459974880

drnikolaev commented 5 years ago

Hi @ginn24 if layer has only CPU implementation, it's better to use fp32, you are right. FP16 means pure fp16 based flow including weights accumulation. NVCaffe does support this: https://github.com/NVIDIA/caffe/tree/models As of SSD itself, this might be a bug I just fixed here: https://github.com/drnikolaev/caffe Please give it a try. If it's still broken, please send me reproduction instructions. Thank you.

drnikolaev commented 5 years ago

Better fp16 is coming in new release, preview is available here: https://github.com/drnikolaev/caffe