search

NVIDIA / sentiment-discovery

Unsupervised Language Modeling at scale for robust sentiment classification
Other
1.06k stars 202 forks source link

running pretrained models on CPU #52

Closed harsham05 closed 5 years ago

harsham05 commented 5 years ago

Do the fp32 pre-trained models default to fp16 when run on CPU? Is the loss of precision when running on the CPU significant such that its not recommended?

raulpuric commented 5 years ago

loading a saved state dict into a model will default to whatever type the model is declared as in your code. The loss of precision going from fp32->fp16 is insignifcant. However, fp16 math is really emulated fp32 math on cpu, so it's even slower than fp32. I would not recommend using fp16 on cpu.