search

szagoruyko / wide-residual-networks

3.8% and 18.3% on CIFAR-10 and CIFAR-100
http://arxiv.org/abs/1605.07146
BSD 2-Clause "Simplified" License
1.3k stars 293 forks source link

FP16 and FP32 #30

Closed shuzi closed 5 years ago

shuzi commented 7 years ago

Hi , would you please point me to the part of code that can switch between FP32 mode and FP16 mode ?

szagoruyko commented 7 years ago

@shuzi see https://github.com/szagoruyko/wide-residual-networks/tree/fp16 branch and set data_type=torch.CudaHalfTensor th train.lua to train in float16

shuzi commented 7 years ago

@szagoruyko

just curious, when you use FP16 on P100 card, the optimization still converges?

szagoruyko commented 7 years ago

@shuzi I don't have a P100 card.

szagoruyko commented 5 years ago

Models in pytorch hub were trained in FP16 https://pytorch.org/hub/pytorch_vision_wide_resnet/