search

tomgoldstein / loss-landscape

Code for visualizing the loss landscape of neural nets
MIT License
2.72k stars 388 forks source link

change type conversion code to fix bug on fp16 #27

Closed TobiasLee closed 4 years ago

TobiasLee commented 4 years ago

There is an implicit float16 -> float32 conversion in the original code if net weights are float16

p.data = w + torch.Tensor(d).type(type(w))

thus results in inaccurate loss computation. The following code can avoid this problem since it take the exact w.dtype when doing the conversion

p.data = w + torch.Tensor(d).type_as(w.dtype)
TobiasLee commented 4 years ago

need further test