Closed vollOlga closed 4 months ago
Nevermind, this is due to an issue with the GTX 1660 Graphics Card series not being able to handle 16 bit floating point precision: https://github.com/pytorch/pytorch/issues/58123
Loss calculation works as intended now when using 32 bit fp.
Note: The current training code causes the model to quickly produce
NaN
loss. I assume that this could be due to exploding gradients (lack of BatchNormalization?)