NaN losses when training with torch.nn.CrossEntropyLoss

ELEKTRONN / elektronn3

A PyTorch-based library for working with 3D and 2D convolutional neural networks, with focus on semantic segmentation of volumetric biomedical image data

MIT License

160 stars 27 forks source link

NaN losses when training with torch.nn.CrossEntropyLoss #32

Closed mdraw closed 5 years ago

mdraw commented 5 years ago

As mentioned in the commit message of https://github.com/ELEKTRONN/elektronn3/commit/e5a60057c8b08934f691bb3e43228fae853b0a3f, CrossEntropyLoss sometimes suddenly produces NaN values in trainings based on https://github.com/ELEKTRONN/elektronn3/blob/master/examples/train_unet_neurodata.py, even at low learning rates.