search

astorfi / 3D-convolutional-speaker-recognition

:speaker: Deep Learning & 3D Convolutional Neural Networks for Speaker Verification
Apache License 2.0
778 stars 275 forks source link

Nan output during training #24

Closed hamiGH closed 6 years ago

hamiGH commented 6 years ago

Hi @astorfi In the development step, i tried running train_softmax.py over VoxCeleb2 dataset and then i got nan output (logits variable in code). how to solve this?

astorfi commented 6 years ago

@hamiGH It is hard to say as I do not know the details of your implementation. My guess is the learning rate must be adjusted. Furthermore, please make sure the batch-normalization is active.