Error while training RASR

flashlight / wav2letter

Facebook AI Research's Automatic Speech Recognition Toolkit

https://github.com/facebookresearch/wav2letter/wiki

Other

6.37k stars 1.01k forks source link

Error while training RASR #968

Open innarid opened 3 years ago

innarid commented 3 years ago

I got error(Floating point exception) while training 10 epoch. error.txt Log of 9 epoch looks good 009_log.txt I tried to change lr and lr decay, but it didn't help. Could you please help me to find the reason of this error? Thanks!

tlikhomanenko commented 3 years ago

Could you run without distributed training? Do you have all 1-9 epochs passed fine?

innarid commented 3 years ago

I got the same error with -enable_distributed=false. Yes, 9 epochs passed fine.

tlikhomanenko commented 3 years ago

Can you confirm that if you rerun training from epoch 1 it is still working?