Closed 008karan closed 4 years ago
When increased the chunking size from 200ms to 600ms training started correctly. Below 600ms its giving nan
and above 700ms range becoming zero in accessing random chunk as snt_len
become less than wlen
.
Can you tell why so @mravanelli
It looks like a gradient issue. Sometimes one can solve them just by adding gradient clipping.
I encountered the same problem, how did you solve it in the end?
I encountered this . See https://github.com/mravanelli/SincNet/issues/102
In training i am getting only nan loss. after 40 epoch also no update.
after debugging I found training batches getting generated as I printed their tensor. but when i am printing
pout
itsnan
output:
for 1st file its showing the tensor but after its
nan
. very weird behaviour. My dataset contain 10 audio files of 10sec length for each speaker. please help!