tlikhomanenko
commented
4 years ago Hi @davidbelle
One thing we did in latest commits for transformer training pipeline support is adding warmup, whose default value is 8k updates. So non-transformer models this should be set to --warmup=0 (sorry for misleading things here, will fix them). You can see that in your training loop there is increasing lr value, not the value you set in the config. The second thing is that we trained with total batchsize 128 and all params optimized for it, so if you have total batch <=32, (even possibly for 64 too) - this could be very different.
I feel like it shouldn't say Epoch 1 the whole time but not sure? Training ran over night, about 12 hours total I think. I would have thought it would take longer. From what I've seen of someone else's post, the WER dropped significantly even after the 4th iteration.
Because you set --reportiters=2000 then it really printing every 2000 updates and if you train on 1 gpu (I suspect this) with batchsize=4 (the whole train has 280k samples) then you will really have ~70k updates before finishing 1 epoch.
For --iter you can set value = 500 280k / total_batchsize, where total_batchsize = --batchsize ngpus. So the main problem that you have very high WER is that you need to train longer.
From what I've seen of someone else's post, the WER dropped significantly even after the 4th iteration.
Here please check what was the total batch size (and again params for 128 total batch are not appropriate for batch 4).
davidbelle
Hello. I have a requirement to perform voice to text on a 8k audio and am not getting good error rates for it be useful. I just want to post here what I did step by an step and hopefully will get some answers.
I have come to understand that I need to create my own models with 8k datasets. I tried out the pre-trained models on sota and was pretty happy with the results (of course I could only test these using 16k audio), so I have gone down the path of creating 8k AM and LM using SOTA librespeech Resnet CTC.
I inspected _prepare_librispeech_wp_and_officiallexicon.py and noticed it was calling data/librispeech/prepare.py which was downloading the datasets, so I ran prepare.py by itself, ran a script to overwrite the downloaded flac's to 8k, modified the prepare_librispeech_wp_and_officiallexicon.py to not call prepare.py, and finally called prepare_librispeech_wp_and_officiallexicon.py.
So now I have the dataset as 8k, and all the bits and pieces needed to run train.
/root/wav2letter/build/Train train --flagsfile train.cfg --minloglevel=0 --logtostderr=1 Here's train.cfg:
Initially I had iter set to 500 as per the original file, when I ran it it output
I came across a post that mentioned that --iter has changed what it actually means and to set it to a much higher number. I don't know how high to go. So I'm guessing this could be the problem. (Here's the mentioned comment)
Here's the head and tail of my training output once it actually starts the epoch:
26 more and then
I feel like it shouldn't say Epoch 1 the whole time but not sure? Training ran over night, about 12 hours total I think. I would have thought it would take longer. From what I've seen of someone else's post, the WER dropped significantly even after the 4th iteration.
Running Test show's the error rate still very high.
Here's running the test
Here's running decode
Running it against my own 8k audio shows unfavourable results. So is it the --iter flag? What should I set this too? Or could it be something else?
Thanks
David