lunixbochs
commented
5 years ago I've been doing quite a bit of training for months using the librispeech conv glu recipe. You're using that one right, not the one from the tutorial?
My general methodology for training on extremely large datasets is:
- Make a minimal train set (which I've been calling a dev set) that is only ~10% of each train set.
- Train on only this dev set with itersave=true until my TER starts to converge - if the TER gets into the 70-80% range, immediately stop training (if I overshoot, copy the 70-80% iter save model back to 001_model_last.bin before continuing training).
- Add in the full training set, and continue training.
Note: I mix in all of my datasets from the very beginning, and just use the subset to make it start to converge earlier, as I've had issues converging quickly on very large datasets.
dmzubr
adamchant
Hey there! Thanks a lot to all contributors of w2l for the great ASR system! I would like to get some tips from other w2l developers. Hope for your exp and wisdom :)
Well, I have 3 sources of annotated audio data: 1) 90h corpus of good quality audio recorded at sound studio (audio-books); 2) 40h of audio records from different speakers recorded with headset microphones (and different mic models); 3) 30h of audio records created in public places (pharmacies) recorded on the stationary mic (Stelberry M-100). This data includes some bad quality audio chunks. I had listen for more then 3 times in headset to finally get it.
Using network architecture from LibriSpeech recipe.
Got the following results of training:
Problem appears when I try to train on corpus with 3) source of records (30h).
In total, I've tried those attempts with different combination of lr, momentum and gradnorm. Almost the same behavior and results every single time.
Machine config in short: Ubuntu 18.04, Geforce GTX 1080ti on 415.27 driver. W2l installation is based on Docker image. All w2l and flashlight tests passed successfully. Successful trainings (on 1) and 2 sources) were with the same hyper-parameters on this machine and Azure Tesla K80x4.
Motivation of using 3) src was to adapt model to specificity of acoustical environment of public place and mic model. So, taking this into consideration, would my approach be the best one in this case? What do you think? Any tips on that? By the way, cleaning and denoising of records have also been used.
Does these results generally mean that the third source of data just has not good enough quality to train on it at all? I supposed, even in this case there should be some minor improvements on TER from epoch to epoch, however not.
I would really appreciate any tips and your thoughts on that, guys. Any minor advice would help a lot!