search

flashlight / wav2letter

Facebook AI Research's Automatic Speech Recognition Toolkit
https://github.com/facebookresearch/wav2letter/wiki
Other
6.39k stars 1.01k forks source link

[How to apply spec augmentation?] #854

Closed Trikaldarshi closed 4 years ago

Trikaldarshi commented 4 years ago

Question

I want to add spec augmentation while training to enhance WER

Additional Context

I am running experiments on one of the Indian languages. I have followed the tutorials available at tutorials for librispeech clean and I was able to reproduce the result for it as well as on my own dataset. Now, I want to apply some data augmentation techniques to improve the results. But, I could not find any documentation of SAUG and how to apply it. Could you please tell me where I need to add SAUG it? As mentioned https://github.com/facebookresearch/wav2letter/blob/master/recipes/streaming_convnets/librispeech/am_500ms_future_context.arch, adding the following two lines in https://github.com/facebookresearch/wav2letter/blob/v0.2/tutorials/1-librispeech_clean/network.arch#L4 will work?

SAUG 80 27 2 100 1.0 2 PD 0 5 3

Or Do I need to change something else too? Do I need to have any changes in the corresponding train.cfg and decode.cfg files available at https://github.com/facebookresearch/wav2letter/blob/v0.2/tutorials/1-librispeech_clean/ ?

tlikhomanenko commented 4 years ago

Better to use saug not as a layer in your arch file, but just setting these params https://github.com/facebookresearch/wav2letter/blob/v0.2/src/common/Defines.cpp#L89 and https://github.com/facebookresearch/wav2letter/blob/v0.2/src/common/Defines.cpp#L137-L144, you need to set them in train config. For decoding you don't need to do anything, it is only applied during training.

Trikaldarshi commented 4 years ago

Hi, I am attaching the training details of without specaug (where I have got good training and good WER on test set) and with specaug which is giving very vague behavior at the time of training. Any suggestions? Could you please suggest what will be the appropriate values for specaug given my training logs?

First I will show you logs of training with specaug. I have applied the flags as follows:

--saug_start_update=1 [is it a right choce?] --saug_fmaskf=27 --saug_fmaskn=2 --saug_tmaskt=100 --saug_tmaskp=1.0 --saug_tmaskn=2

train.cfg file

--datadir=/root/w2l_hindi/ --rundir=/root/w2l_hindi/ --archdir=/root/wav2letter/tutorials/1-librispeech_clean/ --train=lists_combined/train_list_combined_w2l.lst --valid=lists_combined/dev_list_combined_w2l.lst --input=wav --arch=network_10_DO.arch --tokens=/root/w2l_hindi/am_combined/Tokens_Combined.txt --lexicon=/root/w2l_hindi/am_combined/Lexicon_Combined.txt --criterion=ctc --lr=0.1 --maxgradnorm=1.0 --replabel=1 --surround=| --onorm=target --sqnorm=true --mfsc=true --filterbanks=40 --nthread=4 --batchsize=4 --runname=specaug_trainlogs --iter=2000000 --saug_start_update=1 --saug_fmaskf=27 --saug_fmaskn=2 --saug_tmaskt=100 --saug_tmaskp=1.0 --saug_tmaskn=2

Training Log

I1008 15:47:04.469067 1170 Train.cpp:59] Reading flags from file /root/wav2letter/tutorials/1-librispeech_clean/train.cfg I1008 15:47:04.469342 1170 Train.cpp:151] Gflags after parsing --flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --adambeta1=0.90000000000000002; --adambeta2=0.999; --am=; --am_decoder_tr_dropout=0; --am_decoder_tr_layerdrop=0; --am_decoder_tr_layers=1; --arch=network_10_DO.arch; --archdir=/root/wav2letter/tutorials/1-librispeech_clean/; --attention=content; --attentionthreshold=2147483647; --attnWindow=no; --attnconvchannel=0; --attnconvkernel=0; --attndim=0; --batchsize=4; --beamsize=2500; --beamsizetoken=250000; --beamthreshold=25; --blobdata=false; --channels=1; --criterion=ctc; --critoptim=sgd; --datadir=/root/w2l_hindi/; --dataorder=input; --decoderattnround=1; --decoderdropout=0; --decoderrnnlayer=1; --decodertype=wrd; --devwin=0; --emission_dir=; --emission_queue_size=3000; --enable_distributed=false; --encoderdim=0; --eosscore=0; --eostoken=false; --everstoredb=false; --fftcachesize=1; --filterbanks=40; --flagsfile=/root/wav2letter/tutorials/1-librispeech_clean/train.cfg; --framesizems=25; --framestridems=10; --gamma=1; --gumbeltemperature=1; --input=wav; --inputbinsize=100; --inputfeeding=false; --isbeamdump=false; --iter=2000000; --itersave=false; --labelsmooth=0; --leftWindowSize=50; --lexicon=/root/w2l_hindi/am_combined/Lexicon_Combined.txt; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lm_memory=5000; --lm_vocab=; --lmtype=kenlm; --lmweight=0; --localnrmlleftctx=0; --localnrmlrightctx=0; --logadd=false; --lr=0.10000000000000001; --lr_decay=9223372036854775807; --lr_decay_step=9223372036854775807; --lrcosine=false; --lrcrit=0; --max_devices_per_node=8; --maxdecoderoutputlen=200; --maxgradnorm=1; --maxisz=9223372036854775807; --maxload=-1; --maxrate=10; --maxsil=50; --maxtsz=9223372036854775807; --maxword=-1; --melfloor=1; --memstepsize=10485760; --mfcc=false; --mfcccoeffs=13; --mfsc=true; --minisz=0; --minrate=3; --minsil=0; --mintsz=0; --momentum=0; --netoptim=sgd; --noresample=false; --nthread=4; --nthread_decoder=1; --nthread_decoder_am_forward=1; --numattnhead=8; --onorm=target; --optimepsilon=1e-08; --optimrho=0.90000000000000002; --outputbinsize=5; --pctteacherforcing=100; --pcttraineval=100; --pow=false; --pretrainWindow=0; --replabel=1; --reportiters=0; --rightWindowSize=50; --rndv_filepath=; --rundir=/root/w2l_hindi/; --runname=specaug_trainlogs; --samplerate=16000; --sampletarget=0; --samplingstrategy=rand; --saug_fmaskf=27; --saug_fmaskn=2; --saug_start_update=1; --saug_tmaskn=2; --saug_tmaskp=1; --saug_tmaskt=100; --sclite=; --seed=0; --show=false; --showletters=false; --silscore=0; --smearing=none; --smoothingtemperature=1; --softwoffset=10; --softwrate=5; --softwstd=5; --sqnorm=true; --stepsize=9223372036854775807; --surround=|; --tag=; --target=tkn; --test=; --tokens=/root/w2l_hindi/am_combined/Tokens_Combined.txt; --tokensdir=; --train=lists_combined/train_list_combined_w2l.lst; --trainWithWindow=false; --transdiag=0; --unkscore=-inf; --use_memcache=false; --uselexicon=true; --usewordpiece=false; --valid=lists_combined/dev_list_combined_w2l.lst; --validbatchsize=-1; --warmup=1; --weightdecay=0; --wordscore=0; --wordseparator=|; --world_rank=0; --world_size=1; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=; I1008 15:47:04.469882 1170 Train.cpp:152] Experiment path: /root/w2l_hindi/specaug_trainlogs I1008 15:47:04.469888 1170 Train.cpp:153] Experiment runidx: 1 I1008 15:47:04.572295 1170 Train.cpp:199] Number of classes (network): 85 I1008 15:47:04.768990 1170 Train.cpp:206] Number of words: 74713 I1008 15:47:04.829069 1170 Train.cpp:220] Loading architecture file from /root/wav2letter/tutorials/1-librispeech_clean/network_10_DO.arch I1008 15:47:05.615020 1170 Train.cpp:252] [Network] Sequential [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> output] (0): View (-1 1 40 0) (1): Conv2D (40->256, 8x1, 2,1, SAME,SAME, 1, 1) (with bias) (2): ReLU (3): Dropout (0.200000) (4): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (5): ReLU (6): Dropout (0.200000) (7): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (8): ReLU (9): Dropout (0.200000) (10): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (11): ReLU (12): Dropout (0.200000) (13): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (14): ReLU (15): Dropout (0.200000) (16): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (17): ReLU (18): Dropout (0.200000) (19): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (20): ReLU (21): Dropout (0.200000) (22): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (23): ReLU (24): Dropout (0.200000) (25): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (26): ReLU (27): Dropout (0.200000) (28): Reorder (2,0,3,1) (29): Linear (256->512) (with bias) (30): ReLU (31): Linear (512->85) (with bias) I1008 15:47:05.615118 1170 Train.cpp:253] [Network Params: 4453717] I1008 15:47:05.615128 1170 Train.cpp:254] [Criterion] ConnectionistTemporalClassificationCriterion I1008 15:47:05.615136 1170 Train.cpp:262] [Network Optimizer] SGD I1008 15:47:05.615141 1170 Train.cpp:263] [Criterion Optimizer] SGD I1008 15:47:07.312996 1170 W2lListFilesDataset.cpp:141] 152024 files found. I1008 15:47:07.315565 1170 Utils.cpp:102] Filtered 0/152024 samples I1008 15:47:07.333814 1170 W2lListFilesDataset.cpp:62] Total batches (i.e. iters): 38006 I1008 15:47:07.926349 1170 W2lListFilesDataset.cpp:141] 52081 files found. I1008 15:47:07.927153 1170 Utils.cpp:102] Filtered 0/52081 samples I1008 15:47:07.931666 1170 W2lListFilesDataset.cpp:62] Total batches (i.e. iters): 13021 I1008 15:47:07.936139 1170 Train.cpp:568] Shuffling trainset I1008 15:47:07.942173 1170 Train.cpp:575] Epoch 1 started! I1008 16:38:21.139962 1170 Train.cpp:345] epoch: 1 | nupdates: 38006 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:43:25 | bch(ms): 68.55 | smp(ms): 53.45 | fwd(ms): 7.89 | crit-fwd(ms): 5.26 | bwd(ms): 4.39 | optim(ms): 1.97 | loss: 21.22972 | train-TER: 84.31 | train-WER: 102.92 | lists_combined/dev_list_combined_w2l.lst-loss: 28.11373 | lists_combined/dev_list_combined_w2l.lst-TER: 93.44 | lists_combined/dev_list_combined_w2l.lst-WER: 97.05 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 319.14 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1008 16:38:21.286517 1170 Train.cpp:568] Shuffling trainset I1008 16:38:21.293823 1170 Train.cpp:575] Epoch 2 started! I1008 17:07:18.428382 1170 Train.cpp:345] epoch: 2 | nupdates: 76012 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:22 | bch(ms): 33.73 | smp(ms): 20.35 | fwd(ms): 6.78 | crit-fwd(ms): 4.54 | bwd(ms): 3.85 | optim(ms): 1.89 | loss: 20.09806 | train-TER: 74.56 | train-WER: 112.30 | lists_combined/dev_list_combined_w2l.lst-loss: 26.26565 | lists_combined/dev_list_combined_w2l.lst-TER: 93.93 | lists_combined/dev_list_combined_w2l.lst-WER: 97.34 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 648.52 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1008 17:07:18.561858 1170 Train.cpp:568] Shuffling trainset I1008 17:07:18.569450 1170 Train.cpp:575] Epoch 3 started! I1008 17:36:24.552183 1170 Train.cpp:345] epoch: 3 | nupdates: 114018 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:32 | bch(ms): 34.02 | smp(ms): 20.45 | fwd(ms): 6.83 | crit-fwd(ms): 4.57 | bwd(ms): 3.89 | optim(ms): 1.97 | loss: 19.83859 | train-TER: 73.24 | train-WER: 113.71 | lists_combined/dev_list_combined_w2l.lst-loss: 28.63334 | lists_combined/dev_list_combined_w2l.lst-TER: 94.43 | lists_combined/dev_list_combined_w2l.lst-WER: 97.70 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 643.08 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1008 17:36:24.605060 1170 Train.cpp:568] Shuffling trainset I1008 17:36:24.612459 1170 Train.cpp:575] Epoch 4 started! I1008 18:05:14.546697 1170 Train.cpp:345] epoch: 4 | nupdates: 152024 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:24 | bch(ms): 33.80 | smp(ms): 20.32 | fwd(ms): 6.78 | crit-fwd(ms): 4.54 | bwd(ms): 3.87 | optim(ms): 1.95 | loss: 19.47403 | train-TER: 73.33 | train-WER: 107.99 | lists_combined/dev_list_combined_w2l.lst-loss: 17.42173 | lists_combined/dev_list_combined_w2l.lst-TER: 67.05 | lists_combined/dev_list_combined_w2l.lst-WER: 98.13 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 647.27 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1008 18:05:14.686115 1170 Train.cpp:568] Shuffling trainset I1008 18:05:14.693823 1170 Train.cpp:575] Epoch 5 started! I1008 18:34:08.970484 1170 Train.cpp:345] epoch: 5 | nupdates: 190030 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:21 | bch(ms): 33.72 | smp(ms): 20.30 | fwd(ms): 6.75 | crit-fwd(ms): 4.52 | bwd(ms): 3.84 | optim(ms): 1.91 | loss: 17.70303 | train-TER: 69.77 | train-WER: 100.14 | lists_combined/dev_list_combined_w2l.lst-loss: 14.18305 | lists_combined/dev_list_combined_w2l.lst-TER: 52.97 | lists_combined/dev_list_combined_w2l.lst-WER: 97.24 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 648.77 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):20 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1008 18:34:09.108045 1170 Train.cpp:568] Shuffling trainset I1008 18:34:09.115639 1170 Train.cpp:575] Epoch 6 started! I1008 19:03:47.994777 1170 Train.cpp:345] epoch: 6 | nupdates: 228036 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:22:02 | bch(ms): 34.79 | smp(ms): 21.27 | fwd(ms): 6.81 | crit-fwd(ms): 4.57 | bwd(ms): 3.87 | optim(ms): 1.90 | loss: 16.53907 | train-TER: 64.83 | train-WER: 99.50 | lists_combined/dev_list_combined_w2l.lst-loss: 12.60295 | lists_combined/dev_list_combined_w2l.lst-TER: 46.43 | lists_combined/dev_list_combined_w2l.lst-WER: 90.73 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 628.82 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1008 19:03:48.161686 1170 Train.cpp:568] Shuffling trainset I1008 19:03:48.169091 1170 Train.cpp:575] Epoch 7 started! I1008 19:32:59.534289 1170 Train.cpp:345] epoch: 7 | nupdates: 266042 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:36 | bch(ms): 34.10 | smp(ms): 20.56 | fwd(ms): 6.79 | crit-fwd(ms): 4.54 | bwd(ms): 3.87 | optim(ms): 1.93 | loss: 15.87872 | train-TER: 62.26 | train-WER: 98.25 | lists_combined/dev_list_combined_w2l.lst-loss: 11.66328 | lists_combined/dev_list_combined_w2l.lst-TER: 42.92 | lists_combined/dev_list_combined_w2l.lst-WER: 86.97 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 641.49 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1008 19:32:59.694579 1170 Train.cpp:568] Shuffling trainset I1008 19:32:59.701900 1170 Train.cpp:575] Epoch 8 started! I1008 20:01:48.330876 1170 Train.cpp:345] epoch: 8 | nupdates: 304048 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:17 | bch(ms): 33.62 | smp(ms): 20.13 | fwd(ms): 6.76 | crit-fwd(ms): 4.53 | bwd(ms): 3.85 | optim(ms): 1.92 | loss: 15.42177 | train-TER: 60.45 | train-WER: 97.29 | lists_combined/dev_list_combined_w2l.lst-loss: 11.32801 | lists_combined/dev_list_combined_w2l.lst-TER: 40.41 | lists_combined/dev_list_combined_w2l.lst-WER: 86.26 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.62 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1008 20:01:48.495100 1170 Train.cpp:568] Shuffling trainset I1008 20:01:48.502554 1170 Train.cpp:575] Epoch 9 started! I1008 20:30:32.512993 1170 Train.cpp:345] epoch: 9 | nupdates: 342054 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:11 | bch(ms): 33.44 | smp(ms): 19.99 | fwd(ms): 6.75 | crit-fwd(ms): 4.52 | bwd(ms): 3.84 | optim(ms): 1.90 | loss: 15.09321 | train-TER: 59.15 | train-WER: 96.39 | lists_combined/dev_list_combined_w2l.lst-loss: 10.83497 | lists_combined/dev_list_combined_w2l.lst-TER: 38.55 | lists_combined/dev_list_combined_w2l.lst-WER: 83.02 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 654.15 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1008 20:30:32.674388 1170 Train.cpp:568] Shuffling trainset I1008 20:30:32.680675 1170 Train.cpp:575] Epoch 10 started! I1008 20:59:11.452595 1170 Train.cpp:345] epoch: 10 | nupdates: 380060 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:14 | bch(ms): 33.53 | smp(ms): 20.04 | fwd(ms): 6.76 | crit-fwd(ms): 4.53 | bwd(ms): 3.85 | optim(ms): 1.92 | loss: 14.83441 | train-TER: 58.20 | train-WER: 95.48 | lists_combined/dev_list_combined_w2l.lst-loss: 10.52331 | lists_combined/dev_list_combined_w2l.lst-TER: 37.42 | lists_combined/dev_list_combined_w2l.lst-WER: 80.21 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.50 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1008 20:59:11.620302 1170 Train.cpp:568] Shuffling trainset I1008 20:59:11.627919 1170 Train.cpp:575] Epoch 11 started! I1008 21:27:56.649600 1170 Train.cpp:345] epoch: 11 | nupdates: 418066 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:15 | bch(ms): 33.55 | smp(ms): 20.07 | fwd(ms): 6.75 | crit-fwd(ms): 4.52 | bwd(ms): 3.85 | optim(ms): 1.92 | loss: 14.62784 | train-TER: 57.37 | train-WER: 94.88 | lists_combined/dev_list_combined_w2l.lst-loss: 10.25934 | lists_combined/dev_list_combined_w2l.lst-TER: 36.26 | lists_combined/dev_list_combined_w2l.lst-WER: 79.37 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.10 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1008 21:27:56.816691 1170 Train.cpp:568] Shuffling trainset I1008 21:27:56.824138 1170 Train.cpp:575] Epoch 12 started! I1008 21:56:40.461575 1170 Train.cpp:345] epoch: 12 | nupdates: 456072 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:12 | bch(ms): 33.48 | smp(ms): 19.99 | fwd(ms): 6.77 | crit-fwd(ms): 4.54 | bwd(ms): 3.85 | optim(ms): 1.90 | loss: 14.48949 | train-TER: 56.80 | train-WER: 94.43 | lists_combined/dev_list_combined_w2l.lst-loss: 10.03393 | lists_combined/dev_list_combined_w2l.lst-TER: 35.36 | lists_combined/dev_list_combined_w2l.lst-WER: 77.02 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 653.50 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1008 21:56:40.627419 1170 Train.cpp:568] Shuffling trainset I1008 21:56:40.635102 1170 Train.cpp:575] Epoch 13 started! I1008 22:25:31.074088 1170 Train.cpp:345] epoch: 13 | nupdates: 494078 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:18 | bch(ms): 33.65 | smp(ms): 20.09 | fwd(ms): 6.81 | crit-fwd(ms): 4.56 | bwd(ms): 3.87 | optim(ms): 1.92 | loss: 14.30224 | train-TER: 56.03 | train-WER: 93.80 | lists_combined/dev_list_combined_w2l.lst-loss: 9.79159 | lists_combined/dev_list_combined_w2l.lst-TER: 33.80 | lists_combined/dev_list_combined_w2l.lst-WER: 75.54 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.07 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):30 Total native mallocs:44 Total native frees:0 I1008 22:25:31.229833 1170 Train.cpp:568] Shuffling trainset I1008 22:25:31.235770 1170 Train.cpp:575] Epoch 14 started! I1008 22:54:18.970520 1170 Train.cpp:345] epoch: 14 | nupdates: 532084 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:16 | bch(ms): 33.58 | smp(ms): 20.06 | fwd(ms): 6.77 | crit-fwd(ms): 4.53 | bwd(ms): 3.86 | optim(ms): 1.93 | loss: 14.15631 | train-TER: 55.45 | train-WER: 93.20 | lists_combined/dev_list_combined_w2l.lst-loss: 9.61882 | lists_combined/dev_list_combined_w2l.lst-TER: 33.11 | lists_combined/dev_list_combined_w2l.lst-WER: 73.90 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 651.40 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1008 22:54:19.047928 1170 Train.cpp:568] Shuffling trainset I1008 22:54:19.055444 1170 Train.cpp:575] Epoch 15 started! I1008 23:23:05.589893 1170 Train.cpp:345] epoch: 15 | nupdates: 570090 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:13 | bch(ms): 33.50 | smp(ms): 20.03 | fwd(ms): 6.76 | crit-fwd(ms): 4.53 | bwd(ms): 3.85 | optim(ms): 1.90 | loss: 14.04025 | train-TER: 55.00 | train-WER: 92.63 | lists_combined/dev_list_combined_w2l.lst-loss: 9.57677 | lists_combined/dev_list_combined_w2l.lst-TER: 32.55 | lists_combined/dev_list_combined_w2l.lst-WER: 73.13 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.99 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1008 23:23:05.755776 1170 Train.cpp:568] Shuffling trainset I1008 23:23:05.763499 1170 Train.cpp:575] Epoch 16 started! I1008 23:51:53.044519 1170 Train.cpp:345] epoch: 16 | nupdates: 608096 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:15 | bch(ms): 33.55 | smp(ms): 20.07 | fwd(ms): 6.75 | crit-fwd(ms): 4.52 | bwd(ms): 3.85 | optim(ms): 1.92 | loss: 13.92841 | train-TER: 54.56 | train-WER: 92.14 | lists_combined/dev_list_combined_w2l.lst-loss: 9.34211 | lists_combined/dev_list_combined_w2l.lst-TER: 32.06 | lists_combined/dev_list_combined_w2l.lst-WER: 72.67 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 651.99 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1008 23:51:53.202988 1170 Train.cpp:568] Shuffling trainset I1008 23:51:53.210518 1170 Train.cpp:575] Epoch 17 started! I1009 00:26:09.890830 1170 Train.cpp:345] epoch: 17 | nupdates: 646102 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:26:44 | bch(ms): 42.21 | smp(ms): 28.42 | fwd(ms): 6.94 | crit-fwd(ms): 4.66 | bwd(ms): 3.94 | optim(ms): 1.93 | loss: 13.83526 | train-TER: 54.21 | train-WER: 91.72 | lists_combined/dev_list_combined_w2l.lst-loss: 9.29726 | lists_combined/dev_list_combined_w2l.lst-TER: 31.49 | lists_combined/dev_list_combined_w2l.lst-WER: 72.11 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 518.23 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):21 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1009 00:26:10.058614 1170 Train.cpp:568] Shuffling trainset I1009 00:26:10.065481 1170 Train.cpp:575] Epoch 18 started! I1009 00:54:54.606426 1170 Train.cpp:345] epoch: 18 | nupdates: 684108 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:15 | bch(ms): 33.55 | smp(ms): 20.01 | fwd(ms): 6.76 | crit-fwd(ms): 4.53 | bwd(ms): 3.86 | optim(ms): 1.94 | loss: 13.73630 | train-TER: 53.79 | train-WER: 91.57 | lists_combined/dev_list_combined_w2l.lst-loss: 9.27377 | lists_combined/dev_list_combined_w2l.lst-TER: 30.88 | lists_combined/dev_list_combined_w2l.lst-WER: 70.26 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.09 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1009 00:54:54.765398 1170 Train.cpp:568] Shuffling trainset I1009 00:54:54.771952 1170 Train.cpp:575] Epoch 19 started! I1009 01:23:46.635946 1170 Train.cpp:345] epoch: 19 | nupdates: 722114 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:14 | bch(ms): 33.55 | smp(ms): 20.01 | fwd(ms): 6.77 | crit-fwd(ms): 4.53 | bwd(ms): 3.86 | optim(ms): 1.93 | loss: 13.66764 | train-TER: 53.48 | train-WER: 91.28 | lists_combined/dev_list_combined_w2l.lst-loss: 9.23150 | lists_combined/dev_list_combined_w2l.lst-TER: 30.61 | lists_combined/dev_list_combined_w2l.lst-WER: 70.62 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.14 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1009 01:23:46.784828 1170 Train.cpp:568] Shuffling trainset I1009 01:23:46.792307 1170 Train.cpp:575] Epoch 20 started! I1009 01:52:30.115533 1170 Train.cpp:345] epoch: 20 | nupdates: 760120 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:13 | bch(ms): 33.51 | smp(ms): 19.98 | fwd(ms): 6.77 | crit-fwd(ms): 4.53 | bwd(ms): 3.86 | optim(ms): 1.93 | loss: 13.59509 | train-TER: 53.23 | train-WER: 90.88 | lists_combined/dev_list_combined_w2l.lst-loss: 9.13226 | lists_combined/dev_list_combined_w2l.lst-TER: 30.41 | lists_combined/dev_list_combined_w2l.lst-WER: 69.74 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.83 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):27 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1009 01:52:30.199321 1170 Train.cpp:568] Shuffling trainset I1009 01:52:30.206797 1170 Train.cpp:575] Epoch 21 started! I1009 02:24:03.071964 1170 Train.cpp:345] epoch: 21 | nupdates: 798126 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:24:01 | bch(ms): 37.93 | smp(ms): 24.20 | fwd(ms): 6.91 | crit-fwd(ms): 4.63 | bwd(ms): 3.93 | optim(ms): 1.92 | loss: 13.52776 | train-TER: 53.01 | train-WER: 90.58 | lists_combined/dev_list_combined_w2l.lst-loss: 9.10866 | lists_combined/dev_list_combined_w2l.lst-TER: 30.09 | lists_combined/dev_list_combined_w2l.lst-WER: 68.78 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 576.81 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1009 02:24:03.247377 1170 Train.cpp:568] Shuffling trainset I1009 02:24:03.253739 1170 Train.cpp:575] Epoch 22 started! I1009 02:52:42.791410 1170 Train.cpp:345] epoch: 22 | nupdates: 836132 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:14 | bch(ms): 33.53 | smp(ms): 19.98 | fwd(ms): 6.77 | crit-fwd(ms): 4.53 | bwd(ms): 3.86 | optim(ms): 1.94 | loss: 13.43820 | train-TER: 52.58 | train-WER: 90.32 | lists_combined/dev_list_combined_w2l.lst-loss: 8.97386 | lists_combined/dev_list_combined_w2l.lst-TER: 29.55 | lists_combined/dev_list_combined_w2l.lst-WER: 68.13 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.48 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1009 02:52:42.959300 1170 Train.cpp:568] Shuffling trainset I1009 02:52:42.966838 1170 Train.cpp:575] Epoch 23 started! I1009 03:21:26.115923 1170 Train.cpp:345] epoch: 23 | nupdates: 874138 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:14 | bch(ms): 33.54 | smp(ms): 20.02 | fwd(ms): 6.77 | crit-fwd(ms): 4.54 | bwd(ms): 3.86 | optim(ms): 1.91 | loss: 13.41262 | train-TER: 52.44 | train-WER: 90.26 | lists_combined/dev_list_combined_w2l.lst-loss: 8.82082 | lists_combined/dev_list_combined_w2l.lst-TER: 29.47 | lists_combined/dev_list_combined_w2l.lst-WER: 67.52 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.33 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1009 03:21:26.293243 1170 Train.cpp:568] Shuffling trainset I1009 03:21:26.300902 1170 Train.cpp:575] Epoch 24 started! I1009 03:50:18.183362 1170 Train.cpp:345] epoch: 24 | nupdates: 912144 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:20 | bch(ms): 33.68 | smp(ms): 20.15 | fwd(ms): 6.76 | crit-fwd(ms): 4.52 | bwd(ms): 3.86 | optim(ms): 1.94 | loss: 13.35399 | train-TER: 52.23 | train-WER: 89.94 | lists_combined/dev_list_combined_w2l.lst-loss: 8.80583 | lists_combined/dev_list_combined_w2l.lst-TER: 29.05 | lists_combined/dev_list_combined_w2l.lst-WER: 67.16 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 649.45 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1009 03:50:18.361464 1170 Train.cpp:568] Shuffling trainset I1009 03:50:18.369208 1170 Train.cpp:575] Epoch 25 started! I1009 04:19:02.349629 1170 Train.cpp:345] epoch: 25 | nupdates: 950150 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:16 | bch(ms): 33.60 | smp(ms): 20.09 | fwd(ms): 6.76 | crit-fwd(ms): 4.53 | bwd(ms): 3.85 | optim(ms): 1.92 | loss: 13.32987 | train-TER: 52.09 | train-WER: 89.82 | lists_combined/dev_list_combined_w2l.lst-loss: 8.73803 | lists_combined/dev_list_combined_w2l.lst-TER: 28.69 | lists_combined/dev_list_combined_w2l.lst-WER: 66.02 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 651.13 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1009 04:19:02.529381 1170 Train.cpp:568] Shuffling trainset I1009 04:19:02.537055 1170 Train.cpp:575] Epoch 26 started! I1009 04:47:48.077939 1170 Train.cpp:345] epoch: 26 | nupdates: 988156 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:14 | bch(ms): 33.53 | smp(ms): 20.05 | fwd(ms): 6.75 | crit-fwd(ms): 4.52 | bwd(ms): 3.85 | optim(ms): 1.91 | loss: 13.25028 | train-TER: 51.82 | train-WER: 89.52 | lists_combined/dev_list_combined_w2l.lst-loss: 8.64930 | lists_combined/dev_list_combined_w2l.lst-TER: 28.39 | lists_combined/dev_list_combined_w2l.lst-WER: 65.44 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.40 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1009 04:47:48.256067 1170 Train.cpp:568] Shuffling trainset I1009 04:47:48.263667 1170 Train.cpp:575] Epoch 27 started! I1009 05:16:20.511123 1170 Train.cpp:345] epoch: 27 | nupdates: 1026162 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:01 | bch(ms): 33.20 | smp(ms): 19.81 | fwd(ms): 6.72 | crit-fwd(ms): 4.50 | bwd(ms): 3.82 | optim(ms): 1.88 | loss: 13.20747 | train-TER: 51.61 | train-WER: 89.49 | lists_combined/dev_list_combined_w2l.lst-loss: 8.78561 | lists_combined/dev_list_combined_w2l.lst-TER: 28.59 | lists_combined/dev_list_combined_w2l.lst-WER: 66.68 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 658.86 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1009 05:16:20.670861 1170 Train.cpp:568] Shuffling trainset I1009 05:16:20.676733 1170 Train.cpp:575] Epoch 28 started! I1009 05:45:07.020911 1170 Train.cpp:345] epoch: 28 | nupdates: 1064168 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:13 | bch(ms): 33.50 | smp(ms): 20.04 | fwd(ms): 6.75 | crit-fwd(ms): 4.52 | bwd(ms): 3.84 | optim(ms): 1.90 | loss: 13.17557 | train-TER: 51.52 | train-WER: 89.27 | lists_combined/dev_list_combined_w2l.lst-loss: 8.62234 | lists_combined/dev_list_combined_w2l.lst-TER: 27.96 | lists_combined/dev_list_combined_w2l.lst-WER: 65.27 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.98 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1009 05:45:07.198626 1170 Train.cpp:568] Shuffling trainset I1009 05:45:07.206357 1170 Train.cpp:575] Epoch 29 started!

It seems training is very bad.

Now I will show you the normal training where I got good results.

Normal Training log (without specaug)

train.cfg file:

--datadir=/root/w2l_hindi/ --rundir=/root/w2l_hindi/ --archdir=/root/wav2letter/tutorials/1-librispeech_clean/ --train=lists_combined/train_list_combined_w2l.lst --valid=lists_combined/dev_list_combined_w2l.lst --input=wav --arch=network_10_DO.arch --tokens=/root/w2l_hindi/am_combined/Tokens_Combined.txt --lexicon=/root/w2l_hindi/am_combined/Lexicon_Combined.txt --criterion=ctc --lr=0.1 --maxgradnorm=1.0 --replabel=1 --surround=| --onorm=target --sqnorm=true --mfsc=true --filterbanks=40 --nthread=4 --batchsize=4 --runname=specaug_trainlogs --iter=2000000

Training Log

I1004 16:45:46.017194 836 Train.cpp:59] Reading flags from file /root/wav2letter/tutorials/1-librispeech_clean/train.cfg I1004 16:45:46.017727 836 Train.cpp:151] Gflags after parsing --flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --adambeta1=0.90000000000000002; --adambeta2=0.999; --am=; --am_decoder_tr_dropout=0; --am_decoder_tr_layerdrop=0; --am_decoder_tr_layers=1; --arch=network_10_DO.arch; --archdir=/root/wav2letter/tutorials/1-librispeech_clean/; --attention=content; --attentionthreshold=2147483647; --attnWindow=no; --attnconvchannel=0; --attnconvkernel=0; --attndim=0; --batchsize=4; --beamsize=2500; --beamsizetoken=250000; --beamthreshold=25; --blobdata=false; --channels=1; --criterion=ctc; --critoptim=sgd; --datadir=/root/w2l_hindi/; --dataorder=input; --decoderattnround=1; --decoderdropout=0; --decoderrnnlayer=1; --decodertype=wrd; --devwin=0; --emission_dir=; --emission_queue_size=3000; --enable_distributed=false; --encoderdim=0; --eosscore=0; --eostoken=false; --everstoredb=false; --fftcachesize=1; --filterbanks=40; --flagsfile=/root/wav2letter/tutorials/1-librispeech_clean/train.cfg; --framesizems=25; --framestridems=10; --gamma=1; --gumbeltemperature=1; --input=wav; --inputbinsize=100; --inputfeeding=false; --isbeamdump=false; --iter=2000000; --itersave=false; --labelsmooth=0; --leftWindowSize=50; --lexicon=/root/w2l_hindi/am_combined/Lexicon_Combined.txt; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lm_memory=5000; --lm_vocab=; --lmtype=kenlm; --lmweight=0; --localnrmlleftctx=0; --localnrmlrightctx=0; --logadd=false; --lr=0.10000000000000001; --lr_decay=9223372036854775807; --lr_decay_step=9223372036854775807; --lrcosine=false; --lrcrit=0; --max_devices_per_node=8; --maxdecoderoutputlen=200; --maxgradnorm=1; --maxisz=9223372036854775807; --maxload=-1; --maxrate=10; --maxsil=50; --maxtsz=9223372036854775807; --maxword=-1; --melfloor=1; --memstepsize=10485760; --mfcc=false; --mfcccoeffs=13; --mfsc=true; --minisz=0; --minrate=3; --minsil=0; --mintsz=0; --momentum=0; --netoptim=sgd; --noresample=false; --nthread=4; --nthread_decoder=1; --nthread_decoder_am_forward=1; --numattnhead=8; --onorm=target; --optimepsilon=1e-08; --optimrho=0.90000000000000002; --outputbinsize=5; --pctteacherforcing=100; --pcttraineval=100; --pow=false; --pretrainWindow=0; --replabel=1; --reportiters=0; --rightWindowSize=50; --rndv_filepath=; --rundir=/root/w2l_hindi/; --runname=dropout_trainlogs; --samplerate=16000; --sampletarget=0; --samplingstrategy=rand; --saug_fmaskf=27; --saug_fmaskn=2; --saug_start_update=-1; --saug_tmaskn=2; --saug_tmaskp=1; --saug_tmaskt=100; --sclite=; --seed=0; --show=false; --showletters=false; --silscore=0; --smearing=none; --smoothingtemperature=1; --softwoffset=10; --softwrate=5; --softwstd=5; --sqnorm=true; --stepsize=9223372036854775807; --surround=|; --tag=; --target=tkn; --test=; --tokens=/root/w2l_hindi/am_combined/Tokens_Combined.txt; --tokensdir=; --train=lists_combined/train_list_combined_w2l.lst; --trainWithWindow=false; --transdiag=0; --unkscore=-inf; --use_memcache=false; --uselexicon=true; --usewordpiece=false; --valid=lists_combined/dev_list_combined_w2l.lst; --validbatchsize=-1; --warmup=1; --weightdecay=0; --wordscore=0; --wordseparator=|; --world_rank=0; --world_size=1; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=; I1004 16:45:46.018420 836 Train.cpp:152] Experiment path: /root/w2l_hindi/dropout_trainlogs I1004 16:45:46.018426 836 Train.cpp:153] Experiment runidx: 1 I1004 16:45:46.140514 836 Train.cpp:199] Number of classes (network): 85 I1004 16:45:46.507843 836 Train.cpp:206] Number of words: 74713 I1004 16:45:46.637246 836 Train.cpp:220] Loading architecture file from /root/wav2letter/tutorials/1-librispeech_clean/network_10_DO.arch I1004 16:45:48.077551 836 Train.cpp:252] [Network] Sequential [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> output] (0): View (-1 1 40 0) (1): Conv2D (40->256, 8x1, 2,1, SAME,SAME, 1, 1) (with bias) (2): ReLU (3): Dropout (0.200000) (4): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (5): ReLU (6): Dropout (0.200000) (7): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (8): ReLU (9): Dropout (0.200000) (10): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (11): ReLU (12): Dropout (0.200000) (13): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (14): ReLU (15): Dropout (0.200000) (16): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (17): ReLU (18): Dropout (0.200000) (19): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (20): ReLU (21): Dropout (0.200000) (22): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (23): ReLU (24): Dropout (0.200000) (25): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias) (26): ReLU (27): Dropout (0.200000) (28): Reorder (2,0,3,1) (29): Linear (256->512) (with bias) (30): ReLU (31): Linear (512->85) (with bias) I1004 16:45:48.077704 836 Train.cpp:253] [Network Params: 4453717] I1004 16:45:48.077721 836 Train.cpp:254] [Criterion] ConnectionistTemporalClassificationCriterion I1004 16:45:48.077747 836 Train.cpp:262] [Network Optimizer] SGD I1004 16:45:48.077757 836 Train.cpp:263] [Criterion Optimizer] SGD I1004 16:45:50.910687 836 W2lListFilesDataset.cpp:141] 152024 files found. I1004 16:45:50.916275 836 Utils.cpp:102] Filtered 0/152024 samples I1004 16:45:50.944430 836 W2lListFilesDataset.cpp:62] Total batches (i.e. iters): 38006 I1004 16:45:52.142277 836 W2lListFilesDataset.cpp:141] 52081 files found. I1004 16:45:52.144286 836 Utils.cpp:102] Filtered 0/52081 samples I1004 16:45:52.152056 836 W2lListFilesDataset.cpp:62] Total batches (i.e. iters): 13021 I1004 16:45:52.158681 836 Train.cpp:568] Shuffling trainset I1004 16:45:52.168671 836 Train.cpp:575] Epoch 1 started! I1004 17:19:21.028213 836 Train.cpp:345] epoch: 1 | nupdates: 38006 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:24:32 | bch(ms): 38.74 | smp(ms): 22.85 | fwd(ms): 7.69 | crit-fwd(ms): 5.14 | bwd(ms): 4.56 | optim(ms): 2.39 | loss: 14.69201 | train-TER: 61.75 | train-WER: 95.69 | lists_combined/dev_list_combined_w2l.lst-loss: 11.59161 | lists_combined/dev_list_combined_w2l.lst-TER: 42.64 | lists_combined/dev_list_combined_w2l.lst-WER: 86.85 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 564.71 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1004 17:19:21.365993 836 Train.cpp:568] Shuffling trainset I1004 17:19:21.377430 836 Train.cpp:575] Epoch 2 started! I1004 17:53:52.632336 836 Train.cpp:345] epoch: 2 | nupdates: 76012 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:26:10 | bch(ms): 41.33 | smp(ms): 24.64 | fwd(ms): 8.33 | crit-fwd(ms): 4.99 | bwd(ms): 4.45 | optim(ms): 2.14 | loss: 9.52196 | train-TER: 39.86 | train-WER: 84.02 | lists_combined/dev_list_combined_w2l.lst-loss: 9.76754 | lists_combined/dev_list_combined_w2l.lst-TER: 33.55 | lists_combined/dev_list_combined_w2l.lst-WER: 76.51 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 529.35 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1004 17:53:52.908601 836 Train.cpp:568] Shuffling trainset I1004 17:53:52.915084 836 Train.cpp:575] Epoch 3 started! I1004 18:24:59.813577 836 Train.cpp:345] epoch: 3 | nupdates: 114018 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:23:35 | bch(ms): 37.25 | smp(ms): 22.60 | fwd(ms): 7.13 | crit-fwd(ms): 4.87 | bwd(ms): 4.22 | optim(ms): 2.14 | loss: 8.14834 | train-TER: 33.64 | train-WER: 76.58 | lists_combined/dev_list_combined_w2l.lst-loss: 9.09468 | lists_combined/dev_list_combined_w2l.lst-TER: 30.43 | lists_combined/dev_list_combined_w2l.lst-WER: 70.90 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 587.36 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1004 18:25:00.012527 836 Train.cpp:568] Shuffling trainset I1004 18:25:00.019930 836 Train.cpp:575] Epoch 4 started! I1004 18:54:56.944821 836 Train.cpp:345] epoch: 4 | nupdates: 152024 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:22:26 | bch(ms): 35.42 | smp(ms): 21.72 | fwd(ms): 6.80 | crit-fwd(ms): 4.65 | bwd(ms): 3.96 | optim(ms): 1.90 | loss: 7.38968 | train-TER: 30.26 | train-WER: 71.58 | lists_combined/dev_list_combined_w2l.lst-loss: 8.45935 | lists_combined/dev_list_combined_w2l.lst-TER: 28.16 | lists_combined/dev_list_combined_w2l.lst-WER: 66.60 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 617.70 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1004 18:54:57.161537 836 Train.cpp:568] Shuffling trainset I1004 18:54:57.169342 836 Train.cpp:575] Epoch 5 started! I1004 19:26:19.355907 836 Train.cpp:345] epoch: 5 | nupdates: 190030 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:23:34 | bch(ms): 37.21 | smp(ms): 23.40 | fwd(ms): 6.84 | crit-fwd(ms): 4.68 | bwd(ms): 3.99 | optim(ms): 1.94 | loss: 6.89119 | train-TER: 28.05 | train-WER: 67.98 | lists_combined/dev_list_combined_w2l.lst-loss: 8.20595 | lists_combined/dev_list_combined_w2l.lst-TER: 26.68 | lists_combined/dev_list_combined_w2l.lst-WER: 64.19 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 587.95 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):21 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1004 19:26:19.565358 836 Train.cpp:568] Shuffling trainset I1004 19:26:19.571261 836 Train.cpp:575] Epoch 6 started! I1004 19:56:02.677743 836 Train.cpp:345] epoch: 6 | nupdates: 228036 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:22:01 | bch(ms): 34.77 | smp(ms): 21.14 | fwd(ms): 6.76 | crit-fwd(ms): 4.63 | bwd(ms): 3.93 | optim(ms): 1.91 | loss: 6.52918 | train-TER: 26.49 | train-WER: 65.29 | lists_combined/dev_list_combined_w2l.lst-loss: 8.03118 | lists_combined/dev_list_combined_w2l.lst-TER: 25.63 | lists_combined/dev_list_combined_w2l.lst-WER: 61.65 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 629.24 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1004 19:56:02.911392 836 Train.cpp:568] Shuffling trainset I1004 19:56:02.917838 836 Train.cpp:575] Epoch 7 started! I1004 20:25:49.723263 836 Train.cpp:345] epoch: 7 | nupdates: 266042 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:22:11 | bch(ms): 35.02 | smp(ms): 21.21 | fwd(ms): 6.83 | crit-fwd(ms): 4.67 | bwd(ms): 3.99 | optim(ms): 1.94 | loss: 6.23000 | train-TER: 25.27 | train-WER: 63.05 | lists_combined/dev_list_combined_w2l.lst-loss: 7.70509 | lists_combined/dev_list_combined_w2l.lst-TER: 24.99 | lists_combined/dev_list_combined_w2l.lst-WER: 60.14 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 624.66 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1004 20:25:50.085968 836 Train.cpp:568] Shuffling trainset I1004 20:25:50.096889 836 Train.cpp:575] Epoch 8 started! I1004 20:55:17.894244 836 Train.cpp:345] epoch: 8 | nupdates: 304048 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:48 | bch(ms): 34.44 | smp(ms): 20.75 | fwd(ms): 6.78 | crit-fwd(ms): 4.64 | bwd(ms): 3.95 | optim(ms): 1.91 | loss: 5.98664 | train-TER: 24.28 | train-WER: 61.26 | lists_combined/dev_list_combined_w2l.lst-loss: 7.75648 | lists_combined/dev_list_combined_w2l.lst-TER: 24.14 | lists_combined/dev_list_combined_w2l.lst-WER: 58.47 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 635.21 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1004 20:55:18.190292 836 Train.cpp:568] Shuffling trainset I1004 20:55:18.201476 836 Train.cpp:575] Epoch 9 started! I1004 21:24:14.953889 836 Train.cpp:345] epoch: 9 | nupdates: 342054 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:21 | bch(ms): 33.72 | smp(ms): 20.34 | fwd(ms): 6.66 | crit-fwd(ms): 4.56 | bwd(ms): 3.86 | optim(ms): 1.85 | loss: 5.78316 | train-TER: 23.50 | train-WER: 59.82 | lists_combined/dev_list_combined_w2l.lst-loss: 7.75672 | lists_combined/dev_list_combined_w2l.lst-TER: 23.42 | lists_combined/dev_list_combined_w2l.lst-WER: 57.13 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 648.74 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1004 21:24:15.267836 836 Train.cpp:568] Shuffling trainset I1004 21:24:15.275701 836 Train.cpp:575] Epoch 10 started! I1004 21:53:15.849746 836 Train.cpp:345] epoch: 10 | nupdates: 380060 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:23 | bch(ms): 33.78 | smp(ms): 20.41 | fwd(ms): 6.65 | crit-fwd(ms): 4.56 | bwd(ms): 3.86 | optim(ms): 1.86 | loss: 5.59884 | train-TER: 22.82 | train-WER: 58.50 | lists_combined/dev_list_combined_w2l.lst-loss: 7.60903 | lists_combined/dev_list_combined_w2l.lst-TER: 23.19 | lists_combined/dev_list_combined_w2l.lst-WER: 56.40 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 647.54 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1004 21:53:16.133514 836 Train.cpp:568] Shuffling trainset I1004 21:53:16.139902 836 Train.cpp:575] Epoch 11 started! I1004 22:22:11.304687 836 Train.cpp:345] epoch: 11 | nupdates: 418066 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:22 | bch(ms): 33.76 | smp(ms): 20.38 | fwd(ms): 6.65 | crit-fwd(ms): 4.55 | bwd(ms): 3.86 | optim(ms): 1.86 | loss: 5.42824 | train-TER: 22.24 | train-WER: 57.28 | lists_combined/dev_list_combined_w2l.lst-loss: 7.59937 | lists_combined/dev_list_combined_w2l.lst-TER: 22.72 | lists_combined/dev_list_combined_w2l.lst-WER: 55.52 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 648.07 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1004 22:22:11.669816 836 Train.cpp:568] Shuffling trainset I1004 22:22:11.676295 836 Train.cpp:575] Epoch 12 started! I1004 22:50:54.887853 836 Train.cpp:345] epoch: 12 | nupdates: 456072 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:15 | bch(ms): 33.55 | smp(ms): 20.37 | fwd(ms): 6.61 | crit-fwd(ms): 4.53 | bwd(ms): 3.82 | optim(ms): 1.77 | loss: 5.26624 | train-TER: 21.73 | train-WER: 56.31 | lists_combined/dev_list_combined_w2l.lst-loss: 7.60410 | lists_combined/dev_list_combined_w2l.lst-TER: 22.58 | lists_combined/dev_list_combined_w2l.lst-WER: 54.90 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.07 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1004 22:50:55.046376 836 Train.cpp:568] Shuffling trainset I1004 22:50:55.053769 836 Train.cpp:575] Epoch 13 started! I1004 23:19:46.725622 836 Train.cpp:345] epoch: 13 | nupdates: 494078 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:17 | bch(ms): 33.60 | smp(ms): 20.31 | fwd(ms): 6.64 | crit-fwd(ms): 4.55 | bwd(ms): 3.84 | optim(ms): 1.82 | loss: 5.11575 | train-TER: 21.26 | train-WER: 55.37 | lists_combined/dev_list_combined_w2l.lst-loss: 7.47821 | lists_combined/dev_list_combined_w2l.lst-TER: 22.08 | lists_combined/dev_list_combined_w2l.lst-WER: 53.89 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.99 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1004 23:19:47.015522 836 Train.cpp:568] Shuffling trainset I1004 23:19:47.023583 836 Train.cpp:575] Epoch 14 started! I1004 23:48:37.245337 836 Train.cpp:345] epoch: 14 | nupdates: 532084 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:15 | bch(ms): 33.55 | smp(ms): 20.31 | fwd(ms): 6.61 | crit-fwd(ms): 4.53 | bwd(ms): 3.83 | optim(ms): 1.81 | loss: 5.00695 | train-TER: 20.88 | train-WER: 54.62 | lists_combined/dev_list_combined_w2l.lst-loss: 7.32671 | lists_combined/dev_list_combined_w2l.lst-TER: 21.79 | lists_combined/dev_list_combined_w2l.lst-WER: 53.27 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 651.99 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1004 23:48:37.399217 836 Train.cpp:568] Shuffling trainset I1004 23:48:37.405959 836 Train.cpp:575] Epoch 15 started! I1005 00:17:31.429311 836 Train.cpp:345] epoch: 15 | nupdates: 570090 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:17 | bch(ms): 33.61 | smp(ms): 20.30 | fwd(ms): 6.63 | crit-fwd(ms): 4.54 | bwd(ms): 3.84 | optim(ms): 1.83 | loss: 4.90473 | train-TER: 20.53 | train-WER: 53.89 | lists_combined/dev_list_combined_w2l.lst-loss: 7.37521 | lists_combined/dev_list_combined_w2l.lst-TER: 21.48 | lists_combined/dev_list_combined_w2l.lst-WER: 52.57 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.84 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 00:17:31.588961 836 Train.cpp:568] Shuffling trainset I1005 00:17:31.596626 836 Train.cpp:575] Epoch 16 started! I1005 00:46:11.761736 836 Train.cpp:345] epoch: 16 | nupdates: 608096 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:06 | bch(ms): 33.32 | smp(ms): 20.16 | fwd(ms): 6.59 | crit-fwd(ms): 4.52 | bwd(ms): 3.81 | optim(ms): 1.77 | loss: 4.81856 | train-TER: 20.19 | train-WER: 53.27 | lists_combined/dev_list_combined_w2l.lst-loss: 7.24051 | lists_combined/dev_list_combined_w2l.lst-TER: 21.24 | lists_combined/dev_list_combined_w2l.lst-WER: 51.91 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 656.57 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 00:46:11.909168 836 Train.cpp:568] Shuffling trainset I1005 00:46:11.916101 836 Train.cpp:575] Epoch 17 started! I1005 01:14:28.416041 836 Train.cpp:345] epoch: 17 | nupdates: 646102 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:20:56 | bch(ms): 33.07 | smp(ms): 20.00 | fwd(ms): 6.55 | crit-fwd(ms): 4.49 | bwd(ms): 3.78 | optim(ms): 1.75 | loss: 4.73842 | train-TER: 19.89 | train-WER: 52.66 | lists_combined/dev_list_combined_w2l.lst-loss: 7.31798 | lists_combined/dev_list_combined_w2l.lst-TER: 21.13 | lists_combined/dev_list_combined_w2l.lst-WER: 51.75 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 661.51 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 01:14:28.566993 836 Train.cpp:568] Shuffling trainset I1005 01:14:28.574563 836 Train.cpp:575] Epoch 18 started! I1005 01:43:09.190233 836 Train.cpp:345] epoch: 18 | nupdates: 684108 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:16 | bch(ms): 33.58 | smp(ms): 20.30 | fwd(ms): 6.62 | crit-fwd(ms): 4.53 | bwd(ms): 3.83 | optim(ms): 1.83 | loss: 4.66977 | train-TER: 19.64 | train-WER: 52.09 | lists_combined/dev_list_combined_w2l.lst-loss: 7.27907 | lists_combined/dev_list_combined_w2l.lst-TER: 20.95 | lists_combined/dev_list_combined_w2l.lst-WER: 51.06 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 651.51 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 01:43:09.347066 836 Train.cpp:568] Shuffling trainset I1005 01:43:09.353726 836 Train.cpp:575] Epoch 19 started! I1005 02:16:28.082809 836 Train.cpp:345] epoch: 19 | nupdates: 722114 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:15 | bch(ms): 33.56 | smp(ms): 20.27 | fwd(ms): 6.65 | crit-fwd(ms): 4.55 | bwd(ms): 3.84 | optim(ms): 1.81 | loss: 4.60499 | train-TER: 19.36 | train-WER: 51.51 | lists_combined/dev_list_combined_w2l.lst-loss: 7.25474 | lists_combined/dev_list_combined_w2l.lst-TER: 20.79 | lists_combined/dev_list_combined_w2l.lst-WER: 50.96 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 651.84 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 02:16:28.232832 836 Train.cpp:568] Shuffling trainset I1005 02:16:28.240357 836 Train.cpp:575] Epoch 20 started! I1005 02:45:16.546875 836 Train.cpp:345] epoch: 20 | nupdates: 760120 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:19 | bch(ms): 33.66 | smp(ms): 20.32 | fwd(ms): 6.65 | crit-fwd(ms): 4.55 | bwd(ms): 3.85 | optim(ms): 1.84 | loss: 4.55081 | train-TER: 19.16 | train-WER: 51.09 | lists_combined/dev_list_combined_w2l.lst-loss: 7.24694 | lists_combined/dev_list_combined_w2l.lst-TER: 20.52 | lists_combined/dev_list_combined_w2l.lst-WER: 50.21 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 649.89 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):28 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 02:45:16.704602 836 Train.cpp:568] Shuffling trainset I1005 02:45:16.712144 836 Train.cpp:575] Epoch 21 started! I1005 03:14:10.530010 836 Train.cpp:345] epoch: 21 | nupdates: 798126 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:22 | bch(ms): 33.74 | smp(ms): 20.45 | fwd(ms): 6.64 | crit-fwd(ms): 4.55 | bwd(ms): 3.84 | optim(ms): 1.81 | loss: 4.50154 | train-TER: 18.94 | train-WER: 50.62 | lists_combined/dev_list_combined_w2l.lst-loss: 7.28027 | lists_combined/dev_list_combined_w2l.lst-TER: 20.37 | lists_combined/dev_list_combined_w2l.lst-WER: 49.94 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 648.37 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 03:14:10.686422 836 Train.cpp:568] Shuffling trainset I1005 03:14:10.694083 836 Train.cpp:575] Epoch 22 started! I1005 03:42:38.931845 836 Train.cpp:345] epoch: 22 | nupdates: 836132 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:01 | bch(ms): 33.20 | smp(ms): 20.10 | fwd(ms): 6.58 | crit-fwd(ms): 4.51 | bwd(ms): 3.80 | optim(ms): 1.75 | loss: 4.45223 | train-TER: 18.76 | train-WER: 50.27 | lists_combined/dev_list_combined_w2l.lst-loss: 7.18052 | lists_combined/dev_list_combined_w2l.lst-TER: 20.24 | lists_combined/dev_list_combined_w2l.lst-WER: 49.59 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 658.83 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 03:42:39.079479 836 Train.cpp:568] Shuffling trainset I1005 03:42:39.086194 836 Train.cpp:575] Epoch 23 started! I1005 04:11:24.659968 836 Train.cpp:345] epoch: 23 | nupdates: 874138 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:12 | bch(ms): 33.49 | smp(ms): 20.23 | fwd(ms): 6.63 | crit-fwd(ms): 4.54 | bwd(ms): 3.83 | optim(ms): 1.80 | loss: 4.40878 | train-TER: 18.58 | train-WER: 49.89 | lists_combined/dev_list_combined_w2l.lst-loss: 7.13552 | lists_combined/dev_list_combined_w2l.lst-TER: 20.17 | lists_combined/dev_list_combined_w2l.lst-WER: 49.20 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 653.22 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1005 04:11:24.827566 836 Train.cpp:568] Shuffling trainset I1005 04:11:24.834336 836 Train.cpp:575] Epoch 24 started! I1005 04:40:13.074026 836 Train.cpp:345] epoch: 24 | nupdates: 912144 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:17 | bch(ms): 33.61 | smp(ms): 20.37 | fwd(ms): 6.61 | crit-fwd(ms): 4.53 | bwd(ms): 3.83 | optim(ms): 1.80 | loss: 4.37694 | train-TER: 18.43 | train-WER: 49.60 | lists_combined/dev_list_combined_w2l.lst-loss: 7.24847 | lists_combined/dev_list_combined_w2l.lst-TER: 20.15 | lists_combined/dev_list_combined_w2l.lst-WER: 49.37 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.89 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 04:40:13.228339 836 Train.cpp:568] Shuffling trainset I1005 04:40:13.235198 836 Train.cpp:575] Epoch 25 started! I1005 05:08:56.634294 836 Train.cpp:345] epoch: 25 | nupdates: 950150 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:12 | bch(ms): 33.48 | smp(ms): 20.26 | fwd(ms): 6.62 | crit-fwd(ms): 4.54 | bwd(ms): 3.82 | optim(ms): 1.79 | loss: 4.33989 | train-TER: 18.27 | train-WER: 49.24 | lists_combined/dev_list_combined_w2l.lst-loss: 7.15981 | lists_combined/dev_list_combined_w2l.lst-TER: 20.04 | lists_combined/dev_list_combined_w2l.lst-WER: 48.92 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 653.34 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 05:08:56.814008 836 Train.cpp:568] Shuffling trainset I1005 05:08:56.821691 836 Train.cpp:575] Epoch 26 started! I1005 05:37:48.346359 836 Train.cpp:345] epoch: 26 | nupdates: 988156 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:18 | bch(ms): 33.63 | smp(ms): 20.33 | fwd(ms): 6.64 | crit-fwd(ms): 4.55 | bwd(ms): 3.84 | optim(ms): 1.82 | loss: 4.29968 | train-TER: 18.12 | train-WER: 48.92 | lists_combined/dev_list_combined_w2l.lst-loss: 7.03919 | lists_combined/dev_list_combined_w2l.lst-TER: 19.83 | lists_combined/dev_list_combined_w2l.lst-WER: 48.70 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.49 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):31 Total native mallocs:44 Total native frees:0 I1005 05:37:48.519376 836 Train.cpp:568] Shuffling trainset I1005 05:37:48.525861 836 Train.cpp:575] Epoch 27 started! I1005 06:06:33.923302 836 Train.cpp:345] epoch: 27 | nupdates: 1026162 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:13 | bch(ms): 33.51 | smp(ms): 20.32 | fwd(ms): 6.60 | crit-fwd(ms): 4.52 | bwd(ms): 3.81 | optim(ms): 1.79 | loss: 4.27564 | train-TER: 18.01 | train-WER: 48.69 | lists_combined/dev_list_combined_w2l.lst-loss: 7.08509 | lists_combined/dev_list_combined_w2l.lst-TER: 19.79 | lists_combined/dev_list_combined_w2l.lst-WER: 48.35 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 652.78 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 06:06:34.097395 836 Train.cpp:568] Shuffling trainset I1005 06:06:34.103482 836 Train.cpp:575] Epoch 28 started! I1005 06:35:29.554327 836 Train.cpp:345] epoch: 28 | nupdates: 1064168 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:19 | bch(ms): 33.68 | smp(ms): 20.40 | fwd(ms): 6.63 | crit-fwd(ms): 4.54 | bwd(ms): 3.84 | optim(ms): 1.82 | loss: 4.23998 | train-TER: 17.87 | train-WER: 48.42 | lists_combined/dev_list_combined_w2l.lst-loss: 6.95269 | lists_combined/dev_list_combined_w2l.lst-TER: 19.80 | lists_combined/dev_list_combined_w2l.lst-WER: 48.31 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 649.62 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):25 Total native mallocs:44 Total native frees:0 I1005 06:35:29.646842 836 Train.cpp:568] Shuffling trainset I1005 06:35:29.654433 836 Train.cpp:575] Epoch 29 started! I1005 07:04:25.340308 836 Train.cpp:345] epoch: 29 | nupdates: 1102174 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:22 | bch(ms): 33.74 | smp(ms): 20.38 | fwd(ms): 6.66 | crit-fwd(ms): 4.56 | bwd(ms): 3.86 | optim(ms): 1.83 | loss: 4.22253 | train-TER: 17.76 | train-WER: 48.10 | lists_combined/dev_list_combined_w2l.lst-loss: 7.05219 | lists_combined/dev_list_combined_w2l.lst-TER: 19.70 | lists_combined/dev_list_combined_w2l.lst-WER: 48.08 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 648.44 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):21 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 07:04:25.516844 836 Train.cpp:568] Shuffling trainset I1005 07:04:25.524642 836 Train.cpp:575] Epoch 30 started! I1005 07:34:24.972700 836 Train.cpp:345] epoch: 30 | nupdates: 1140180 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:21:18 | bch(ms): 33.65 | smp(ms): 20.32 | fwd(ms): 6.64 | crit-fwd(ms): 4.55 | bwd(ms): 3.85 | optim(ms): 1.83 | loss: 4.18222 | train-TER: 17.61 | train-WER: 47.85 | lists_combined/dev_list_combined_w2l.lst-loss: 6.89049 | lists_combined/dev_list_combined_w2l.lst-TER: 19.61 | lists_combined/dev_list_combined_w2l.lst-WER: 48.00 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 650.16 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1005 07:34:25.094820 836 Train.cpp:568] Shuffling trainset I1005 07:34:25.101013 836 Train.cpp:575] Epoch 31 started! I1005 08:37:48.277294 836 Train.cpp:345] epoch: 31 | nupdates: 1178186 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:46:57 | bch(ms): 74.13 | smp(ms): 58.80 | fwd(ms): 7.93 | crit-fwd(ms): 5.46 | bwd(ms): 4.49 | optim(ms): 1.88 | loss: 4.15091 | train-TER: 17.49 | train-WER: 47.57 | lists_combined/dev_list_combined_w2l.lst-loss: 7.06231 | lists_combined/dev_list_combined_w2l.lst-TER: 19.51 | lists_combined/dev_list_combined_w2l.lst-WER: 47.91 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 295.12 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 08:37:48.472975 836 Train.cpp:568] Shuffling trainset I1005 08:37:48.479029 836 Train.cpp:575] Epoch 32 started! I1005 09:45:09.941092 836 Train.cpp:345] epoch: 32 | nupdates: 1216192 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:49:59 | bch(ms): 78.92 | smp(ms): 63.53 | fwd(ms): 7.95 | crit-fwd(ms): 5.47 | bwd(ms): 4.51 | optim(ms): 1.90 | loss: 4.12651 | train-TER: 17.40 | train-WER: 47.39 | lists_combined/dev_list_combined_w2l.lst-loss: 7.07490 | lists_combined/dev_list_combined_w2l.lst-TER: 19.48 | lists_combined/dev_list_combined_w2l.lst-WER: 47.49 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 277.18 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 09:45:10.229650 836 Train.cpp:568] Shuffling trainset I1005 09:45:10.237155 836 Train.cpp:575] Epoch 33 started! I1005 11:22:51.075546 836 Train.cpp:345] epoch: 33 | nupdates: 1254198 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 01:22:50 | bch(ms): 130.79 | smp(ms): 85.02 | fwd(ms): 26.27 | crit-fwd(ms): 7.44 | bwd(ms): 10.17 | optim(ms): 3.15 | loss: 4.11096 | train-TER: 17.30 | train-WER: 47.13 | lists_combined/dev_list_combined_w2l.lst-loss: 7.01432 | lists_combined/dev_list_combined_w2l.lst-TER: 19.44 | lists_combined/dev_list_combined_w2l.lst-WER: 47.32 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 167.26 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 11:22:51.363813 836 Train.cpp:568] Shuffling trainset I1005 11:22:51.369840 836 Train.cpp:575] Epoch 34 started! I1005 12:11:23.648370 836 Train.cpp:345] epoch: 34 | nupdates: 1292204 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:37:21 | bch(ms): 58.99 | smp(ms): 44.12 | fwd(ms): 7.62 | crit-fwd(ms): 5.24 | bwd(ms): 4.34 | optim(ms): 1.88 | loss: 4.08913 | train-TER: 17.21 | train-WER: 46.94 | lists_combined/dev_list_combined_w2l.lst-loss: 6.97634 | lists_combined/dev_list_combined_w2l.lst-TER: 19.25 | lists_combined/dev_list_combined_w2l.lst-WER: 47.18 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 370.85 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 12:11:23.762384 836 Train.cpp:568] Shuffling trainset I1005 12:11:23.768893 836 Train.cpp:575] Epoch 35 started! I1005 13:08:22.409348 836 Train.cpp:345] epoch: 35 | nupdates: 1330210 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:39:51 | bch(ms): 62.92 | smp(ms): 48.03 | fwd(ms): 7.63 | crit-fwd(ms): 5.24 | bwd(ms): 4.34 | optim(ms): 1.89 | loss: 4.06533 | train-TER: 17.11 | train-WER: 46.77 | lists_combined/dev_list_combined_w2l.lst-loss: 7.01603 | lists_combined/dev_list_combined_w2l.lst-TER: 19.29 | lists_combined/dev_list_combined_w2l.lst-WER: 47.02 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 347.70 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 13:08:22.567930 836 Train.cpp:568] Shuffling trainset I1005 13:08:22.574388 836 Train.cpp:575] Epoch 36 started! I1005 14:54:28.551764 836 Train.cpp:345] epoch: 36 | nupdates: 1368216 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:51:42 | bch(ms): 81.63 | smp(ms): 66.25 | fwd(ms): 7.97 | crit-fwd(ms): 5.48 | bwd(ms): 4.51 | optim(ms): 1.88 | loss: 4.04555 | train-TER: 17.03 | train-WER: 46.58 | lists_combined/dev_list_combined_w2l.lst-loss: 7.07964 | lists_combined/dev_list_combined_w2l.lst-TER: 19.15 | lists_combined/dev_list_combined_w2l.lst-WER: 46.74 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 267.99 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1005 14:54:28.772959 836 Train.cpp:568] Shuffling trainset I1005 14:54:28.784948 836 Train.cpp:575] Epoch 37 started! I1005 16:06:03.466958 836 Train.cpp:345] epoch: 37 | nupdates: 1406222 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:53:50 | bch(ms): 84.99 | smp(ms): 69.13 | fwd(ms): 8.16 | crit-fwd(ms): 5.53 | bwd(ms): 4.66 | optim(ms): 1.97 | loss: 4.03071 | train-TER: 16.96 | train-WER: 46.41 | lists_combined/dev_list_combined_w2l.lst-loss: 6.94276 | lists_combined/dev_list_combined_w2l.lst-TER: 19.25 | lists_combined/dev_list_combined_w2l.lst-WER: 46.91 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 257.39 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 16:06:03.665633 836 Train.cpp:568] Shuffling trainset I1005 16:06:03.679847 836 Train.cpp:575] Epoch 38 started! I1005 17:22:58.651983 836 Train.cpp:345] epoch: 38 | nupdates: 1444228 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 01:00:17 | bch(ms): 95.18 | smp(ms): 78.78 | fwd(ms): 8.46 | crit-fwd(ms): 5.53 | bwd(ms): 4.93 | optim(ms): 1.94 | loss: 4.00953 | train-TER: 16.86 | train-WER: 46.16 | lists_combined/dev_list_combined_w2l.lst-loss: 7.00212 | lists_combined/dev_list_combined_w2l.lst-TER: 19.01 | lists_combined/dev_list_combined_w2l.lst-WER: 46.43 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 229.85 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 17:22:58.857200 836 Train.cpp:568] Shuffling trainset I1005 17:22:58.865422 836 Train.cpp:575] Epoch 39 started! I1005 18:32:06.426584 836 Train.cpp:345] epoch: 39 | nupdates: 1482234 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:52:16 | bch(ms): 82.52 | smp(ms): 67.10 | fwd(ms): 7.98 | crit-fwd(ms): 5.49 | bwd(ms): 4.52 | optim(ms): 1.89 | loss: 3.99679 | train-TER: 16.82 | train-WER: 46.10 | lists_combined/dev_list_combined_w2l.lst-loss: 7.04353 | lists_combined/dev_list_combined_w2l.lst-TER: 19.07 | lists_combined/dev_list_combined_w2l.lst-WER: 46.80 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 265.09 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):24 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1005 18:32:06.644505 836 Train.cpp:568] Shuffling trainset I1005 18:32:06.653376 836 Train.cpp:575] Epoch 40 started! I1005 19:47:39.612273 836 Train.cpp:345] epoch: 40 | nupdates: 1520240 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:59:07 | bch(ms): 93.34 | smp(ms): 77.72 | fwd(ms): 8.11 | crit-fwd(ms): 5.51 | bwd(ms): 4.61 | optim(ms): 1.88 | loss: 3.98094 | train-TER: 16.74 | train-WER: 45.91 | lists_combined/dev_list_combined_w2l.lst-loss: 6.91318 | lists_combined/dev_list_combined_w2l.lst-TER: 18.93 | lists_combined/dev_list_combined_w2l.lst-WER: 46.40 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 234.38 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 19:47:39.948226 836 Train.cpp:568] Shuffling trainset I1005 19:47:39.956845 836 Train.cpp:575] Epoch 41 started! I1005 20:59:18.367830 836 Train.cpp:345] epoch: 41 | nupdates: 1558246 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:54:07 | bch(ms): 85.45 | smp(ms): 70.04 | fwd(ms): 7.99 | crit-fwd(ms): 5.50 | bwd(ms): 4.52 | optim(ms): 1.88 | loss: 3.96222 | train-TER: 16.66 | train-WER: 45.75 | lists_combined/dev_list_combined_w2l.lst-loss: 6.87396 | lists_combined/dev_list_combined_w2l.lst-TER: 18.99 | lists_combined/dev_list_combined_w2l.lst-WER: 46.37 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 256.02 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):22 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1005 20:59:18.749796 836 Train.cpp:568] Shuffling trainset I1005 20:59:18.757725 836 Train.cpp:575] Epoch 42 started! I1005 22:20:27.369956 836 Train.cpp:345] epoch: 42 | nupdates: 1596252 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 01:04:20 | bch(ms): 101.58 | smp(ms): 85.65 | fwd(ms): 8.32 | crit-fwd(ms): 5.53 | bwd(ms): 4.73 | optim(ms): 1.85 | loss: 3.94486 | train-TER: 16.61 | train-WER: 45.61 | lists_combined/dev_list_combined_w2l.lst-loss: 7.00967 | lists_combined/dev_list_combined_w2l.lst-TER: 18.84 | lists_combined/dev_list_combined_w2l.lst-WER: 46.06 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 215.36 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 22:20:27.588786 836 Train.cpp:568] Shuffling trainset I1005 22:20:27.595651 836 Train.cpp:575] Epoch 43 started! I1005 23:25:53.007688 836 Train.cpp:345] epoch: 43 | nupdates: 1634258 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:48:45 | bch(ms): 76.97 | smp(ms): 61.73 | fwd(ms): 7.93 | crit-fwd(ms): 5.45 | bwd(ms): 4.48 | optim(ms): 1.82 | loss: 3.93306 | train-TER: 16.54 | train-WER: 45.49 | lists_combined/dev_list_combined_w2l.lst-loss: 6.89692 | lists_combined/dev_list_combined_w2l.lst-TER: 18.83 | lists_combined/dev_list_combined_w2l.lst-WER: 46.08 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 284.24 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1005 23:25:53.397115 836 Train.cpp:568] Shuffling trainset I1005 23:25:53.404131 836 Train.cpp:575] Epoch 44 started! I1006 00:31:11.660015 836 Train.cpp:345] epoch: 44 | nupdates: 1672264 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:48:38 | bch(ms): 76.79 | smp(ms): 61.58 | fwd(ms): 7.92 | crit-fwd(ms): 5.45 | bwd(ms): 4.47 | optim(ms): 1.81 | loss: 3.91813 | train-TER: 16.47 | train-WER: 45.33 | lists_combined/dev_list_combined_w2l.lst-loss: 6.92238 | lists_combined/dev_list_combined_w2l.lst-TER: 18.89 | lists_combined/dev_list_combined_w2l.lst-WER: 46.04 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 284.89 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1006 00:31:11.967164 836 Train.cpp:568] Shuffling trainset I1006 00:31:11.973430 836 Train.cpp:575] Epoch 45 started! I1006 01:36:17.634524 836 Train.cpp:345] epoch: 45 | nupdates: 1710270 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:48:32 | bch(ms): 76.62 | smp(ms): 61.40 | fwd(ms): 7.93 | crit-fwd(ms): 5.46 | bwd(ms): 4.47 | optim(ms): 1.81 | loss: 3.90781 | train-TER: 16.44 | train-WER: 45.28 | lists_combined/dev_list_combined_w2l.lst-loss: 6.94322 | lists_combined/dev_list_combined_w2l.lst-TER: 18.76 | lists_combined/dev_list_combined_w2l.lst-WER: 45.96 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 285.51 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1006 01:36:17.854876 836 Train.cpp:568] Shuffling trainset I1006 01:36:17.861268 836 Train.cpp:575] Epoch 46 started! I1006 02:47:27.609741 836 Train.cpp:345] epoch: 46 | nupdates: 1748276 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:54:37 | bch(ms): 86.23 | smp(ms): 70.94 | fwd(ms): 7.97 | crit-fwd(ms): 5.48 | bwd(ms): 4.49 | optim(ms): 1.81 | loss: 3.89711 | train-TER: 16.38 | train-WER: 45.14 | lists_combined/dev_list_combined_w2l.lst-loss: 7.04323 | lists_combined/dev_list_combined_w2l.lst-TER: 18.79 | lists_combined/dev_list_combined_w2l.lst-WER: 45.78 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 253.70 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1006 02:47:27.900152 836 Train.cpp:568] Shuffling trainset I1006 02:47:27.907727 836 Train.cpp:575] Epoch 47 started! I1006 03:52:52.133862 836 Train.cpp:345] epoch: 47 | nupdates: 1786282 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:48:37 | bch(ms): 76.77 | smp(ms): 61.51 | fwd(ms): 7.94 | crit-fwd(ms): 5.46 | bwd(ms): 4.48 | optim(ms): 1.82 | loss: 3.88154 | train-TER: 16.30 | train-WER: 44.93 | lists_combined/dev_list_combined_w2l.lst-loss: 6.87829 | lists_combined/dev_list_combined_w2l.lst-TER: 18.69 | lists_combined/dev_list_combined_w2l.lst-WER: 45.73 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 284.97 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1006 03:52:52.361037 836 Train.cpp:568] Shuffling trainset I1006 03:52:52.367882 836 Train.cpp:575] Epoch 48 started! I1006 04:58:20.962522 836 Train.cpp:345] epoch: 48 | nupdates: 1824288 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:48:50 | bch(ms): 77.10 | smp(ms): 61.87 | fwd(ms): 7.94 | crit-fwd(ms): 5.46 | bwd(ms): 4.48 | optim(ms): 1.81 | loss: 3.86555 | train-TER: 16.26 | train-WER: 44.86 | lists_combined/dev_list_combined_w2l.lst-loss: 6.92679 | lists_combined/dev_list_combined_w2l.lst-TER: 18.58 | lists_combined/dev_list_combined_w2l.lst-WER: 45.67 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 283.75 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1006 04:58:21.197827 836 Train.cpp:568] Shuffling trainset I1006 04:58:21.204349 836 Train.cpp:575] Epoch 49 started! I1006 06:04:30.588872 836 Train.cpp:345] epoch: 49 | nupdates: 1862294 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:49:26 | bch(ms): 78.05 | smp(ms): 62.82 | fwd(ms): 7.94 | crit-fwd(ms): 5.46 | bwd(ms): 4.48 | optim(ms): 1.80 | loss: 3.85393 | train-TER: 16.21 | train-WER: 44.76 | lists_combined/dev_list_combined_w2l.lst-loss: 6.96968 | lists_combined/dev_list_combined_w2l.lst-TER: 18.60 | lists_combined/dev_list_combined_w2l.lst-WER: 45.44 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 280.30 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):29 Total native mallocs:44 Total native frees:0 I1006 06:04:30.775688 836 Train.cpp:568] Shuffling trainset I1006 06:04:30.782145 836 Train.cpp:575] Epoch 50 started! I1006 07:11:12.250488 836 Train.cpp:345] epoch: 50 | nupdates: 1900300 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:49:55 | bch(ms): 78.83 | smp(ms): 63.58 | fwd(ms): 7.94 | crit-fwd(ms): 5.47 | bwd(ms): 4.48 | optim(ms): 1.81 | loss: 3.84295 | train-TER: 16.15 | train-WER: 44.63 | lists_combined/dev_list_combined_w2l.lst-loss: 6.91451 | lists_combined/dev_list_combined_w2l.lst-TER: 18.62 | lists_combined/dev_list_combined_w2l.lst-WER: 45.37 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 277.53 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):28 Size of free block pool (large):27 Total native mallocs:44 Total native frees:0 I1006 07:11:12.420307 836 Train.cpp:568] Shuffling trainset I1006 07:11:12.426507 836 Train.cpp:575] Epoch 51 started! I1006 08:16:34.063868 836 Train.cpp:345] epoch: 51 | nupdates: 1938306 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:48:46 | bch(ms): 77.00 | smp(ms): 61.72 | fwd(ms): 7.96 | crit-fwd(ms): 5.47 | bwd(ms): 4.49 | optim(ms): 1.81 | loss: 3.82909 | train-TER: 16.10 | train-WER: 44.54 | lists_combined/dev_list_combined_w2l.lst-loss: 6.78036 | lists_combined/dev_list_combined_w2l.lst-TER: 18.57 | lists_combined/dev_list_combined_w2l.lst-WER: 45.51 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 284.10 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):23 Size of free block pool (large):26 Total native mallocs:44 Total native frees:0 I1006 08:16:34.132355 836 Train.cpp:568] Shuffling trainset I1006 08:16:34.139956 836 Train.cpp:575] Epoch 52 started! I1006 09:25:21.557397 836 Train.cpp:345] epoch: 52 | nupdates: 1976312 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:50:44 | bch(ms): 80.10 | smp(ms): 64.84 | fwd(ms): 7.95 | crit-fwd(ms): 5.47 | bwd(ms): 4.49 | optim(ms): 1.82 | loss: 3.82672 | train-TER: 16.07 | train-WER: 44.47 | lists_combined/dev_list_combined_w2l.lst-loss: 6.81808 | lists_combined/dev_list_combined_w2l.lst-TER: 18.69 | lists_combined/dev_list_combined_w2l.lst-WER: 45.50 | avg-isz: 546 | avg-tsz: 062 | max-tsz: 303 | hrs: 230.95 | thrpt(sec/sec): 273.10 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):25 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1006 09:25:21.674216 836 Train.cpp:568] Shuffling trainset I1006 09:25:21.680546 836 Train.cpp:575] Epoch 53 started! I1006 10:16:17.791509 836 Train.cpp:345] epoch: 53 | nupdates: 2000001 | lr: 0.050000 | lrcriterion: 0.000000 | runtime: 00:34:04 | bch(ms): 86.32 | smp(ms): 71.12 | fwd(ms): 7.89 | crit-fwd(ms): 5.40 | bwd(ms): 4.46 | optim(ms): 1.83 | loss: 3.75766 | train-TER: 15.97 | train-WER: 44.26 | lists_combined/dev_list_combined_w2l.lst-loss: 6.94832 | lists_combined/dev_list_combined_w2l.lst-TER: 18.53 | lists_combined/dev_list_combined_w2l.lst-WER: 45.14 | avg-isz: 539 | avg-tsz: 061 | max-tsz: 282 | hrs: 141.98 | thrpt(sec/sec): 249.96 Memory Manager Stats MemoryManager type: CachingMemoryManager Number of allocated blocks:22 Size of free block pool (small):26 Size of free block pool (large):28 Total native mallocs:44 Total native frees:0 I1006 10:16:18.134868 836 Train.cpp:751] Finished training

The above training logs seems to be appropriate to me and gives good results on test set at the time of decoding

One more doubt

I have set an lr = 0.1 then why at the time of training in the logs, then why it shows 0.05 all the time?

Trikaldarshi commented 4 years ago

@tlikhomanenko Could you please suggest something for the spec_aug parameter values like --saug_start_update=1 [is it a right choce?] --saug_fmaskf=27 --saug_fmaskn=2 --saug_tmaskt=100 --saug_tmaskp=1.0 --saug_tmaskn=2

I tried a few sets but didn't work for me.

tlikhomanenko commented 4 years ago

to fix lr issue, please set lr_decay=100000 (it is overflow of param, we fixed in some commit, probably you don't have the fix).

try this

--saug_start_update=1 
--saug_fmaskf=27
--saug_fmaskn=2
--saug_tmaskt=30
--saug_tmaskp=0.05
--saug_tmaskn=2
Trikaldarshi commented 4 years ago

Thank you very much for your help. It worked and improved the results. Closing this issue.