search

flashlight / wav2letter

Facebook AI Research's Automatic Speech Recognition Toolkit
https://github.com/facebookresearch/wav2letter/wiki
Other
6.37k stars 1.01k forks source link

Libri-1K training #728

Open AlexandderGorodetski opened 4 years ago

AlexandderGorodetski commented 4 years ago

Hello,

I am trying to run Libri-1K transformer training. After 20 epoch train-WER=327 (please see log below). Is something wrong with my running?

I0530 04:41:50.446601 981 Train.cpp:573] Epoch 20 started! I0530 09:16:43.902400 981 Train.cpp:345] epoch: 20 | nupdates: 667967 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:37 | bch(ms): 407.27 | smp(ms): 0.82 | fwd(ms): 111.73 | crit-fwd( ms): 7.48 | bwd(ms): 190.41 | optim(ms): 66.70 | loss: 268.62477 | train-TER: 176.39 | train-WER: 327.60 | dev-clean-loss: 159.58057 | dev-clean-TER: 255.53 | dev-clean-WER: 442.62 | dev-other-loss: 139. 75895 | dev-other-TER: 270.28 | dev-other-WER: 455.52 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 241.49

Thanks, Alexander.

tlikhomanenko commented 4 years ago

Could you post the full log? and also info if this is CTC or s2s criterion. Also please add link or info about your training config.

AlexandderGorodetski commented 4 years ago

It is s2s criterion. In the next comment I will copy&pase content of config, log and perf files.

AlexandderGorodetski commented 4 years ago

{ "config": [ { "key": "runIdx", "value": "1" }, { "key": "timestamp", "value": "2020-05-26, 2020-05-26" }, { "key": "hostname", "value": "5653fe88dcf4" }, { "key": "runPath", "value": "/media_alex/common/projects/wav2letter/exp/libri-1K/am_transformer_seq2seq_librispeech" }, { "key": "username", "value": "" }, { "key": "gflags", "value": "--flagfile=\n--fromenv=\n--tryfromenv=\n--undefok=\n--tab_completion_columns=80\n--tab_completion_word=\n--help=false\n--helpfull=false\n--helpmatch=\n--helpon=\n--helppackage=false\n--helpshort=false\n--helpxml=false\n--version=false\n--adambeta1=0.90000000000000002\n--adambeta2=0.999\n--am=\n--am_decoder_tr_dropout=0.10000000000000001\n--am_decoder_tr_layerdrop=0.10000000000000001\n--am_decoder_tr_layers=6\n--arch=am_arch/am_transformer_s2s.arch\n--archdir=/root/wav2letter/recipes/models/sota/2019\n--attention=keyvalue\n--attentionthreshold=2147483647\n--attnWindow=softPretrain\n--attnconvchannel=0\n--attnconvkernel=0\n--attndim=0\n--batchsize=8\n--beamsize=2500\n--beamsizetoken=250000\n--beamthreshold=25\n--blobdata=false\n--channels=1\n--criterion=transformer\n--critoptim=adagrad\n--datadir=\n--dataorder=output_spiral\n--decoderattnround=1\n--decoderdropout=0\n--decoderrnnlayer=1\n--decodertype=wrd\n--devwin=0\n--emission_dir=\n--emission_queue_size=3000\n--enable_distributed=true\n--encoderdim=256\n--eosscore=0\n--eostoken=true\n--everstoredb=false\n--fftcachesize=1\n--filterbanks=80\n--flagsfile=train_am_transformer_s2s.cfg\n--framesizems=25\n--framestridems=10\n--gamma=1\n--gumbeltemperature=1\n--input=flac\n--inputbinsize=25\n--inputfeeding=false\n--isbeamdump=false\n--iter=9223372036854775807\n--itersave=false\n--labelsmooth=0.050000000000000003\n--leftWindowSize=50\n--lexicon=/media_alex/common/projects/wav2letter/db/libri-1K/model/am/librispeech-train+dev-unigram-10000-nbest10.lexicon\n--linlr=-1\n--linlrcrit=-1\n--linseg=0\n--lm=\n--lm_memory=5000\n--lm_vocab=\n--lmtype=kenlm\n--lmweight=0\n--localnrmlleftctx=0\n--localnrmlrightctx=0\n--logadd=false\n--lr=0.029999999999999999\n--lr_decay=9223372036854775807\n--lr_decay_step=9223372036854775807\n--lrcosine=false\n--lrcrit=0.029999999999999999\n--max_devices_per_node=8\n--maxdecoderoutputlen=120\n--maxgradnorm=0.10000000000000001\n--maxisz=9223372036854775807\n--maxload=-1\n--maxrate=10\n--maxsil=50\n--maxtsz=9223372036854775807\n--maxword=-1\n--melfloor=1\n--memstepsize=5000000\n--mfcc=false\n--mfcccoeffs=13\n--mfsc=true\n--minisz=0\n--minrate=3\n--minsil=0\n--mintsz=0\n--momentum=0\n--netoptim=adagrad\n--noresample=false\n--nthread=6\n--nthread_decoder=1\n--nthread_decoder_am_forward=1\n--numattnhead=8\n--onorm=target\n--optimepsilon=1e-08\n--optimrho=0.90000000000000002\n--outputbinsize=5\n--pctteacherforcing=99\n--pcttraineval=1\n--pow=false\n--pretrainWindow=3\n--replabel=0\n--reportiters=0\n--rightWindowSize=50\n--rndv_filepath=\n--rundir=/media_alex/common/projects/wav2letter/exp/libri-1K\n--runname=am_transformer_seq2seq_librispeech\n--samplerate=16000\n--sampletarget=0.01\n--samplingstrategy=rand\n--saug_fmaskf=27\n--saug_fmaskn=2\n--saug_start_update=-1\n--saug_tmaskn=2\n--saug_tmaskp=1\n--saug_tmaskt=100\n--sclite=\n--seed=0\n--show=false\n--showletters=false\n--silscore=0\n--smearing=none\n--smoothingtemperature=1\n--softwoffset=10\n--softwrate=5\n--softwstd=4\n--sqnorm=true\n--stepsize=9223372036854775807\n--surround=\n--tag=\n--target=tkn\n--test=\n--tokens=librispeech-train-all-unigram-10000.tokens\n--tokensdir=/media_alex/common/projects/wav2letter/db/libri-1K/model/am\n--train=/media_alex/common/projects/wav2letter/db/libri-1K/data/lists/train-clean-100.lst,/media_alex/common/projects/wav2letter/db/libri-1K/data/lists/train-clean-360.lst,/media_alex/common/projects/wav2letter/db/libri-1K/data/lists/train-other-500.lst\n--trainWithWindow=true\n--transdiag=0\n--unkscore=-inf\n--use_memcache=false\n--uselexicon=true\n--usewordpiece=true\n--valid=dev-clean:/media_alex/common/projects/wav2letter/db/libri-1K/data/lists/dev-clean.lst,dev-other:/mediaalex/common/projects/wav2letter/db/libri-1K/data/lists/dev-other.lst\n--warmup=8000\n--weightdecay=0\n--wordscore=0\n--wordseparator=\n--world_rank=0\n--world_size=1\n--alsologtoemail=\n--alsologtostderr=false\n--colorlogtostderr=false\n--drop_log_memory=true\n--log_backtrace_at=\n--log_dir=\n--log_link=\n--log_prefix=true\n--logbuflevel=0\n--logbufsecs=30\n--logemaillevel=999\n--logfile_mode=436\n--logmailer=/bin/mail\n--logtostderr=true\n--max_log_size=1800\n--minloglevel=0\n--stderrthreshold=2\n--stop_logging_if_full_disk=false\n--symbolize_stacktrace=true\n--v=0\n--vmodule=\n" }, { "key": "commandline", "value": "/root/wav2letter/build/Train train --flagsfile train_am_transformer_s2s.cfg --minloglevel=0 --logtostderr=1" }, { "key": "programname", "value": "/root/wav2letter/build/Train" } ] }

AlexandderGorodetski commented 4 years ago

epoch: 1 | nupdates: 4 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 00:00:15 | bch(ms): 3913.51 | smp(ms): 112.27 | fwd(ms): 1820.82 | crit-fwd(ms): 635.73 | bwd(ms): 1153.31 | optim(ms): 493.75 | loss: 474.52074 | train-TER: 0.00 | train-WER: 0.00 | dev-clean-loss: 236.83615 | dev-clean-TER: 407.98 | dev-clean-WER: 589.89 | dev-other-loss: 210.84624 | dev-other-TER: 474.91 | dev-other-WER: 669.18 | avg-isz: 1483 | avg-tsz: 049 | max-tsz: 061 | hrs: 0.13 | thrpt(sec/sec): 30.33 epoch: 2 | nupdates: 35159 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:28 | bch(ms): 405.30 | smp(ms): 0.89 | fwd(ms): 112.79 | crit-fwd(ms): 8.02 | bwd(ms): 189.02 | optim(ms): 65.77 | loss: 321.18602 | train-TER: 217.97 | train-WER: 332.97 | dev-clean-loss: 182.20035 | dev-clean-TER: 407.91 | dev-clean-WER: 589.86 | dev-other-loss: 160.49133 | dev-other-TER: 474.85 | dev-other-WER: 669.14 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 242.66 epoch: 3 | nupdates: 70315 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:26 | bch(ms): 408.66 | smp(ms): 0.81 | fwd(ms): 111.97 | crit-fwd(ms): 7.64 | bwd(ms): 189.56 | optim(ms): 66.00 | loss: 296.58603 | train-TER: 231.24 | train-WER: 356.60 | dev-clean-loss: 174.73162 | dev-clean-TER: 407.80 | dev-clean-WER: 589.80 | dev-other-loss: 153.43288 | dev-other-TER: 474.68 | dev-other-WER: 669.01 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 240.67 epoch: 4 | nupdates: 105471 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:35 | bch(ms): 408.92 | smp(ms): 0.81 | fwd(ms): 111.95 | crit-fwd(ms): 7.65 | bwd(ms): 189.82 | optim(ms): 66.09 | loss: 288.46082 | train-TER: 210.69 | train-WER: 353.08 | dev-clean-loss: 171.21415 | dev-clean-TER: 339.16 | dev-clean-WER: 548.01 | dev-other-loss: 150.11066 | dev-other-TER: 385.46 | dev-other-WER: 605.89 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 240.52 epoch: 5 | nupdates: 140627 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:58 | bch(ms): 409.55 | smp(ms): 0.83 | fwd(ms): 111.72 | crit-fwd(ms): 7.49 | bwd(ms): 190.20 | optim(ms): 67.52 | loss: 283.93068 | train-TER: 189.01 | train-WER: 343.27 | dev-clean-loss: 168.98421 | dev-clean-TER: 241.70 | dev-clean-WER: 413.02 | dev-other-loss: 147.96650 | dev-other-TER: 262.60 | dev-other-WER: 435.70 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 240.15 epoch: 6 | nupdates: 175783 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:57 | bch(ms): 407.84 | smp(ms): 0.83 | fwd(ms): 111.83 | crit-fwd(ms): 7.61 | bwd(ms): 189.99 | optim(ms): 66.81 | loss: 280.91889 | train-TER: 182.03 | train-WER: 335.07 | dev-clean-loss: 167.06601 | dev-clean-TER: 215.92 | dev-clean-WER: 373.11 | dev-other-loss: 146.56506 | dev-other-TER: 231.11 | dev-other-WER: 386.52 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 241.16 epoch: 7 | nupdates: 210939 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:00:50 | bch(ms): 411.04 | smp(ms): 2.11 | fwd(ms): 112.99 | crit-fwd(ms): 8.17 | bwd(ms): 190.67 | optim(ms): 66.80 | loss: 278.78642 | train-TER: 181.12 | train-WER: 334.50 | dev-clean-loss: 166.01244 | dev-clean-TER: 197.06 | dev-clean-WER: 340.50 | dev-other-loss: 145.41342 | dev-other-TER: 208.79 | dev-other-WER: 347.56 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 239.28 epoch: 8 | nupdates: 246095 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:00:00 | bch(ms): 409.62 | smp(ms): 0.97 | fwd(ms): 112.19 | crit-fwd(ms): 7.74 | bwd(ms): 191.06 | optim(ms): 67.01 | loss: 277.06449 | train-TER: 180.49 | train-WER: 333.80 | dev-clean-loss: 164.91638 | dev-clean-TER: 205.70 | dev-clean-WER: 354.55 | dev-other-loss: 144.29309 | dev-other-TER: 216.86 | dev-other-WER: 360.08 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 240.11 epoch: 9 | nupdates: 281251 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:40 | bch(ms): 407.34 | smp(ms): 0.82 | fwd(ms): 111.43 | crit-fwd(ms): 7.38 | bwd(ms): 190.10 | optim(ms): 66.91 | loss: 275.67091 | train-TER: 180.19 | train-WER: 333.59 | dev-clean-loss: 164.14041 | dev-clean-TER: 206.74 | dev-clean-WER: 357.39 | dev-other-loss: 143.83753 | dev-other-TER: 218.28 | dev-other-WER: 363.58 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 241.45 epoch: 10 | nupdates: 316407 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:27 | bch(ms): 406.98 | smp(ms): 0.81 | fwd(ms): 111.55 | crit-fwd(ms): 7.46 | bwd(ms): 190.12 | optim(ms): 66.40 | loss: 274.56721 | train-TER: 179.35 | train-WER: 332.38 | dev-clean-loss: 163.38458 | dev-clean-TER: 211.50 | dev-clean-WER: 365.98 | dev-other-loss: 143.14340 | dev-other-TER: 222.45 | dev-other-WER: 371.31 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 241.67 epoch: 11 | nupdates: 351563 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:39 | bch(ms): 407.31 | smp(ms): 0.91 | fwd(ms): 111.46 | crit-fwd(ms): 7.41 | bwd(ms): 189.98 | optim(ms): 66.82 | loss: 273.55285 | train-TER: 179.89 | train-WER: 333.18 | dev-clean-loss: 162.81614 | dev-clean-TER: 216.88 | dev-clean-WER: 375.51 | dev-other-loss: 142.69722 | dev-other-TER: 227.90 | dev-other-WER: 381.10 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 241.47 epoch: 12 | nupdates: 386719 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:52 | bch(ms): 409.39 | smp(ms): 0.84 | fwd(ms): 111.66 | crit-fwd(ms): 7.42 | bwd(ms): 190.92 | optim(ms): 67.83 | loss: 272.77044 | train-TER: 178.82 | train-WER: 331.49 | dev-clean-loss: 162.13990 | dev-clean-TER: 222.48 | dev-clean-WER: 385.41 | dev-other-loss: 142.38879 | dev-other-TER: 233.06 | dev-other-WER: 390.72 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 107 | hrs: 960.47 | thrpt(sec/sec): 240.24 epoch: 13 | nupdates: 421875 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:55 | bch(ms): 409.49 | smp(ms): 0.84 | fwd(ms): 111.67 | crit-fwd(ms): 7.45 | bwd(ms): 190.95 | optim(ms): 67.70 | loss: 272.03696 | train-TER: 179.71 | train-WER: 333.26 | dev-clean-loss: 161.56650 | dev-clean-TER: 235.14 | dev-clean-WER: 407.74 | dev-other-loss: 142.02924 | dev-other-TER: 248.08 | dev-other-WER: 417.02 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 240.19 epoch: 14 | nupdates: 457031 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:00:38 | bch(ms): 410.69 | smp(ms): 0.85 | fwd(ms): 111.66 | crit-fwd(ms): 7.39 | bwd(ms): 191.33 | optim(ms): 68.41 | loss: 271.37681 | train-TER: 179.71 | train-WER: 333.30 | dev-clean-loss: 161.41158 | dev-clean-TER: 227.06 | dev-clean-WER: 393.53 | dev-other-loss: 141.45859 | dev-other-TER: 236.68 | dev-other-WER: 397.38 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 239.48 epoch: 15 | nupdates: 492187 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:00:13 | bch(ms): 409.99 | smp(ms): 0.85 | fwd(ms): 111.72 | crit-fwd(ms): 7.45 | bwd(ms): 191.06 | optim(ms): 68.18 | loss: 270.76164 | train-TER: 178.83 | train-WER: 331.85 | dev-clean-loss: 160.79378 | dev-clean-TER: 237.07 | dev-clean-WER: 411.48 | dev-other-loss: 140.96830 | dev-other-TER: 249.30 | dev-other-WER: 420.08 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 239.89 epoch: 16 | nupdates: 527343 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:47 | bch(ms): 409.25 | smp(ms): 0.83 | fwd(ms): 111.45 | crit-fwd(ms): 7.32 | bwd(ms): 190.88 | optim(ms): 68.17 | loss: 270.29345 | train-TER: 177.79 | train-WER: 330.12 | dev-clean-loss: 160.59392 | dev-clean-TER: 237.15 | dev-clean-WER: 411.61 | dev-other-loss: 140.85086 | dev-other-TER: 248.86 | dev-other-WER: 419.42 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 240.32 epoch: 17 | nupdates: 562499 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:00:03 | bch(ms): 409.71 | smp(ms): 0.88 | fwd(ms): 111.50 | crit-fwd(ms): 7.33 | bwd(ms): 190.94 | optim(ms): 68.51 | loss: 269.78664 | train-TER: 177.31 | train-WER: 329.13 | dev-clean-loss: 160.38124 | dev-clean-TER: 243.65 | dev-clean-WER: 422.71 | dev-other-loss: 140.47210 | dev-other-TER: 256.32 | dev-other-WER: 432.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 240.05 epoch: 18 | nupdates: 597655 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:55 | bch(ms): 409.47 | smp(ms): 0.84 | fwd(ms): 111.48 | crit-fwd(ms): 7.28 | bwd(ms): 190.74 | optim(ms): 68.65 | loss: 269.40423 | train-TER: 177.00 | train-WER: 328.78 | dev-clean-loss: 160.02742 | dev-clean-TER: 250.20 | dev-clean-WER: 433.70 | dev-other-loss: 140.13095 | dev-other-TER: 264.97 | dev-other-WER: 446.57 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 240.20 epoch: 19 | nupdates: 632811 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:00:04 | bch(ms): 409.73 | smp(ms): 0.99 | fwd(ms): 111.67 | crit-fwd(ms): 7.42 | bwd(ms): 191.08 | optim(ms): 68.30 | loss: 268.99767 | train-TER: 176.54 | train-WER: 328.07 | dev-clean-loss: 159.64364 | dev-clean-TER: 255.31 | dev-clean-WER: 442.22 | dev-other-loss: 140.17385 | dev-other-TER: 271.30 | dev-other-WER: 457.02 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 240.04 epoch: 20 | nupdates: 667967 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:37 | bch(ms): 407.27 | smp(ms): 0.82 | fwd(ms): 111.73 | crit-fwd(ms): 7.48 | bwd(ms): 190.41 | optim(ms): 66.70 | loss: 268.62477 | train-TER: 176.39 | train-WER: 327.60 | dev-clean-loss: 159.58057 | dev-clean-TER: 255.53 | dev-clean-WER: 442.62 | dev-other-loss: 139.75895 | dev-other-TER: 270.28 | dev-other-WER: 455.52 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 241.49 epoch: 21 | nupdates: 703123 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:59 | bch(ms): 407.89 | smp(ms): 0.83 | fwd(ms): 111.75 | crit-fwd(ms): 7.49 | bwd(ms): 190.58 | optim(ms): 67.36 | loss: 268.28157 | train-TER: 175.00 | train-WER: 325.05 | dev-clean-loss: 159.39702 | dev-clean-TER: 254.49 | dev-clean-WER: 440.89 | dev-other-loss: 139.72042 | dev-other-TER: 267.11 | dev-other-WER: 450.61 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 241.13 epoch: 22 | nupdates: 738279 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:16 | bch(ms): 408.38 | smp(ms): 1.19 | fwd(ms): 111.54 | crit-fwd(ms): 7.38 | bwd(ms): 190.58 | optim(ms): 67.74 | loss: 267.99973 | train-TER: 174.86 | train-WER: 324.96 | dev-clean-loss: 159.09193 | dev-clean-TER: 255.34 | dev-clean-WER: 442.33 | dev-other-loss: 139.37328 | dev-other-TER: 267.21 | dev-other-WER: 450.94 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 240.84 epoch: 23 | nupdates: 773435 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:54 | bch(ms): 407.74 | smp(ms): 0.94 | fwd(ms): 111.36 | crit-fwd(ms): 7.29 | bwd(ms): 190.49 | optim(ms): 67.78 | loss: 267.71163 | train-TER: 173.55 | train-WER: 322.70 | dev-clean-loss: 158.95212 | dev-clean-TER: 260.97 | dev-clean-WER: 451.94 | dev-other-loss: 139.14714 | dev-other-TER: 273.32 | dev-other-WER: 461.28 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 107 | hrs: 960.47 | thrpt(sec/sec): 241.21 epoch: 24 | nupdates: 808591 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:45 | bch(ms): 407.50 | smp(ms): 0.83 | fwd(ms): 111.26 | crit-fwd(ms): 7.24 | bwd(ms): 190.56 | optim(ms): 68.17 | loss: 267.35995 | train-TER: 171.47 | train-WER: 319.09 | dev-clean-loss: 159.02618 | dev-clean-TER: 265.78 | dev-clean-WER: 459.61 | dev-other-loss: 139.18588 | dev-other-TER: 280.13 | dev-other-WER: 472.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 241.36 epoch: 25 | nupdates: 843747 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:55 | bch(ms): 407.78 | smp(ms): 0.84 | fwd(ms): 111.29 | crit-fwd(ms): 7.24 | bwd(ms): 190.61 | optim(ms): 68.51 | loss: 267.17583 | train-TER: 170.99 | train-WER: 318.09 | dev-clean-loss: 158.31644 | dev-clean-TER: 258.21 | dev-clean-WER: 447.22 | dev-other-loss: 139.10440 | dev-other-TER: 267.74 | dev-other-WER: 452.26 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 241.19 epoch: 26 | nupdates: 878903 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 04:01:57 | bch(ms): 412.94 | smp(ms): 5.17 | fwd(ms): 111.92 | crit-fwd(ms): 7.48 | bwd(ms): 190.90 | optim(ms): 68.40 | loss: 266.89739 | train-TER: 169.91 | train-WER: 316.00 | dev-clean-loss: 158.25573 | dev-clean-TER: 264.41 | dev-clean-WER: 457.33 | dev-other-loss: 138.90373 | dev-other-TER: 276.28 | dev-other-WER: 466.05 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 238.18 epoch: 27 | nupdates: 914059 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:59:12 | bch(ms): 408.26 | smp(ms): 0.85 | fwd(ms): 111.74 | crit-fwd(ms): 7.39 | bwd(ms): 190.93 | optim(ms): 68.64 | loss: 266.65983 | train-TER: 168.98 | train-WER: 314.41 | dev-clean-loss: 158.35988 | dev-clean-TER: 258.02 | dev-clean-WER: 446.75 | dev-other-loss: 138.84695 | dev-other-TER: 266.04 | dev-other-WER: 449.36 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 240.91 epoch: 28 | nupdates: 949215 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:07 | bch(ms): 406.40 | smp(ms): 0.83 | fwd(ms): 111.01 | crit-fwd(ms): 7.08 | bwd(ms): 190.62 | optim(ms): 68.18 | loss: 266.41760 | train-TER: 167.68 | train-WER: 311.99 | dev-clean-loss: 158.33944 | dev-clean-TER: 262.87 | dev-clean-WER: 454.44 | dev-other-loss: 138.38775 | dev-other-TER: 272.09 | dev-other-WER: 458.76 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 242.01 epoch: 29 | nupdates: 984371 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:48 | bch(ms): 405.87 | smp(ms): 0.83 | fwd(ms): 111.21 | crit-fwd(ms): 7.18 | bwd(ms): 190.57 | optim(ms): 67.90 | loss: 266.27566 | train-TER: 165.56 | train-WER: 308.10 | dev-clean-loss: 158.13176 | dev-clean-TER: 257.40 | dev-clean-WER: 445.37 | dev-other-loss: 138.57675 | dev-other-TER: 262.73 | dev-other-WER: 443.50 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 242.33 epoch: 30 | nupdates: 1019527 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:58 | bch(ms): 406.14 | smp(ms): 0.84 | fwd(ms): 111.45 | crit-fwd(ms): 7.31 | bwd(ms): 190.59 | optim(ms): 67.99 | loss: 266.02431 | train-TER: 164.53 | train-WER: 306.15 | dev-clean-loss: 157.83917 | dev-clean-TER: 262.45 | dev-clean-WER: 453.90 | dev-other-loss: 138.30048 | dev-other-TER: 268.87 | dev-other-WER: 453.87 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 242.16 epoch: 31 | nupdates: 1054683 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:58:06 | bch(ms): 406.37 | smp(ms): 0.83 | fwd(ms): 111.64 | crit-fwd(ms): 7.39 | bwd(ms): 190.85 | optim(ms): 68.03 | loss: 265.81333 | train-TER: 163.17 | train-WER: 303.63 | dev-clean-loss: 157.69846 | dev-clean-TER: 267.31 | dev-clean-WER: 461.75 | dev-other-loss: 138.26481 | dev-other-TER: 276.02 | dev-other-WER: 465.29 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 242.03 epoch: 32 | nupdates: 1089839 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:33 | bch(ms): 405.42 | smp(ms): 0.84 | fwd(ms): 111.31 | crit-fwd(ms): 7.22 | bwd(ms): 190.84 | optim(ms): 68.12 | loss: 265.65036 | train-TER: 160.21 | train-WER: 297.97 | dev-clean-loss: 157.25775 | dev-clean-TER: 262.68 | dev-clean-WER: 453.95 | dev-other-loss: 138.11651 | dev-other-TER: 267.16 | dev-other-WER: 450.61 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 242.59 epoch: 33 | nupdates: 1124995 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:48 | bch(ms): 405.87 | smp(ms): 1.40 | fwd(ms): 111.49 | crit-fwd(ms): 7.33 | bwd(ms): 191.01 | optim(ms): 67.83 | loss: 265.44579 | train-TER: 158.58 | train-WER: 295.08 | dev-clean-loss: 157.44991 | dev-clean-TER: 259.66 | dev-clean-WER: 448.73 | dev-other-loss: 137.96345 | dev-other-TER: 261.35 | dev-other-WER: 440.85 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 242.33 epoch: 34 | nupdates: 1160151 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:15 | bch(ms): 404.93 | smp(ms): 0.83 | fwd(ms): 111.60 | crit-fwd(ms): 7.36 | bwd(ms): 190.85 | optim(ms): 67.97 | loss: 265.26384 | train-TER: 156.59 | train-WER: 291.12 | dev-clean-loss: 157.15497 | dev-clean-TER: 255.27 | dev-clean-WER: 441.08 | dev-other-loss: 138.09927 | dev-other-TER: 255.32 | dev-other-WER: 430.40 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 242.89 epoch: 35 | nupdates: 1195307 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:57:12 | bch(ms): 404.84 | smp(ms): 0.84 | fwd(ms): 111.55 | crit-fwd(ms): 7.33 | bwd(ms): 190.93 | optim(ms): 68.26 | loss: 265.08748 | train-TER: 154.37 | train-WER: 286.52 | dev-clean-loss: 157.30106 | dev-clean-TER: 253.43 | dev-clean-WER: 438.50 | dev-other-loss: 137.63830 | dev-other-TER: 251.99 | dev-other-WER: 425.76 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 242.94 epoch: 36 | nupdates: 1230463 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:56:52 | bch(ms): 404.27 | smp(ms): 1.34 | fwd(ms): 111.58 | crit-fwd(ms): 7.38 | bwd(ms): 190.74 | optim(ms): 67.97 | loss: 264.94989 | train-TER: 152.04 | train-WER: 282.44 | dev-clean-loss: 157.00974 | dev-clean-TER: 259.10 | dev-clean-WER: 447.15 | dev-other-loss: 137.64722 | dev-other-TER: 260.30 | dev-other-WER: 438.31 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 243.29 epoch: 37 | nupdates: 1265619 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:55:59 | bch(ms): 402.76 | smp(ms): 0.82 | fwd(ms): 111.01 | crit-fwd(ms): 7.09 | bwd(ms): 190.63 | optim(ms): 68.02 | loss: 264.72618 | train-TER: 149.94 | train-WER: 278.68 | dev-clean-loss: 157.24842 | dev-clean-TER: 251.44 | dev-clean-WER: 434.27 | dev-other-loss: 137.49073 | dev-other-TER: 248.66 | dev-other-WER: 419.03 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 244.20 epoch: 38 | nupdates: 1300775 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:56:03 | bch(ms): 402.88 | smp(ms): 0.89 | fwd(ms): 111.27 | crit-fwd(ms): 7.22 | bwd(ms): 190.93 | optim(ms): 68.28 | loss: 264.64282 | train-TER: 146.83 | train-WER: 272.31 | dev-clean-loss: 157.04727 | dev-clean-TER: 251.75 | dev-clean-WER: 435.09 | dev-other-loss: 137.50750 | dev-other-TER: 249.57 | dev-other-WER: 420.91 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 244.12 epoch: 39 | nupdates: 1335931 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:56:22 | bch(ms): 403.41 | smp(ms): 0.85 | fwd(ms): 111.50 | crit-fwd(ms): 7.28 | bwd(ms): 191.27 | optim(ms): 68.50 | loss: 264.50003 | train-TER: 144.91 | train-WER: 269.09 | dev-clean-loss: 156.94164 | dev-clean-TER: 249.14 | dev-clean-WER: 430.23 | dev-other-loss: 137.41813 | dev-other-TER: 245.09 | dev-other-WER: 412.66 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 243.80 epoch: 40 | nupdates: 1371087 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:56:24 | bch(ms): 403.48 | smp(ms): 0.85 | fwd(ms): 111.59 | crit-fwd(ms): 7.30 | bwd(ms): 191.31 | optim(ms): 68.80 | loss: 264.32941 | train-TER: 143.21 | train-WER: 265.24 | dev-clean-loss: 156.82633 | dev-clean-TER: 246.52 | dev-clean-WER: 425.50 | dev-other-loss: 137.44124 | dev-other-TER: 242.35 | dev-other-WER: 407.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 243.76 epoch: 41 | nupdates: 1406243 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:56:11 | bch(ms): 403.11 | smp(ms): 1.56 | fwd(ms): 111.66 | crit-fwd(ms): 7.41 | bwd(ms): 191.23 | optim(ms): 68.39 | loss: 264.20440 | train-TER: 139.47 | train-WER: 258.69 | dev-clean-loss: 156.76834 | dev-clean-TER: 238.48 | dev-clean-WER: 411.80 | dev-other-loss: 137.15032 | dev-other-TER: 231.17 | dev-other-WER: 388.78 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 243.99 epoch: 42 | nupdates: 1441399 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:55:09 | bch(ms): 401.34 | smp(ms): 0.85 | fwd(ms): 111.43 | crit-fwd(ms): 7.27 | bwd(ms): 191.12 | optim(ms): 68.43 | loss: 264.05314 | train-TER: 136.28 | train-WER: 252.33 | dev-clean-loss: 156.25451 | dev-clean-TER: 236.36 | dev-clean-WER: 407.17 | dev-other-loss: 137.35367 | dev-other-TER: 230.01 | dev-other-WER: 386.04 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 245.06 epoch: 43 | nupdates: 1476555 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:55:03 | bch(ms): 401.18 | smp(ms): 0.85 | fwd(ms): 111.39 | crit-fwd(ms): 7.26 | bwd(ms): 191.21 | optim(ms): 68.52 | loss: 263.92446 | train-TER: 134.62 | train-WER: 249.15 | dev-clean-loss: 156.34083 | dev-clean-TER: 222.77 | dev-clean-WER: 385.20 | dev-other-loss: 137.07056 | dev-other-TER: 211.87 | dev-other-WER: 356.63 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 245.16 epoch: 44 | nupdates: 1511711 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:55:20 | bch(ms): 401.64 | smp(ms): 1.40 | fwd(ms): 111.57 | crit-fwd(ms): 7.29 | bwd(ms): 191.25 | optim(ms): 68.87 | loss: 263.81132 | train-TER: 131.60 | train-WER: 243.56 | dev-clean-loss: 156.13673 | dev-clean-TER: 224.64 | dev-clean-WER: 386.34 | dev-other-loss: 137.01737 | dev-other-TER: 215.89 | dev-other-WER: 360.75 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 108 | hrs: 960.47 | thrpt(sec/sec): 244.88 epoch: 45 | nupdates: 1546867 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:54:30 | bch(ms): 400.24 | smp(ms): 0.84 | fwd(ms): 111.35 | crit-fwd(ms): 7.20 | bwd(ms): 191.13 | optim(ms): 68.73 | loss: 263.68357 | train-TER: 130.06 | train-WER: 240.18 | dev-clean-loss: 156.15754 | dev-clean-TER: 213.24 | dev-clean-WER: 367.43 | dev-other-loss: 136.83997 | dev-other-TER: 201.39 | dev-other-WER: 336.82 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 245.73 epoch: 46 | nupdates: 1582023 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:54:09 | bch(ms): 399.62 | smp(ms): 0.84 | fwd(ms): 111.49 | crit-fwd(ms): 7.31 | bwd(ms): 191.09 | optim(ms): 68.64 | loss: 263.54657 | train-TER: 126.79 | train-WER: 234.65 | dev-clean-loss: 155.99286 | dev-clean-TER: 216.07 | dev-clean-WER: 371.32 | dev-other-loss: 136.92398 | dev-other-TER: 205.17 | dev-other-WER: 342.33 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 246.11 epoch: 47 | nupdates: 1617179 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:33 | bch(ms): 398.60 | smp(ms): 0.85 | fwd(ms): 111.32 | crit-fwd(ms): 7.23 | bwd(ms): 190.81 | optim(ms): 68.36 | loss: 263.46954 | train-TER: 125.35 | train-WER: 231.10 | dev-clean-loss: 156.05933 | dev-clean-TER: 196.14 | dev-clean-WER: 337.92 | dev-other-loss: 136.52624 | dev-other-TER: 183.28 | dev-other-WER: 305.61 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 109 | hrs: 960.47 | thrpt(sec/sec): 246.75 epoch: 48 | nupdates: 1652335 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:08 | bch(ms): 397.91 | smp(ms): 1.16 | fwd(ms): 111.46 | crit-fwd(ms): 7.30 | bwd(ms): 190.68 | optim(ms): 68.28 | loss: 263.35012 | train-TER: 120.88 | train-WER: 223.31 | dev-clean-loss: 155.96741 | dev-clean-TER: 198.54 | dev-clean-WER: 341.13 | dev-other-loss: 136.54581 | dev-other-TER: 184.13 | dev-other-WER: 306.76 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 247.17 epoch: 49 | nupdates: 1687491 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:28 | bch(ms): 398.46 | smp(ms): 1.30 | fwd(ms): 111.38 | crit-fwd(ms): 7.27 | bwd(ms): 191.14 | optim(ms): 68.33 | loss: 263.22798 | train-TER: 120.83 | train-WER: 222.15 | dev-clean-loss: 155.83255 | dev-clean-TER: 175.22 | dev-clean-WER: 302.32 | dev-other-loss: 136.39798 | dev-other-TER: 159.83 | dev-other-WER: 265.14 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 246.83 epoch: 50 | nupdates: 1722647 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:22 | bch(ms): 398.31 | smp(ms): 0.84 | fwd(ms): 111.96 | crit-fwd(ms): 7.46 | bwd(ms): 191.25 | optim(ms): 68.73 | loss: 263.09735 | train-TER: 116.49 | train-WER: 214.18 | dev-clean-loss: 155.51932 | dev-clean-TER: 166.92 | dev-clean-WER: 287.96 | dev-other-loss: 136.25914 | dev-other-TER: 151.05 | dev-other-WER: 249.51 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 246.92 epoch: 51 | nupdates: 1757803 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:10 | bch(ms): 397.95 | smp(ms): 0.85 | fwd(ms): 111.81 | crit-fwd(ms): 7.41 | bwd(ms): 191.19 | optim(ms): 69.01 | loss: 263.03153 | train-TER: 115.00 | train-WER: 210.58 | dev-clean-loss: 155.49137 | dev-clean-TER: 162.18 | dev-clean-WER: 278.34 | dev-other-loss: 136.26025 | dev-other-TER: 145.05 | dev-other-WER: 238.48 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 247.15 epoch: 52 | nupdates: 1792959 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:52:39 | bch(ms): 397.08 | smp(ms): 0.89 | fwd(ms): 111.93 | crit-fwd(ms): 7.54 | bwd(ms): 191.03 | optim(ms): 68.42 | loss: 262.91839 | train-TER: 113.68 | train-WER: 208.25 | dev-clean-loss: 155.11989 | dev-clean-TER: 150.45 | dev-clean-WER: 258.30 | dev-other-loss: 136.21301 | dev-other-TER: 134.09 | dev-other-WER: 219.06 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 247.69 epoch: 53 | nupdates: 1828115 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:52:21 | bch(ms): 396.55 | smp(ms): 1.01 | fwd(ms): 111.94 | crit-fwd(ms): 7.59 | bwd(ms): 191.08 | optim(ms): 67.87 | loss: 262.80065 | train-TER: 112.19 | train-WER: 205.74 | dev-clean-loss: 155.48579 | dev-clean-TER: 138.46 | dev-clean-WER: 236.92 | dev-other-loss: 136.03214 | dev-other-TER: 123.06 | dev-other-WER: 198.55 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 248.02 epoch: 54 | nupdates: 1863271 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:51:16 | bch(ms): 394.72 | smp(ms): 1.20 | fwd(ms): 111.91 | crit-fwd(ms): 7.57 | bwd(ms): 190.58 | optim(ms): 67.03 | loss: 262.70432 | train-TER: 110.28 | train-WER: 201.13 | dev-clean-loss: 155.40727 | dev-clean-TER: 120.43 | dev-clean-WER: 202.53 | dev-other-loss: 136.24333 | dev-other-TER: 108.92 | dev-other-WER: 170.62 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 107 | hrs: 960.47 | thrpt(sec/sec): 249.17 epoch: 55 | nupdates: 1898427 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:51:02 | bch(ms): 394.32 | smp(ms): 0.84 | fwd(ms): 111.91 | crit-fwd(ms): 7.54 | bwd(ms): 190.63 | optim(ms): 67.21 | loss: 262.66040 | train-TER: 109.39 | train-WER: 198.59 | dev-clean-loss: 155.32088 | dev-clean-TER: 121.19 | dev-clean-WER: 204.33 | dev-other-loss: 136.07195 | dev-other-TER: 109.23 | dev-other-WER: 171.03 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 249.42 epoch: 56 | nupdates: 1933583 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:51:15 | bch(ms): 394.69 | smp(ms): 0.83 | fwd(ms): 111.46 | crit-fwd(ms): 7.28 | bwd(ms): 190.78 | optim(ms): 68.15 | loss: 262.54934 | train-TER: 108.17 | train-WER: 195.99 | dev-clean-loss: 155.33469 | dev-clean-TER: 115.00 | dev-clean-WER: 191.11 | dev-other-loss: 136.31209 | dev-other-TER: 104.57 | dev-other-WER: 160.46 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 249.19 epoch: 57 | nupdates: 1968739 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:51:07 | bch(ms): 394.45 | smp(ms): 0.83 | fwd(ms): 111.79 | crit-fwd(ms): 7.46 | bwd(ms): 190.66 | optim(ms): 67.89 | loss: 262.49295 | train-TER: 107.05 | train-WER: 194.12 | dev-clean-loss: 155.23888 | dev-clean-TER: 101.85 | dev-clean-WER: 163.77 | dev-other-loss: 136.47731 | dev-other-TER: 95.33 | dev-other-WER: 140.18 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 249.34 epoch: 58 | nupdates: 2003895 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:24 | bch(ms): 393.22 | smp(ms): 0.84 | fwd(ms): 112.09 | crit-fwd(ms): 7.64 | bwd(ms): 190.48 | optim(ms): 67.03 | loss: 262.37333 | train-TER: 105.10 | train-WER: 189.56 | dev-clean-loss: 155.56919 | dev-clean-TER: 99.54 | dev-clean-WER: 157.12 | dev-other-loss: 136.81476 | dev-other-TER: 94.01 | dev-other-WER: 135.09 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 250.12 epoch: 59 | nupdates: 2039051 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:15 | bch(ms): 392.99 | smp(ms): 0.89 | fwd(ms): 111.84 | crit-fwd(ms): 7.49 | bwd(ms): 190.52 | optim(ms): 67.19 | loss: 262.29969 | train-TER: 104.72 | train-WER: 187.61 | dev-clean-loss: 155.69976 | dev-clean-TER: 89.20 | dev-clean-WER: 132.42 | dev-other-loss: 137.19604 | dev-other-TER: 87.89 | dev-other-WER: 118.95 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 250.27 epoch: 60 | nupdates: 2074207 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:12 | bch(ms): 392.89 | smp(ms): 0.82 | fwd(ms): 111.61 | crit-fwd(ms): 7.36 | bwd(ms): 190.52 | optim(ms): 67.89 | loss: 262.15188 | train-TER: 102.63 | train-WER: 183.59 | dev-clean-loss: 156.00474 | dev-clean-TER: 86.63 | dev-clean-WER: 122.56 | dev-other-loss: 138.01695 | dev-other-TER: 87.13 | dev-other-WER: 112.43 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 250.33 epoch: 61 | nupdates: 2109363 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:21 | bch(ms): 393.14 | smp(ms): 0.83 | fwd(ms): 111.70 | crit-fwd(ms): 7.35 | bwd(ms): 190.69 | optim(ms): 68.28 | loss: 262.08771 | train-TER: 100.49 | train-WER: 179.25 | dev-clean-loss: 156.11602 | dev-clean-TER: 87.14 | dev-clean-WER: 122.78 | dev-other-loss: 138.19123 | dev-other-TER: 88.16 | dev-other-WER: 113.35 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 250.17 epoch: 62 | nupdates: 2144519 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:45 | bch(ms): 392.11 | smp(ms): 0.83 | fwd(ms): 111.32 | crit-fwd(ms): 7.18 | bwd(ms): 190.70 | optim(ms): 68.19 | loss: 262.00544 | train-TER: 98.73 | train-WER: 175.00 | dev-clean-loss: 157.03518 | dev-clean-TER: 84.37 | dev-clean-WER: 111.11 | dev-other-loss: 139.27559 | dev-other-TER: 87.09 | dev-other-WER: 105.77 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 250.83 epoch: 63 | nupdates: 2179675 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:42 | bch(ms): 392.04 | smp(ms): 0.83 | fwd(ms): 111.45 | crit-fwd(ms): 7.25 | bwd(ms): 190.45 | optim(ms): 68.30 | loss: 261.88611 | train-TER: 99.04 | train-WER: 174.97 | dev-clean-loss: 158.18390 | dev-clean-TER: 84.73 | dev-clean-WER: 105.09 | dev-other-loss: 141.32088 | dev-other-TER: 87.92 | dev-other-WER: 101.99 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.88 epoch: 64 | nupdates: 2214831 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:29 | bch(ms): 391.66 | smp(ms): 0.83 | fwd(ms): 111.75 | crit-fwd(ms): 7.43 | bwd(ms): 190.39 | optim(ms): 67.80 | loss: 261.86054 | train-TER: 97.88 | train-WER: 172.75 | dev-clean-loss: 159.23077 | dev-clean-TER: 85.01 | dev-clean-WER: 102.95 | dev-other-loss: 142.36364 | dev-other-TER: 88.55 | dev-other-WER: 101.02 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 251.12 epoch: 65 | nupdates: 2249987 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:15 | bch(ms): 391.27 | smp(ms): 0.84 | fwd(ms): 111.92 | crit-fwd(ms): 7.59 | bwd(ms): 190.52 | optim(ms): 67.03 | loss: 261.79236 | train-TER: 98.74 | train-WER: 173.88 | dev-clean-loss: 159.85490 | dev-clean-TER: 85.65 | dev-clean-WER: 100.48 | dev-other-loss: 143.88864 | dev-other-TER: 89.25 | dev-other-WER: 99.70 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 251.37 epoch: 66 | nupdates: 2285143 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:17 | bch(ms): 391.33 | smp(ms): 0.91 | fwd(ms): 111.86 | crit-fwd(ms): 7.54 | bwd(ms): 190.68 | optim(ms): 67.26 | loss: 261.65671 | train-TER: 97.31 | train-WER: 170.93 | dev-clean-loss: 161.21840 | dev-clean-TER: 86.98 | dev-clean-WER: 98.59 | dev-other-loss: 145.18651 | dev-other-TER: 90.39 | dev-other-WER: 98.54 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 251.33 epoch: 67 | nupdates: 2320299 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:18 | bch(ms): 391.36 | smp(ms): 0.84 | fwd(ms): 111.80 | crit-fwd(ms): 7.45 | bwd(ms): 190.66 | optim(ms): 67.85 | loss: 261.62688 | train-TER: 95.86 | train-WER: 167.45 | dev-clean-loss: 162.51443 | dev-clean-TER: 87.96 | dev-clean-WER: 98.55 | dev-other-loss: 146.84209 | dev-other-TER: 91.10 | dev-other-WER: 98.46 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 251.31 epoch: 68 | nupdates: 2355455 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:00 | bch(ms): 390.84 | smp(ms): 0.83 | fwd(ms): 111.72 | crit-fwd(ms): 7.42 | bwd(ms): 190.46 | optim(ms): 67.78 | loss: 261.53104 | train-TER: 95.69 | train-WER: 166.96 | dev-clean-loss: 163.97196 | dev-clean-TER: 89.29 | dev-clean-WER: 97.16 | dev-other-loss: 148.08402 | dev-other-TER: 92.17 | dev-other-WER: 97.80 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 251.64 epoch: 69 | nupdates: 2390611 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:44 | bch(ms): 390.40 | smp(ms): 0.83 | fwd(ms): 111.89 | crit-fwd(ms): 7.59 | bwd(ms): 190.53 | optim(ms): 67.30 | loss: 261.45357 | train-TER: 94.26 | train-WER: 164.21 | dev-clean-loss: 165.44775 | dev-clean-TER: 90.43 | dev-clean-WER: 97.06 | dev-other-loss: 149.57780 | dev-other-TER: 93.06 | dev-other-WER: 97.91 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 251.93 epoch: 70 | nupdates: 2425767 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:30 | bch(ms): 390.00 | smp(ms): 0.82 | fwd(ms): 111.99 | crit-fwd(ms): 7.60 | bwd(ms): 190.59 | optim(ms): 66.83 | loss: 261.34970 | train-TER: 94.62 | train-WER: 163.62 | dev-clean-loss: 166.61415 | dev-clean-TER: 91.37 | dev-clean-WER: 96.85 | dev-other-loss: 150.78956 | dev-other-TER: 93.70 | dev-other-WER: 97.71 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 252.19 epoch: 71 | nupdates: 2460923 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:57 | bch(ms): 390.75 | smp(ms): 0.93 | fwd(ms): 111.94 | crit-fwd(ms): 7.55 | bwd(ms): 190.62 | optim(ms): 67.66 | loss: 261.29525 | train-TER: 94.23 | train-WER: 162.09 | dev-clean-loss: 166.95464 | dev-clean-TER: 91.81 | dev-clean-WER: 97.21 | dev-other-loss: 150.47135 | dev-other-TER: 94.06 | dev-other-WER: 97.91 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 251.70 epoch: 72 | nupdates: 2496079 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:16 | bch(ms): 391.31 | smp(ms): 0.84 | fwd(ms): 111.62 | crit-fwd(ms): 7.35 | bwd(ms): 190.85 | optim(ms): 68.57 | loss: 261.25106 | train-TER: 93.41 | train-WER: 160.65 | dev-clean-loss: 167.29638 | dev-clean-TER: 92.49 | dev-clean-WER: 96.98 | dev-other-loss: 150.46426 | dev-other-TER: 94.63 | dev-other-WER: 97.89 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 251.34 epoch: 73 | nupdates: 2531235 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:37 | bch(ms): 391.90 | smp(ms): 0.85 | fwd(ms): 111.67 | crit-fwd(ms): 7.35 | bwd(ms): 191.13 | optim(ms): 68.96 | loss: 261.18468 | train-TER: 92.29 | train-WER: 159.03 | dev-clean-loss: 167.77998 | dev-clean-TER: 92.77 | dev-clean-WER: 97.01 | dev-other-loss: 150.82855 | dev-other-TER: 94.85 | dev-other-WER: 97.88 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.96 epoch: 74 | nupdates: 2566391 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:57 | bch(ms): 390.77 | smp(ms): 0.85 | fwd(ms): 111.47 | crit-fwd(ms): 7.28 | bwd(ms): 190.69 | optim(ms): 68.52 | loss: 261.13077 | train-TER: 92.54 | train-WER: 159.11 | dev-clean-loss: 167.71907 | dev-clean-TER: 93.32 | dev-clean-WER: 96.97 | dev-other-loss: 150.43553 | dev-other-TER: 95.27 | dev-other-WER: 97.92 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 251.69 epoch: 75 | nupdates: 2601547 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:13 | bch(ms): 389.51 | smp(ms): 0.84 | fwd(ms): 111.38 | crit-fwd(ms): 7.25 | bwd(ms): 190.45 | optim(ms): 68.23 | loss: 261.06694 | train-TER: 90.13 | train-WER: 154.19 | dev-clean-loss: 166.59511 | dev-clean-TER: 92.94 | dev-clean-WER: 97.10 | dev-other-loss: 148.80614 | dev-other-TER: 94.97 | dev-other-WER: 98.00 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 252.50 epoch: 76 | nupdates: 2636703 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:40 | bch(ms): 388.57 | smp(ms): 0.82 | fwd(ms): 111.52 | crit-fwd(ms): 7.37 | bwd(ms): 190.10 | optim(ms): 67.46 | loss: 260.98578 | train-TER: 90.73 | train-WER: 154.36 | dev-clean-loss: 167.49420 | dev-clean-TER: 93.68 | dev-clean-WER: 97.07 | dev-other-loss: 149.74775 | dev-other-TER: 95.55 | dev-other-WER: 98.05 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 253.11 epoch: 77 | nupdates: 2671859 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:28 | bch(ms): 388.23 | smp(ms): 0.82 | fwd(ms): 111.31 | crit-fwd(ms): 7.21 | bwd(ms): 190.04 | optim(ms): 67.54 | loss: 260.88555 | train-TER: 90.14 | train-WER: 153.49 | dev-clean-loss: 167.35766 | dev-clean-TER: 93.69 | dev-clean-WER: 97.10 | dev-other-loss: 149.37206 | dev-other-TER: 95.61 | dev-other-WER: 98.09 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.34 epoch: 78 | nupdates: 2707015 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:26 | bch(ms): 398.40 | smp(ms): 9.98 | fwd(ms): 112.48 | crit-fwd(ms): 7.75 | bwd(ms): 189.96 | optim(ms): 67.65 | loss: 260.86193 | train-TER: 88.94 | train-WER: 150.51 | dev-clean-loss: 167.09488 | dev-clean-TER: 93.73 | dev-clean-WER: 97.06 | dev-other-loss: 148.49668 | dev-other-TER: 95.62 | dev-other-WER: 98.07 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 246.87 epoch: 79 | nupdates: 2742171 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:31 | bch(ms): 388.31 | smp(ms): 0.82 | fwd(ms): 111.38 | crit-fwd(ms): 7.26 | bwd(ms): 190.16 | optim(ms): 67.66 | loss: 260.78765 | train-TER: 89.50 | train-WER: 151.32 | dev-clean-loss: 166.98801 | dev-clean-TER: 94.00 | dev-clean-WER: 97.19 | dev-other-loss: 148.57394 | dev-other-TER: 95.88 | dev-other-WER: 98.14 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 253.28 epoch: 80 | nupdates: 2777327 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:15 | bch(ms): 387.85 | smp(ms): 0.82 | fwd(ms): 111.23 | crit-fwd(ms): 7.14 | bwd(ms): 190.25 | optim(ms): 67.76 | loss: 260.72257 | train-TER: 87.57 | train-WER: 147.22 | dev-clean-loss: 165.71631 | dev-clean-TER: 93.88 | dev-clean-WER: 97.16 | dev-other-loss: 147.62807 | dev-other-TER: 95.80 | dev-other-WER: 98.15 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.59 epoch: 81 | nupdates: 2812483 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:54 | bch(ms): 387.25 | smp(ms): 0.82 | fwd(ms): 111.22 | crit-fwd(ms): 7.22 | bwd(ms): 190.43 | optim(ms): 67.04 | loss: 260.66833 | train-TER: 87.82 | train-WER: 147.18 | dev-clean-loss: 165.80330 | dev-clean-TER: 93.88 | dev-clean-WER: 97.02 | dev-other-loss: 146.97341 | dev-other-TER: 95.77 | dev-other-WER: 98.12 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 253.97 epoch: 82 | nupdates: 2847639 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:18 | bch(ms): 387.95 | smp(ms): 1.19 | fwd(ms): 111.78 | crit-fwd(ms): 7.50 | bwd(ms): 190.55 | optim(ms): 66.64 | loss: 260.59017 | train-TER: 87.50 | train-WER: 146.58 | dev-clean-loss: 165.30953 | dev-clean-TER: 93.99 | dev-clean-WER: 96.94 | dev-other-loss: 146.64897 | dev-other-TER: 95.89 | dev-other-WER: 98.12 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.52 epoch: 83 | nupdates: 2882795 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:34 | bch(ms): 386.70 | smp(ms): 0.82 | fwd(ms): 111.56 | crit-fwd(ms): 7.39 | bwd(ms): 190.12 | optim(ms): 66.61 | loss: 260.52372 | train-TER: 86.82 | train-WER: 145.48 | dev-clean-loss: 165.39087 | dev-clean-TER: 94.19 | dev-clean-WER: 97.05 | dev-other-loss: 146.77371 | dev-other-TER: 95.99 | dev-other-WER: 98.17 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 254.34 epoch: 84 | nupdates: 2917951 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:09 | bch(ms): 385.97 | smp(ms): 0.80 | fwd(ms): 111.35 | crit-fwd(ms): 7.29 | bwd(ms): 189.82 | optim(ms): 66.72 | loss: 260.43491 | train-TER: 85.85 | train-WER: 143.12 | dev-clean-loss: 164.90279 | dev-clean-TER: 94.12 | dev-clean-WER: 97.02 | dev-other-loss: 145.50106 | dev-other-TER: 95.97 | dev-other-WER: 98.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 107 | hrs: 960.47 | thrpt(sec/sec): 254.82 epoch: 85 | nupdates: 2953107 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:20 | bch(ms): 386.29 | smp(ms): 0.81 | fwd(ms): 111.47 | crit-fwd(ms): 7.30 | bwd(ms): 190.04 | optim(ms): 66.79 | loss: 260.41393 | train-TER: 85.62 | train-WER: 142.22 | dev-clean-loss: 164.30638 | dev-clean-TER: 94.09 | dev-clean-WER: 96.77 | dev-other-loss: 145.04400 | dev-other-TER: 95.92 | dev-other-WER: 98.04 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 254.61 epoch: 86 | nupdates: 2988263 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:30 | bch(ms): 388.28 | smp(ms): 0.83 | fwd(ms): 111.80 | crit-fwd(ms): 7.45 | bwd(ms): 190.71 | optim(ms): 67.44 | loss: 260.34703 | train-TER: 86.28 | train-WER: 143.44 | dev-clean-loss: 165.07110 | dev-clean-TER: 94.65 | dev-clean-WER: 96.90 | dev-other-loss: 145.75688 | dev-other-TER: 96.33 | dev-other-WER: 98.11 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.30 epoch: 87 | nupdates: 3023419 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:00 | bch(ms): 389.14 | smp(ms): 0.87 | fwd(ms): 111.85 | crit-fwd(ms): 7.46 | bwd(ms): 191.29 | optim(ms): 68.07 | loss: 260.28478 | train-TER: 84.43 | train-WER: 139.54 | dev-clean-loss: 163.27011 | dev-clean-TER: 93.86 | dev-clean-WER: 96.71 | dev-other-loss: 144.23765 | dev-other-TER: 95.82 | dev-other-WER: 98.05 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 252.75 epoch: 88 | nupdates: 3058575 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:58 | bch(ms): 389.07 | smp(ms): 1.18 | fwd(ms): 111.88 | crit-fwd(ms): 7.50 | bwd(ms): 191.18 | optim(ms): 67.84 | loss: 260.22713 | train-TER: 84.46 | train-WER: 139.56 | dev-clean-loss: 163.29060 | dev-clean-TER: 94.25 | dev-clean-WER: 96.73 | dev-other-loss: 144.52239 | dev-other-TER: 96.07 | dev-other-WER: 98.07 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.79 epoch: 89 | nupdates: 3093731 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:43 | bch(ms): 388.66 | smp(ms): 0.85 | fwd(ms): 111.65 | crit-fwd(ms): 7.39 | bwd(ms): 191.00 | optim(ms): 68.18 | loss: 260.15900 | train-TER: 84.96 | train-WER: 139.74 | dev-clean-loss: 162.92082 | dev-clean-TER: 94.09 | dev-clean-WER: 96.65 | dev-other-loss: 143.75726 | dev-other-TER: 95.94 | dev-other-WER: 98.02 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 253.06 epoch: 90 | nupdates: 3128887 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:46 | bch(ms): 388.75 | smp(ms): 0.84 | fwd(ms): 111.66 | crit-fwd(ms): 7.38 | bwd(ms): 190.96 | optim(ms): 68.38 | loss: 260.15623 | train-TER: 84.63 | train-WER: 139.22 | dev-clean-loss: 162.62590 | dev-clean-TER: 93.97 | dev-clean-WER: 96.56 | dev-other-loss: 143.44386 | dev-other-TER: 95.86 | dev-other-WER: 97.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.00 epoch: 91 | nupdates: 3164043 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:47 | bch(ms): 388.76 | smp(ms): 0.84 | fwd(ms): 111.90 | crit-fwd(ms): 7.48 | bwd(ms): 191.03 | optim(ms): 68.21 | loss: 260.08566 | train-TER: 84.14 | train-WER: 137.79 | dev-clean-loss: 162.29835 | dev-clean-TER: 94.05 | dev-clean-WER: 96.41 | dev-other-loss: 143.09313 | dev-other-TER: 95.93 | dev-other-WER: 97.95 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.99 epoch: 92 | nupdates: 3199199 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:14 | bch(ms): 389.54 | smp(ms): 0.86 | fwd(ms): 111.94 | crit-fwd(ms): 7.46 | bwd(ms): 191.34 | optim(ms): 68.86 | loss: 260.01662 | train-TER: 82.87 | train-WER: 134.86 | dev-clean-loss: 162.04375 | dev-clean-TER: 94.08 | dev-clean-WER: 96.46 | dev-other-loss: 142.87217 | dev-other-TER: 95.92 | dev-other-WER: 97.98 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 252.48 epoch: 93 | nupdates: 3234355 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:41 | bch(ms): 388.60 | smp(ms): 0.84 | fwd(ms): 111.51 | crit-fwd(ms): 7.26 | bwd(ms): 191.13 | optim(ms): 68.63 | loss: 260.00366 | train-TER: 82.93 | train-WER: 135.21 | dev-clean-loss: 162.12996 | dev-clean-TER: 94.15 | dev-clean-WER: 96.39 | dev-other-loss: 142.69295 | dev-other-TER: 96.01 | dev-other-WER: 97.96 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.10 epoch: 94 | nupdates: 3269511 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:11 | bch(ms): 387.74 | smp(ms): 0.84 | fwd(ms): 111.58 | crit-fwd(ms): 7.32 | bwd(ms): 190.74 | optim(ms): 68.28 | loss: 259.92158 | train-TER: 81.81 | train-WER: 132.99 | dev-clean-loss: 161.31622 | dev-clean-TER: 93.75 | dev-clean-WER: 96.31 | dev-other-loss: 141.85798 | dev-other-TER: 95.66 | dev-other-WER: 97.82 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.66 epoch: 95 | nupdates: 3304667 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:48 | bch(ms): 385.40 | smp(ms): 0.82 | fwd(ms): 111.60 | crit-fwd(ms): 7.37 | bwd(ms): 189.85 | optim(ms): 67.27 | loss: 259.85186 | train-TER: 80.94 | train-WER: 130.38 | dev-clean-loss: 161.72460 | dev-clean-TER: 94.00 | dev-clean-WER: 96.39 | dev-other-loss: 142.38152 | dev-other-TER: 95.84 | dev-other-WER: 97.92 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 255.20 epoch: 96 | nupdates: 3339823 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:54 | bch(ms): 388.98 | smp(ms): 1.58 | fwd(ms): 111.36 | crit-fwd(ms): 7.24 | bwd(ms): 190.92 | optim(ms): 68.78 | loss: 259.83224 | train-TER: 82.50 | train-WER: 134.06 | dev-clean-loss: 161.25779 | dev-clean-TER: 94.01 | dev-clean-WER: 96.29 | dev-other-loss: 142.05668 | dev-other-TER: 95.85 | dev-other-WER: 97.88 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.85 epoch: 97 | nupdates: 3374979 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:28 | bch(ms): 388.23 | smp(ms): 0.85 | fwd(ms): 111.81 | crit-fwd(ms): 7.44 | bwd(ms): 191.04 | optim(ms): 68.45 | loss: 259.74877 | train-TER: 81.38 | train-WER: 131.66 | dev-clean-loss: 160.57660 | dev-clean-TER: 93.40 | dev-clean-WER: 96.10 | dev-other-loss: 141.10853 | dev-other-TER: 95.38 | dev-other-WER: 97.71 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.33 epoch: 98 | nupdates: 3410135 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:28 | bch(ms): 388.23 | smp(ms): 0.84 | fwd(ms): 112.02 | crit-fwd(ms): 7.53 | bwd(ms): 191.05 | optim(ms): 68.24 | loss: 259.72790 | train-TER: 81.14 | train-WER: 131.21 | dev-clean-loss: 161.12000 | dev-clean-TER: 93.87 | dev-clean-WER: 96.12 | dev-other-loss: 141.24080 | dev-other-TER: 95.76 | dev-other-WER: 97.82 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.34 epoch: 99 | nupdates: 3445291 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:25 | bch(ms): 386.43 | smp(ms): 0.85 | fwd(ms): 111.22 | crit-fwd(ms): 7.30 | bwd(ms): 190.23 | optim(ms): 68.01 | loss: 259.67564 | train-TER: 82.00 | train-WER: 132.53 | dev-clean-loss: 160.31366 | dev-clean-TER: 93.48 | dev-clean-WER: 95.93 | dev-other-loss: 140.73836 | dev-other-TER: 95.40 | dev-other-WER: 97.63 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 115 | hrs: 960.47 | thrpt(sec/sec): 254.52 epoch: 100 | nupdates: 3480447 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:29 | bch(ms): 386.56 | smp(ms): 0.85 | fwd(ms): 111.75 | crit-fwd(ms): 7.53 | bwd(ms): 190.41 | optim(ms): 67.69 | loss: 259.59484 | train-TER: 80.73 | train-WER: 129.88 | dev-clean-loss: 160.14576 | dev-clean-TER: 93.51 | dev-clean-WER: 95.90 | dev-other-loss: 140.57077 | dev-other-TER: 95.44 | dev-other-WER: 97.61 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 254.43 epoch: 101 | nupdates: 3515603 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:53 | bch(ms): 387.23 | smp(ms): 0.84 | fwd(ms): 112.20 | crit-fwd(ms): 7.70 | bwd(ms): 190.82 | optim(ms): 67.22 | loss: 259.59996 | train-TER: 81.46 | train-WER: 131.64 | dev-clean-loss: 160.19224 | dev-clean-TER: 93.61 | dev-clean-WER: 95.98 | dev-other-loss: 140.77480 | dev-other-TER: 95.50 | dev-other-WER: 97.71 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.99 epoch: 102 | nupdates: 3550759 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:41 | bch(ms): 386.89 | smp(ms): 0.84 | fwd(ms): 111.63 | crit-fwd(ms): 7.38 | bwd(ms): 190.73 | optim(ms): 67.90 | loss: 259.49187 | train-TER: 80.64 | train-WER: 129.66 | dev-clean-loss: 159.54582 | dev-clean-TER: 93.27 | dev-clean-WER: 95.78 | dev-other-loss: 140.12226 | dev-other-TER: 95.22 | dev-other-WER: 97.55 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 254.22 epoch: 103 | nupdates: 3585915 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:57 | bch(ms): 387.34 | smp(ms): 0.84 | fwd(ms): 111.79 | crit-fwd(ms): 7.46 | bwd(ms): 190.83 | optim(ms): 68.11 | loss: 259.44423 | train-TER: 80.32 | train-WER: 128.80 | dev-clean-loss: 159.33418 | dev-clean-TER: 93.15 | dev-clean-WER: 95.69 | dev-other-loss: 140.03028 | dev-other-TER: 95.04 | dev-other-WER: 97.46 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.92 epoch: 104 | nupdates: 3621071 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:01 | bch(ms): 387.46 | smp(ms): 0.85 | fwd(ms): 111.83 | crit-fwd(ms): 7.43 | bwd(ms): 191.01 | optim(ms): 68.14 | loss: 259.37726 | train-TER: 79.75 | train-WER: 127.66 | dev-clean-loss: 159.42352 | dev-clean-TER: 93.10 | dev-clean-WER: 95.66 | dev-other-loss: 139.54446 | dev-other-TER: 94.96 | dev-other-WER: 97.44 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.84 epoch: 105 | nupdates: 3656227 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:50 | bch(ms): 387.14 | smp(ms): 0.85 | fwd(ms): 111.66 | crit-fwd(ms): 7.37 | bwd(ms): 190.75 | optim(ms): 68.29 | loss: 259.38499 | train-TER: 79.94 | train-WER: 127.68 | dev-clean-loss: 159.12320 | dev-clean-TER: 93.08 | dev-clean-WER: 95.65 | dev-other-loss: 139.55264 | dev-other-TER: 94.96 | dev-other-WER: 97.44 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 254.05 epoch: 106 | nupdates: 3691383 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:15 | bch(ms): 387.87 | smp(ms): 1.09 | fwd(ms): 111.52 | crit-fwd(ms): 7.29 | bwd(ms): 190.91 | optim(ms): 68.62 | loss: 259.35301 | train-TER: 80.80 | train-WER: 128.80 | dev-clean-loss: 158.86960 | dev-clean-TER: 93.02 | dev-clean-WER: 95.56 | dev-other-loss: 139.62160 | dev-other-TER: 94.85 | dev-other-WER: 97.31 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.57 epoch: 107 | nupdates: 3726539 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:56 | bch(ms): 387.32 | smp(ms): 0.84 | fwd(ms): 111.91 | crit-fwd(ms): 7.52 | bwd(ms): 190.90 | optim(ms): 68.19 | loss: 259.28721 | train-TER: 79.28 | train-WER: 126.50 | dev-clean-loss: 159.04308 | dev-clean-TER: 93.26 | dev-clean-WER: 95.57 | dev-other-loss: 139.75505 | dev-other-TER: 95.05 | dev-other-WER: 97.33 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.93 epoch: 108 | nupdates: 3761695 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:54 | bch(ms): 387.27 | smp(ms): 0.85 | fwd(ms): 111.40 | crit-fwd(ms): 7.25 | bwd(ms): 190.87 | optim(ms): 68.65 | loss: 259.26876 | train-TER: 79.27 | train-WER: 126.39 | dev-clean-loss: 158.61104 | dev-clean-TER: 92.85 | dev-clean-WER: 95.47 | dev-other-loss: 139.03558 | dev-other-TER: 94.71 | dev-other-WER: 97.22 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.97 epoch: 109 | nupdates: 3796851 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:06 | bch(ms): 387.60 | smp(ms): 0.98 | fwd(ms): 111.54 | crit-fwd(ms): 7.29 | bwd(ms): 190.78 | optim(ms): 68.73 | loss: 259.22552 | train-TER: 79.64 | train-WER: 127.18 | dev-clean-loss: 158.71878 | dev-clean-TER: 92.78 | dev-clean-WER: 95.41 | dev-other-loss: 139.18106 | dev-other-TER: 94.56 | dev-other-WER: 97.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.75 epoch: 110 | nupdates: 3832007 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:16 | bch(ms): 387.87 | smp(ms): 0.85 | fwd(ms): 111.53 | crit-fwd(ms): 7.27 | bwd(ms): 191.06 | optim(ms): 68.85 | loss: 259.18394 | train-TER: 79.89 | train-WER: 127.55 | dev-clean-loss: 158.49532 | dev-clean-TER: 92.69 | dev-clean-WER: 95.38 | dev-other-loss: 139.02144 | dev-other-TER: 94.51 | dev-other-WER: 97.11 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.57 epoch: 111 | nupdates: 3867163 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:09 | bch(ms): 387.67 | smp(ms): 0.84 | fwd(ms): 111.58 | crit-fwd(ms): 7.29 | bwd(ms): 190.88 | optim(ms): 68.84 | loss: 259.10165 | train-TER: 79.07 | train-WER: 126.22 | dev-clean-loss: 158.58265 | dev-clean-TER: 92.96 | dev-clean-WER: 95.43 | dev-other-loss: 138.68741 | dev-other-TER: 94.66 | dev-other-WER: 97.10 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 253.70 epoch: 112 | nupdates: 3902319 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:19 | bch(ms): 386.27 | smp(ms): 0.83 | fwd(ms): 111.40 | crit-fwd(ms): 7.24 | bwd(ms): 190.58 | optim(ms): 68.06 | loss: 259.05985 | train-TER: 79.28 | train-WER: 126.14 | dev-clean-loss: 157.63804 | dev-clean-TER: 92.07 | dev-clean-WER: 95.12 | dev-other-loss: 138.23064 | dev-other-TER: 94.03 | dev-other-WER: 96.90 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 254.62 epoch: 113 | nupdates: 3937475 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:56 | bch(ms): 385.62 | smp(ms): 0.83 | fwd(ms): 111.55 | crit-fwd(ms): 7.38 | bwd(ms): 190.32 | optim(ms): 67.64 | loss: 258.99092 | train-TER: 78.42 | train-WER: 125.14 | dev-clean-loss: 158.16593 | dev-clean-TER: 92.43 | dev-clean-WER: 95.17 | dev-other-loss: 138.53234 | dev-other-TER: 94.26 | dev-other-WER: 96.93 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 255.05 epoch: 114 | nupdates: 3972631 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:48 | bch(ms): 385.38 | smp(ms): 0.82 | fwd(ms): 111.33 | crit-fwd(ms): 7.26 | bwd(ms): 190.20 | optim(ms): 67.70 | loss: 258.99155 | train-TER: 79.17 | train-WER: 125.78 | dev-clean-loss: 157.84247 | dev-clean-TER: 92.08 | dev-clean-WER: 95.03 | dev-other-loss: 137.97870 | dev-other-TER: 93.94 | dev-other-WER: 96.83 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 255.21 epoch: 115 | nupdates: 4007787 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:21 | bch(ms): 386.33 | smp(ms): 0.84 | fwd(ms): 111.60 | crit-fwd(ms): 7.37 | bwd(ms): 190.38 | optim(ms): 68.03 | loss: 258.92076 | train-TER: 79.04 | train-WER: 126.08 | dev-clean-loss: 157.71754 | dev-clean-TER: 91.96 | dev-clean-WER: 94.97 | dev-other-loss: 137.97817 | dev-other-TER: 93.85 | dev-other-WER: 96.79 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 110 | hrs: 960.47 | thrpt(sec/sec): 254.58 epoch: 116 | nupdates: 4042943 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:16 | bch(ms): 386.17 | smp(ms): 0.84 | fwd(ms): 112.06 | crit-fwd(ms): 7.63 | bwd(ms): 190.61 | optim(ms): 67.33 | loss: 258.89442 | train-TER: 78.44 | train-WER: 124.78 | dev-clean-loss: 157.52986 | dev-clean-TER: 91.80 | dev-clean-WER: 94.91 | dev-other-loss: 137.74699 | dev-other-TER: 93.69 | dev-other-WER: 96.72 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 254.69 epoch: 117 | nupdates: 4078099 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:39 | bch(ms): 386.84 | smp(ms): 0.84 | fwd(ms): 112.16 | crit-fwd(ms): 7.65 | bwd(ms): 190.65 | optim(ms): 67.63 | loss: 258.84706 | train-TER: 79.46 | train-WER: 126.34 | dev-clean-loss: 157.41823 | dev-clean-TER: 91.91 | dev-clean-WER: 94.83 | dev-other-loss: 137.80123 | dev-other-TER: 93.72 | dev-other-WER: 96.64 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 254.25 epoch: 118 | nupdates: 4113255 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:16 | bch(ms): 386.18 | smp(ms): 0.83 | fwd(ms): 112.08 | crit-fwd(ms): 7.65 | bwd(ms): 190.42 | optim(ms): 67.09 | loss: 258.83961 | train-TER: 80.18 | train-WER: 128.12 | dev-clean-loss: 156.99349 | dev-clean-TER: 91.67 | dev-clean-WER: 94.83 | dev-other-loss: 137.72929 | dev-other-TER: 93.53 | dev-other-WER: 96.61 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 254.68 epoch: 119 | nupdates: 4148411 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:18 | bch(ms): 386.25 | smp(ms): 0.83 | fwd(ms): 111.72 | crit-fwd(ms): 7.44 | bwd(ms): 190.44 | optim(ms): 67.61 | loss: 258.78283 | train-TER: 79.88 | train-WER: 127.08 | dev-clean-loss: 157.02879 | dev-clean-TER: 91.45 | dev-clean-WER: 94.82 | dev-other-loss: 137.52548 | dev-other-TER: 93.37 | dev-other-WER: 96.63 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 254.64 epoch: 120 | nupdates: 4183567 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:43 | bch(ms): 385.23 | smp(ms): 0.82 | fwd(ms): 111.65 | crit-fwd(ms): 7.39 | bwd(ms): 189.95 | optim(ms): 67.21 | loss: 258.72236 | train-TER: 79.76 | train-WER: 127.18 | dev-clean-loss: 156.98670 | dev-clean-TER: 91.46 | dev-clean-WER: 94.73 | dev-other-loss: 137.49244 | dev-other-TER: 93.34 | dev-other-WER: 96.52 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 255.31 epoch: 121 | nupdates: 4218723 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:44 | bch(ms): 385.27 | smp(ms): 1.18 | fwd(ms): 111.47 | crit-fwd(ms): 7.31 | bwd(ms): 189.84 | optim(ms): 67.32 | loss: 258.72927 | train-TER: 79.44 | train-WER: 126.46 | dev-clean-loss: 156.54345 | dev-clean-TER: 90.93 | dev-clean-WER: 94.58 | dev-other-loss: 137.10734 | dev-other-TER: 92.84 | dev-other-WER: 96.47 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 255.29 epoch: 122 | nupdates: 4253879 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:38 | bch(ms): 385.10 | smp(ms): 0.81 | fwd(ms): 111.63 | crit-fwd(ms): 7.40 | bwd(ms): 189.83 | optim(ms): 67.13 | loss: 258.65746 | train-TER: 80.03 | train-WER: 127.80 | dev-clean-loss: 156.85398 | dev-clean-TER: 91.13 | dev-clean-WER: 94.59 | dev-other-loss: 137.05342 | dev-other-TER: 92.99 | dev-other-WER: 96.43 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 255.39 epoch: 123 | nupdates: 4289035 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:20 | bch(ms): 384.59 | smp(ms): 0.81 | fwd(ms): 111.51 | crit-fwd(ms): 7.31 | bwd(ms): 189.68 | optim(ms): 66.98 | loss: 258.59703 | train-TER: 79.68 | train-WER: 127.37 | dev-clean-loss: 156.36957 | dev-clean-TER: 90.62 | dev-clean-WER: 94.49 | dev-other-loss: 136.70222 | dev-other-TER: 92.58 | dev-other-WER: 96.41 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 255.73 epoch: 124 | nupdates: 4324191 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:51 | bch(ms): 385.46 | smp(ms): 0.83 | fwd(ms): 111.71 | crit-fwd(ms): 7.41 | bwd(ms): 190.07 | optim(ms): 67.40 | loss: 258.52723 | train-TER: 78.74 | train-WER: 125.55 | dev-clean-loss: 156.69372 | dev-clean-TER: 90.81 | dev-clean-WER: 94.47 | dev-other-loss: 136.93204 | dev-other-TER: 92.59 | dev-other-WER: 96.29 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 255.16 epoch: 125 | nupdates: 4359347 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:23 | bch(ms): 386.38 | smp(ms): 0.83 | fwd(ms): 111.73 | crit-fwd(ms): 7.43 | bwd(ms): 190.46 | optim(ms): 67.81 | loss: 258.52271 | train-TER: 79.04 | train-WER: 126.59 | dev-clean-loss: 156.15160 | dev-clean-TER: 90.68 | dev-clean-WER: 94.45 | dev-other-loss: 136.96404 | dev-other-TER: 92.61 | dev-other-WER: 96.33 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 254.55 epoch: 126 | nupdates: 4394503 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:58 | bch(ms): 387.38 | smp(ms): 1.20 | fwd(ms): 111.76 | crit-fwd(ms): 7.45 | bwd(ms): 190.68 | optim(ms): 67.99 | loss: 258.49495 | train-TER: 79.49 | train-WER: 127.62 | dev-clean-loss: 156.10806 | dev-clean-TER: 90.48 | dev-clean-WER: 94.35 | dev-other-loss: 136.68027 | dev-other-TER: 92.30 | dev-other-WER: 96.20 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.89 epoch: 127 | nupdates: 4429659 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:58 | bch(ms): 387.38 | smp(ms): 0.84 | fwd(ms): 111.67 | crit-fwd(ms): 7.40 | bwd(ms): 190.61 | optim(ms): 68.46 | loss: 258.45029 | train-TER: 79.95 | train-WER: 128.36 | dev-clean-loss: 156.08215 | dev-clean-TER: 90.22 | dev-clean-WER: 94.29 | dev-other-loss: 136.59479 | dev-other-TER: 92.07 | dev-other-WER: 96.24 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.89 epoch: 128 | nupdates: 4464815 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:56 | bch(ms): 387.31 | smp(ms): 0.84 | fwd(ms): 111.58 | crit-fwd(ms): 7.36 | bwd(ms): 190.75 | optim(ms): 68.29 | loss: 258.42800 | train-TER: 80.26 | train-WER: 128.91 | dev-clean-loss: 156.05849 | dev-clean-TER: 90.03 | dev-clean-WER: 94.28 | dev-other-loss: 136.35127 | dev-other-TER: 91.92 | dev-other-WER: 96.12 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.94 epoch: 129 | nupdates: 4499971 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:19 | bch(ms): 387.98 | smp(ms): 0.84 | fwd(ms): 111.83 | crit-fwd(ms): 7.44 | bwd(ms): 190.97 | optim(ms): 68.49 | loss: 258.36758 | train-TER: 79.71 | train-WER: 128.31 | dev-clean-loss: 156.13134 | dev-clean-TER: 90.23 | dev-clean-WER: 94.22 | dev-other-loss: 136.45945 | dev-other-TER: 92.05 | dev-other-WER: 96.17 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.50 epoch: 130 | nupdates: 4535127 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:16 | bch(ms): 387.90 | smp(ms): 0.85 | fwd(ms): 111.63 | crit-fwd(ms): 7.33 | bwd(ms): 190.92 | optim(ms): 68.61 | loss: 258.33606 | train-TER: 80.20 | train-WER: 128.83 | dev-clean-loss: 156.21171 | dev-clean-TER: 90.26 | dev-clean-WER: 94.28 | dev-other-loss: 136.72820 | dev-other-TER: 92.12 | dev-other-WER: 96.18 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.55 epoch: 131 | nupdates: 4570283 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:53 | bch(ms): 387.23 | smp(ms): 0.84 | fwd(ms): 111.43 | crit-fwd(ms): 7.25 | bwd(ms): 190.74 | optim(ms): 68.39 | loss: 258.30001 | train-TER: 79.73 | train-WER: 128.51 | dev-clean-loss: 155.69105 | dev-clean-TER: 89.62 | dev-clean-WER: 94.15 | dev-other-loss: 136.11993 | dev-other-TER: 91.54 | dev-other-WER: 96.10 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 253.99 epoch: 132 | nupdates: 4605439 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:19 | bch(ms): 387.98 | smp(ms): 1.18 | fwd(ms): 111.96 | crit-fwd(ms): 7.58 | bwd(ms): 190.75 | optim(ms): 67.97 | loss: 258.28554 | train-TER: 80.78 | train-WER: 130.36 | dev-clean-loss: 155.79097 | dev-clean-TER: 89.42 | dev-clean-WER: 94.09 | dev-other-loss: 136.26015 | dev-other-TER: 91.29 | dev-other-WER: 95.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.50 epoch: 133 | nupdates: 4640595 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:21 | bch(ms): 388.03 | smp(ms): 0.84 | fwd(ms): 111.95 | crit-fwd(ms): 7.54 | bwd(ms): 190.94 | optim(ms): 68.16 | loss: 258.20357 | train-TER: 80.64 | train-WER: 130.37 | dev-clean-loss: 155.58804 | dev-clean-TER: 89.45 | dev-clean-WER: 94.07 | dev-other-loss: 136.00232 | dev-other-TER: 91.27 | dev-other-WER: 96.01 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.46 epoch: 134 | nupdates: 4675751 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:48 | bch(ms): 388.80 | smp(ms): 0.86 | fwd(ms): 111.84 | crit-fwd(ms): 7.45 | bwd(ms): 191.12 | optim(ms): 68.68 | loss: 258.15152 | train-TER: 81.45 | train-WER: 131.73 | dev-clean-loss: 155.33767 | dev-clean-TER: 89.09 | dev-clean-WER: 94.00 | dev-other-loss: 135.72141 | dev-other-TER: 90.95 | dev-other-WER: 95.95 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 252.96 epoch: 135 | nupdates: 4710907 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:39 | bch(ms): 388.54 | smp(ms): 0.84 | fwd(ms): 112.00 | crit-fwd(ms): 7.51 | bwd(ms): 191.08 | optim(ms): 68.21 | loss: 258.18005 | train-TER: 81.61 | train-WER: 132.45 | dev-clean-loss: 155.46899 | dev-clean-TER: 89.17 | dev-clean-WER: 94.01 | dev-other-loss: 135.80036 | dev-other-TER: 90.98 | dev-other-WER: 95.94 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 253.13 epoch: 136 | nupdates: 4746063 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:23 | bch(ms): 388.09 | smp(ms): 0.84 | fwd(ms): 111.71 | crit-fwd(ms): 7.38 | bwd(ms): 190.89 | optim(ms): 68.30 | loss: 258.12683 | train-TER: 81.74 | train-WER: 132.70 | dev-clean-loss: 155.32771 | dev-clean-TER: 88.76 | dev-clean-WER: 93.98 | dev-other-loss: 135.82437 | dev-other-TER: 90.65 | dev-other-WER: 95.95 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.43 epoch: 137 | nupdates: 4781219 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:23 | bch(ms): 388.09 | smp(ms): 1.03 | fwd(ms): 111.63 | crit-fwd(ms): 7.33 | bwd(ms): 190.84 | optim(ms): 68.31 | loss: 258.06615 | train-TER: 81.43 | train-WER: 131.69 | dev-clean-loss: 155.19278 | dev-clean-TER: 88.74 | dev-clean-WER: 93.96 | dev-other-loss: 135.81947 | dev-other-TER: 90.58 | dev-other-WER: 95.88 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 253.43 epoch: 138 | nupdates: 4816375 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:32 | bch(ms): 388.35 | smp(ms): 0.89 | fwd(ms): 111.94 | crit-fwd(ms): 7.54 | bwd(ms): 190.94 | optim(ms): 68.31 | loss: 258.11559 | train-TER: 81.12 | train-WER: 131.50 | dev-clean-loss: 154.93656 | dev-clean-TER: 88.38 | dev-clean-WER: 93.95 | dev-other-loss: 135.55102 | dev-other-TER: 90.32 | dev-other-WER: 95.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 253.26 epoch: 139 | nupdates: 4851531 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:14 | bch(ms): 387.84 | smp(ms): 0.84 | fwd(ms): 111.65 | crit-fwd(ms): 7.38 | bwd(ms): 190.72 | optim(ms): 68.21 | loss: 258.00978 | train-TER: 81.49 | train-WER: 132.92 | dev-clean-loss: 154.47822 | dev-clean-TER: 87.93 | dev-clean-WER: 93.86 | dev-other-loss: 135.32953 | dev-other-TER: 89.82 | dev-other-WER: 95.90 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 253.59 epoch: 140 | nupdates: 4886687 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:18 | bch(ms): 387.94 | smp(ms): 0.84 | fwd(ms): 111.54 | crit-fwd(ms): 7.29 | bwd(ms): 190.74 | optim(ms): 68.43 | loss: 257.98566 | train-TER: 81.66 | train-WER: 132.74 | dev-clean-loss: 154.90387 | dev-clean-TER: 88.27 | dev-clean-WER: 93.90 | dev-other-loss: 135.54692 | dev-other-TER: 90.15 | dev-other-WER: 95.87 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.53 epoch: 141 | nupdates: 4921843 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:32 | bch(ms): 388.35 | smp(ms): 0.84 | fwd(ms): 111.60 | crit-fwd(ms): 7.30 | bwd(ms): 190.93 | optim(ms): 68.48 | loss: 257.93742 | train-TER: 82.22 | train-WER: 134.02 | dev-clean-loss: 154.53823 | dev-clean-TER: 87.87 | dev-clean-WER: 93.85 | dev-other-loss: 135.15855 | dev-other-TER: 89.70 | dev-other-WER: 95.85 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 253.26 epoch: 142 | nupdates: 4956999 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:37 | bch(ms): 388.49 | smp(ms): 0.84 | fwd(ms): 111.78 | crit-fwd(ms): 7.38 | bwd(ms): 190.91 | optim(ms): 68.42 | loss: 257.92868 | train-TER: 82.19 | train-WER: 134.04 | dev-clean-loss: 154.31551 | dev-clean-TER: 87.68 | dev-clean-WER: 93.76 | dev-other-loss: 135.12497 | dev-other-TER: 89.56 | dev-other-WER: 95.80 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.17 epoch: 143 | nupdates: 4992155 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:20 | bch(ms): 387.99 | smp(ms): 0.83 | fwd(ms): 111.75 | crit-fwd(ms): 7.41 | bwd(ms): 190.70 | optim(ms): 68.05 | loss: 257.91130 | train-TER: 82.68 | train-WER: 135.26 | dev-clean-loss: 154.53828 | dev-clean-TER: 87.58 | dev-clean-WER: 93.84 | dev-other-loss: 135.29013 | dev-other-TER: 89.58 | dev-other-WER: 96.11 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.49 epoch: 144 | nupdates: 5027311 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:29 | bch(ms): 386.56 | smp(ms): 0.83 | fwd(ms): 111.55 | crit-fwd(ms): 7.34 | bwd(ms): 190.20 | optim(ms): 67.41 | loss: 257.82463 | train-TER: 82.31 | train-WER: 134.48 | dev-clean-loss: 154.29588 | dev-clean-TER: 87.13 | dev-clean-WER: 93.76 | dev-other-loss: 135.01481 | dev-other-TER: 89.23 | dev-other-WER: 96.09 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 254.43 epoch: 145 | nupdates: 5062467 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:07 | bch(ms): 387.62 | smp(ms): 1.31 | fwd(ms): 111.85 | crit-fwd(ms): 7.47 | bwd(ms): 190.31 | optim(ms): 67.50 | loss: 257.77365 | train-TER: 82.21 | train-WER: 134.63 | dev-clean-loss: 154.08671 | dev-clean-TER: 86.96 | dev-clean-WER: 93.71 | dev-other-loss: 134.89036 | dev-other-TER: 89.05 | dev-other-WER: 95.94 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.74 epoch: 146 | nupdates: 5097623 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:11 | bch(ms): 387.75 | smp(ms): 0.83 | fwd(ms): 111.59 | crit-fwd(ms): 7.30 | bwd(ms): 190.56 | optim(ms): 68.12 | loss: 257.79496 | train-TER: 82.14 | train-WER: 134.55 | dev-clean-loss: 154.14262 | dev-clean-TER: 86.94 | dev-clean-WER: 93.66 | dev-other-loss: 134.88979 | dev-other-TER: 89.02 | dev-other-WER: 96.02 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.65 epoch: 147 | nupdates: 5132779 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:59 | bch(ms): 387.40 | smp(ms): 0.83 | fwd(ms): 111.52 | crit-fwd(ms): 7.27 | bwd(ms): 190.47 | optim(ms): 67.95 | loss: 257.76017 | train-TER: 82.31 | train-WER: 134.64 | dev-clean-loss: 154.10222 | dev-clean-TER: 86.47 | dev-clean-WER: 93.63 | dev-other-loss: 134.67872 | dev-other-TER: 88.67 | dev-other-WER: 96.11 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.88 epoch: 148 | nupdates: 5167935 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:57 | bch(ms): 387.36 | smp(ms): 0.82 | fwd(ms): 111.53 | crit-fwd(ms): 7.29 | bwd(ms): 190.37 | optim(ms): 67.86 | loss: 257.74441 | train-TER: 82.63 | train-WER: 135.72 | dev-clean-loss: 153.90700 | dev-clean-TER: 86.05 | dev-clean-WER: 93.63 | dev-other-loss: 134.46129 | dev-other-TER: 88.34 | dev-other-WER: 96.10 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 253.91 epoch: 149 | nupdates: 5203091 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:57 | bch(ms): 387.33 | smp(ms): 1.16 | fwd(ms): 111.34 | crit-fwd(ms): 7.20 | bwd(ms): 190.26 | optim(ms): 67.77 | loss: 257.70611 | train-TER: 82.99 | train-WER: 136.52 | dev-clean-loss: 153.81132 | dev-clean-TER: 86.24 | dev-clean-WER: 93.66 | dev-other-loss: 134.47911 | dev-other-TER: 88.29 | dev-other-WER: 95.96 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 253.92 epoch: 150 | nupdates: 5238247 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:52 | bch(ms): 387.20 | smp(ms): 0.82 | fwd(ms): 111.27 | crit-fwd(ms): 7.17 | bwd(ms): 190.29 | optim(ms): 67.90 | loss: 257.68441 | train-TER: 83.54 | train-WER: 137.33 | dev-clean-loss: 153.86353 | dev-clean-TER: 86.28 | dev-clean-WER: 93.56 | dev-other-loss: 134.70497 | dev-other-TER: 88.41 | dev-other-WER: 95.98 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 254.01 epoch: 151 | nupdates: 5273403 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:59 | bch(ms): 387.40 | smp(ms): 0.83 | fwd(ms): 111.58 | crit-fwd(ms): 7.33 | bwd(ms): 190.13 | optim(ms): 67.82 | loss: 257.59082 | train-TER: 83.65 | train-WER: 138.04 | dev-clean-loss: 153.91617 | dev-clean-TER: 86.07 | dev-clean-WER: 93.62 | dev-other-loss: 134.79177 | dev-other-TER: 88.16 | dev-other-WER: 95.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.88 epoch: 152 | nupdates: 5308559 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:08 | bch(ms): 387.65 | smp(ms): 0.83 | fwd(ms): 111.24 | crit-fwd(ms): 7.17 | bwd(ms): 190.37 | optim(ms): 68.10 | loss: 257.53004 | train-TER: 83.89 | train-WER: 138.84 | dev-clean-loss: 153.68358 | dev-clean-TER: 85.55 | dev-clean-WER: 93.57 | dev-other-loss: 134.23774 | dev-other-TER: 87.78 | dev-other-WER: 95.89 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 107 | hrs: 960.47 | thrpt(sec/sec): 253.71 epoch: 153 | nupdates: 5343715 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:26 | bch(ms): 389.86 | smp(ms): 1.20 | fwd(ms): 111.80 | crit-fwd(ms): 7.24 | bwd(ms): 191.23 | optim(ms): 68.38 | loss: 257.54859 | train-TER: 84.31 | train-WER: 139.16 | dev-clean-loss: 153.62729 | dev-clean-TER: 85.27 | dev-clean-WER: 93.57 | dev-other-loss: 134.28350 | dev-other-TER: 87.42 | dev-other-WER: 95.79 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.28 epoch: 154 | nupdates: 5378871 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:39 | bch(ms): 390.26 | smp(ms): 0.83 | fwd(ms): 112.40 | crit-fwd(ms): 7.41 | bwd(ms): 191.46 | optim(ms): 68.32 | loss: 257.54569 | train-TER: 84.40 | train-WER: 139.85 | dev-clean-loss: 153.08680 | dev-clean-TER: 84.67 | dev-clean-WER: 93.49 | dev-other-loss: 134.00631 | dev-other-TER: 86.88 | dev-other-WER: 95.70 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 252.02 epoch: 155 | nupdates: 5414027 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:48 | bch(ms): 388.81 | smp(ms): 0.83 | fwd(ms): 111.94 | crit-fwd(ms): 7.45 | bwd(ms): 190.66 | optim(ms): 68.12 | loss: 257.48899 | train-TER: 84.30 | train-WER: 139.57 | dev-clean-loss: 153.09680 | dev-clean-TER: 84.34 | dev-clean-WER: 93.51 | dev-other-loss: 133.80229 | dev-other-TER: 86.49 | dev-other-WER: 95.62 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 252.96 epoch: 156 | nupdates: 5449183 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:00 | bch(ms): 387.42 | smp(ms): 0.82 | fwd(ms): 111.35 | crit-fwd(ms): 7.20 | bwd(ms): 190.31 | optim(ms): 67.71 | loss: 257.45502 | train-TER: 84.88 | train-WER: 140.53 | dev-clean-loss: 153.14582 | dev-clean-TER: 84.31 | dev-clean-WER: 93.47 | dev-other-loss: 134.03258 | dev-other-TER: 86.45 | dev-other-WER: 95.56 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 253.86 epoch: 157 | nupdates: 5484339 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:56:45 | bch(ms): 404.06 | smp(ms): 16.93 | fwd(ms): 113.09 | crit-fwd(ms): 8.01 | bwd(ms): 190.05 | optim(ms): 66.41 | loss: 257.39802 | train-TER: 85.21 | train-WER: 141.87 | dev-clean-loss: 153.08824 | dev-clean-TER: 84.22 | dev-clean-WER: 93.52 | dev-other-loss: 133.78610 | dev-other-TER: 86.39 | dev-other-WER: 95.62 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 243.41 epoch: 158 | nupdates: 5519495 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:00 | bch(ms): 385.71 | smp(ms): 0.81 | fwd(ms): 112.34 | crit-fwd(ms): 7.83 | bwd(ms): 189.49 | optim(ms): 65.47 | loss: 257.35797 | train-TER: 86.13 | train-WER: 143.31 | dev-clean-loss: 153.15055 | dev-clean-TER: 83.76 | dev-clean-WER: 93.56 | dev-other-loss: 133.74750 | dev-other-TER: 85.98 | dev-other-WER: 95.57 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 254.99 epoch: 159 | nupdates: 5554651 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:45:35 | bch(ms): 385.01 | smp(ms): 0.81 | fwd(ms): 111.33 | crit-fwd(ms): 7.22 | bwd(ms): 189.09 | optim(ms): 66.36 | loss: 257.33948 | train-TER: 85.39 | train-WER: 142.42 | dev-clean-loss: 152.97234 | dev-clean-TER: 83.60 | dev-clean-WER: 93.53 | dev-other-loss: 133.79695 | dev-other-TER: 85.78 | dev-other-WER: 95.60 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 255.46 epoch: 160 | nupdates: 5589807 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:46:54 | bch(ms): 387.25 | smp(ms): 0.83 | fwd(ms): 111.41 | crit-fwd(ms): 7.33 | bwd(ms): 189.81 | optim(ms): 67.29 | loss: 257.31320 | train-TER: 87.08 | train-WER: 145.36 | dev-clean-loss: 152.86047 | dev-clean-TER: 83.31 | dev-clean-WER: 93.59 | dev-other-loss: 133.82100 | dev-other-TER: 85.55 | dev-other-WER: 95.59 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 112 | hrs: 960.47 | thrpt(sec/sec): 253.98 epoch: 161 | nupdates: 5624963 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:41 | bch(ms): 388.61 | smp(ms): 0.84 | fwd(ms): 111.62 | crit-fwd(ms): 7.33 | bwd(ms): 190.48 | optim(ms): 67.96 | loss: 257.28598 | train-TER: 86.09 | train-WER: 143.63 | dev-clean-loss: 153.11426 | dev-clean-TER: 83.35 | dev-clean-WER: 93.63 | dev-other-loss: 133.73761 | dev-other-TER: 85.60 | dev-other-WER: 95.71 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 253.09 epoch: 162 | nupdates: 5660119 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:58 | bch(ms): 389.08 | smp(ms): 0.83 | fwd(ms): 111.43 | crit-fwd(ms): 7.22 | bwd(ms): 190.66 | optim(ms): 68.24 | loss: 257.21608 | train-TER: 87.10 | train-WER: 145.72 | dev-clean-loss: 152.84156 | dev-clean-TER: 83.08 | dev-clean-WER: 93.61 | dev-other-loss: 133.58709 | dev-other-TER: 85.26 | dev-other-WER: 95.56 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 252.78 epoch: 163 | nupdates: 5695275 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:56 | bch(ms): 389.02 | smp(ms): 1.30 | fwd(ms): 111.76 | crit-fwd(ms): 7.45 | bwd(ms): 190.41 | optim(ms): 67.62 | loss: 257.23799 | train-TER: 86.89 | train-WER: 145.51 | dev-clean-loss: 152.82700 | dev-clean-TER: 82.54 | dev-clean-WER: 93.69 | dev-other-loss: 133.43517 | dev-other-TER: 84.74 | dev-other-WER: 95.58 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.82 epoch: 164 | nupdates: 5730431 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:54 | bch(ms): 390.67 | smp(ms): 1.20 | fwd(ms): 111.57 | crit-fwd(ms): 7.30 | bwd(ms): 191.03 | optim(ms): 68.70 | loss: 257.22058 | train-TER: 87.99 | train-WER: 147.77 | dev-clean-loss: 152.72781 | dev-clean-TER: 82.19 | dev-clean-WER: 93.87 | dev-other-loss: 133.49828 | dev-other-TER: 84.46 | dev-other-WER: 95.70 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 251.75 epoch: 165 | nupdates: 5765587 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:00 | bch(ms): 390.85 | smp(ms): 0.84 | fwd(ms): 111.49 | crit-fwd(ms): 7.26 | bwd(ms): 191.21 | optim(ms): 68.96 | loss: 257.17852 | train-TER: 89.09 | train-WER: 148.94 | dev-clean-loss: 152.50482 | dev-clean-TER: 82.07 | dev-clean-WER: 93.88 | dev-other-loss: 133.26595 | dev-other-TER: 84.33 | dev-other-WER: 95.68 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 251.64 epoch: 166 | nupdates: 5800743 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:43 | bch(ms): 390.35 | smp(ms): 0.85 | fwd(ms): 111.47 | crit-fwd(ms): 7.34 | bwd(ms): 190.92 | optim(ms): 68.74 | loss: 257.12018 | train-TER: 88.68 | train-WER: 148.78 | dev-clean-loss: 152.45420 | dev-clean-TER: 81.61 | dev-clean-WER: 93.98 | dev-other-loss: 133.23615 | dev-other-TER: 83.82 | dev-other-WER: 95.56 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 251.96 epoch: 167 | nupdates: 5835899 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:55 | bch(ms): 390.71 | smp(ms): 0.85 | fwd(ms): 111.29 | crit-fwd(ms): 7.36 | bwd(ms): 190.73 | optim(ms): 69.23 | loss: 257.08309 | train-TER: 89.78 | train-WER: 150.81 | dev-clean-loss: 152.30927 | dev-clean-TER: 81.50 | dev-clean-WER: 93.90 | dev-other-loss: 133.35587 | dev-other-TER: 83.74 | dev-other-WER: 95.59 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 251.73 epoch: 168 | nupdates: 5871055 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:08 | bch(ms): 391.07 | smp(ms): 0.83 | fwd(ms): 111.65 | crit-fwd(ms): 7.30 | bwd(ms): 191.38 | optim(ms): 68.66 | loss: 257.09030 | train-TER: 89.51 | train-WER: 150.36 | dev-clean-loss: 152.37439 | dev-clean-TER: 81.20 | dev-clean-WER: 93.90 | dev-other-loss: 133.04801 | dev-other-TER: 83.50 | dev-other-WER: 95.51 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 251.50 epoch: 169 | nupdates: 5906211 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:20 | bch(ms): 391.42 | smp(ms): 0.84 | fwd(ms): 112.08 | crit-fwd(ms): 7.57 | bwd(ms): 191.47 | optim(ms): 68.12 | loss: 257.01927 | train-TER: 90.92 | train-WER: 153.28 | dev-clean-loss: 152.48178 | dev-clean-TER: 80.86 | dev-clean-WER: 93.90 | dev-other-loss: 133.15829 | dev-other-TER: 83.08 | dev-other-WER: 95.46 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 251.27 epoch: 170 | nupdates: 5941367 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:13 | bch(ms): 391.21 | smp(ms): 0.84 | fwd(ms): 112.10 | crit-fwd(ms): 7.60 | bwd(ms): 191.44 | optim(ms): 67.77 | loss: 257.00670 | train-TER: 91.56 | train-WER: 154.54 | dev-clean-loss: 152.15277 | dev-clean-TER: 80.31 | dev-clean-WER: 94.23 | dev-other-loss: 132.94506 | dev-other-TER: 82.65 | dev-other-WER: 95.71 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 251.41 epoch: 171 | nupdates: 5976523 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:48 | bch(ms): 390.50 | smp(ms): 0.83 | fwd(ms): 111.54 | crit-fwd(ms): 7.28 | bwd(ms): 191.17 | optim(ms): 68.19 | loss: 256.97052 | train-TER: 91.21 | train-WER: 153.75 | dev-clean-loss: 152.12403 | dev-clean-TER: 79.83 | dev-clean-WER: 94.51 | dev-other-loss: 132.88262 | dev-other-TER: 82.08 | dev-other-WER: 95.79 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 251.86 epoch: 172 | nupdates: 6011679 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:34 | bch(ms): 390.09 | smp(ms): 0.84 | fwd(ms): 111.46 | crit-fwd(ms): 7.24 | bwd(ms): 190.88 | optim(ms): 68.11 | loss: 256.90750 | train-TER: 91.58 | train-WER: 154.05 | dev-clean-loss: 152.16270 | dev-clean-TER: 80.04 | dev-clean-WER: 94.36 | dev-other-loss: 133.01482 | dev-other-TER: 82.24 | dev-other-WER: 95.73 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.13 epoch: 173 | nupdates: 6046835 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:58 | bch(ms): 389.07 | smp(ms): 0.83 | fwd(ms): 111.60 | crit-fwd(ms): 7.36 | bwd(ms): 190.52 | optim(ms): 67.04 | loss: 256.87018 | train-TER: 93.30 | train-WER: 157.30 | dev-clean-loss: 152.02816 | dev-clean-TER: 79.75 | dev-clean-WER: 94.29 | dev-other-loss: 132.82660 | dev-other-TER: 81.91 | dev-other-WER: 95.58 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 252.79 epoch: 174 | nupdates: 6081991 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:50 | bch(ms): 388.84 | smp(ms): 0.83 | fwd(ms): 111.52 | crit-fwd(ms): 7.31 | bwd(ms): 190.33 | optim(ms): 67.17 | loss: 256.89640 | train-TER: 92.87 | train-WER: 156.69 | dev-clean-loss: 151.68679 | dev-clean-TER: 79.08 | dev-clean-WER: 94.78 | dev-other-loss: 132.55885 | dev-other-TER: 81.29 | dev-other-WER: 95.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 252.94 epoch: 175 | nupdates: 6117147 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:54 | bch(ms): 390.66 | smp(ms): 2.61 | fwd(ms): 111.59 | crit-fwd(ms): 7.39 | bwd(ms): 190.33 | optim(ms): 66.97 | loss: 256.81383 | train-TER: 94.17 | train-WER: 158.73 | dev-clean-loss: 151.87699 | dev-clean-TER: 79.17 | dev-clean-WER: 94.79 | dev-other-loss: 132.67529 | dev-other-TER: 81.33 | dev-other-WER: 95.94 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 251.76 epoch: 176 | nupdates: 6152303 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:42 | bch(ms): 392.03 | smp(ms): 3.64 | fwd(ms): 111.54 | crit-fwd(ms): 7.36 | bwd(ms): 190.21 | optim(ms): 67.30 | loss: 256.81824 | train-TER: 94.31 | train-WER: 159.55 | dev-clean-loss: 151.71694 | dev-clean-TER: 79.11 | dev-clean-WER: 94.67 | dev-other-loss: 132.70313 | dev-other-TER: 81.23 | dev-other-WER: 95.89 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.88 epoch: 177 | nupdates: 6187459 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:39 | bch(ms): 393.66 | smp(ms): 4.95 | fwd(ms): 111.50 | crit-fwd(ms): 7.35 | bwd(ms): 190.38 | optim(ms): 67.47 | loss: 256.79767 | train-TER: 95.22 | train-WER: 160.59 | dev-clean-loss: 151.57764 | dev-clean-TER: 78.60 | dev-clean-WER: 95.62 | dev-other-loss: 132.44590 | dev-other-TER: 80.80 | dev-other-WER: 96.66 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 249.84 epoch: 178 | nupdates: 6222615 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:15 | bch(ms): 389.58 | smp(ms): 0.82 | fwd(ms): 111.52 | crit-fwd(ms): 7.30 | bwd(ms): 190.39 | optim(ms): 67.44 | loss: 256.74600 | train-TER: 95.90 | train-WER: 161.68 | dev-clean-loss: 151.31898 | dev-clean-TER: 78.43 | dev-clean-WER: 95.51 | dev-other-loss: 132.48649 | dev-other-TER: 80.32 | dev-other-WER: 95.98 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 252.46 epoch: 179 | nupdates: 6257771 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:30 | bch(ms): 389.98 | smp(ms): 0.82 | fwd(ms): 111.60 | crit-fwd(ms): 7.37 | bwd(ms): 190.78 | optim(ms): 67.16 | loss: 256.70996 | train-TER: 96.51 | train-WER: 162.67 | dev-clean-loss: 151.48194 | dev-clean-TER: 78.12 | dev-clean-WER: 96.26 | dev-other-loss: 132.31816 | dev-other-TER: 80.21 | dev-other-WER: 96.92 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 252.20 epoch: 180 | nupdates: 6292927 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:28 | bch(ms): 389.94 | smp(ms): 0.82 | fwd(ms): 111.59 | crit-fwd(ms): 7.38 | bwd(ms): 190.70 | optim(ms): 67.10 | loss: 256.67377 | train-TER: 97.37 | train-WER: 163.72 | dev-clean-loss: 151.58034 | dev-clean-TER: 77.81 | dev-clean-WER: 96.15 | dev-other-loss: 132.11481 | dev-other-TER: 79.96 | dev-other-WER: 96.95 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.22 epoch: 181 | nupdates: 6328083 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:03 | bch(ms): 390.93 | smp(ms): 0.83 | fwd(ms): 111.79 | crit-fwd(ms): 7.46 | bwd(ms): 191.08 | optim(ms): 67.34 | loss: 256.69717 | train-TER: 97.62 | train-WER: 164.68 | dev-clean-loss: 151.32011 | dev-clean-TER: 77.82 | dev-clean-WER: 97.21 | dev-other-loss: 132.41029 | dev-other-TER: 79.85 | dev-other-WER: 97.71 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 251.59 epoch: 182 | nupdates: 6363239 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:22 | bch(ms): 391.47 | smp(ms): 0.84 | fwd(ms): 111.74 | crit-fwd(ms): 7.43 | bwd(ms): 191.09 | optim(ms): 67.83 | loss: 256.64246 | train-TER: 97.80 | train-WER: 165.07 | dev-clean-loss: 151.45288 | dev-clean-TER: 77.71 | dev-clean-WER: 96.85 | dev-other-loss: 132.11498 | dev-other-TER: 79.74 | dev-other-WER: 97.37 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 251.24 epoch: 183 | nupdates: 6398395 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:30 | bch(ms): 391.70 | smp(ms): 0.84 | fwd(ms): 111.52 | crit-fwd(ms): 7.29 | bwd(ms): 191.10 | optim(ms): 68.31 | loss: 256.57251 | train-TER: 98.19 | train-WER: 165.39 | dev-clean-loss: 151.32559 | dev-clean-TER: 77.56 | dev-clean-WER: 97.69 | dev-other-loss: 132.15515 | dev-other-TER: 79.50 | dev-other-WER: 97.80 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 107 | hrs: 960.47 | thrpt(sec/sec): 251.09 epoch: 184 | nupdates: 6433551 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:15 | bch(ms): 391.27 | smp(ms): 0.83 | fwd(ms): 111.49 | crit-fwd(ms): 7.26 | bwd(ms): 190.87 | optim(ms): 68.17 | loss: 256.58194 | train-TER: 98.48 | train-WER: 165.61 | dev-clean-loss: 151.27784 | dev-clean-TER: 77.87 | dev-clean-WER: 98.51 | dev-other-loss: 132.22312 | dev-other-TER: 79.66 | dev-other-WER: 98.41 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 251.37 epoch: 185 | nupdates: 6468707 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:36 | bch(ms): 391.86 | smp(ms): 0.84 | fwd(ms): 111.76 | crit-fwd(ms): 7.42 | bwd(ms): 191.14 | optim(ms): 67.79 | loss: 256.56800 | train-TER: 100.21 | train-WER: 168.75 | dev-clean-loss: 151.13067 | dev-clean-TER: 77.27 | dev-clean-WER: 98.46 | dev-other-loss: 132.07044 | dev-other-TER: 79.23 | dev-other-WER: 98.33 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.99 epoch: 186 | nupdates: 6503863 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:13 | bch(ms): 391.22 | smp(ms): 0.84 | fwd(ms): 111.72 | crit-fwd(ms): 7.44 | bwd(ms): 190.96 | optim(ms): 67.43 | loss: 256.52746 | train-TER: 100.70 | train-WER: 169.45 | dev-clean-loss: 151.00694 | dev-clean-TER: 77.21 | dev-clean-WER: 98.54 | dev-other-loss: 131.98985 | dev-other-TER: 78.89 | dev-other-WER: 97.88 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 251.40 epoch: 187 | nupdates: 6539019 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:10 | bch(ms): 389.43 | smp(ms): 0.83 | fwd(ms): 111.17 | crit-fwd(ms): 7.47 | bwd(ms): 189.89 | optim(ms): 67.39 | loss: 256.51928 | train-TER: 99.82 | train-WER: 167.80 | dev-clean-loss: 151.00005 | dev-clean-TER: 77.78 | dev-clean-WER: 100.62 | dev-other-loss: 132.17019 | dev-other-TER: 79.27 | dev-other-WER: 99.77 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.55 epoch: 188 | nupdates: 6574175 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:12 | bch(ms): 389.47 | smp(ms): 0.84 | fwd(ms): 111.05 | crit-fwd(ms): 7.43 | bwd(ms): 189.82 | optim(ms): 67.54 | loss: 256.47559 | train-TER: 100.82 | train-WER: 169.37 | dev-clean-loss: 151.00809 | dev-clean-TER: 77.58 | dev-clean-WER: 100.62 | dev-other-loss: 131.99761 | dev-other-TER: 79.06 | dev-other-WER: 99.56 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 252.53 epoch: 189 | nupdates: 6609331 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:50 | bch(ms): 390.56 | smp(ms): 0.83 | fwd(ms): 111.12 | crit-fwd(ms): 7.32 | bwd(ms): 190.21 | optim(ms): 68.05 | loss: 256.42094 | train-TER: 101.38 | train-WER: 170.24 | dev-clean-loss: 151.10533 | dev-clean-TER: 77.89 | dev-clean-WER: 100.94 | dev-other-loss: 132.06297 | dev-other-TER: 79.16 | dev-other-WER: 99.95 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 251.83 epoch: 190 | nupdates: 6644487 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:31 | bch(ms): 391.73 | smp(ms): 0.83 | fwd(ms): 111.39 | crit-fwd(ms): 7.20 | bwd(ms): 190.89 | optim(ms): 68.25 | loss: 256.39708 | train-TER: 102.00 | train-WER: 171.11 | dev-clean-loss: 151.01147 | dev-clean-TER: 77.78 | dev-clean-WER: 102.10 | dev-other-loss: 131.93125 | dev-other-TER: 78.74 | dev-other-WER: 100.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 251.07 epoch: 191 | nupdates: 6679643 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:33 | bch(ms): 391.77 | smp(ms): 0.84 | fwd(ms): 111.28 | crit-fwd(ms): 7.14 | bwd(ms): 191.00 | optim(ms): 68.48 | loss: 256.38072 | train-TER: 101.26 | train-WER: 169.69 | dev-clean-loss: 151.00810 | dev-clean-TER: 77.91 | dev-clean-WER: 103.47 | dev-other-loss: 131.86704 | dev-other-TER: 78.44 | dev-other-WER: 100.63 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 251.05 epoch: 192 | nupdates: 6714799 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:39 | bch(ms): 391.95 | smp(ms): 0.83 | fwd(ms): 111.22 | crit-fwd(ms): 7.14 | bwd(ms): 191.00 | optim(ms): 68.45 | loss: 256.33752 | train-TER: 102.69 | train-WER: 171.88 | dev-clean-loss: 150.87421 | dev-clean-TER: 77.98 | dev-clean-WER: 103.28 | dev-other-loss: 131.86192 | dev-other-TER: 78.59 | dev-other-WER: 101.01 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 250.93 epoch: 193 | nupdates: 6749955 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:32 | bch(ms): 393.47 | smp(ms): 0.85 | fwd(ms): 111.30 | crit-fwd(ms): 7.16 | bwd(ms): 191.40 | optim(ms): 69.36 | loss: 256.31688 | train-TER: 102.92 | train-WER: 172.10 | dev-clean-loss: 150.82075 | dev-clean-TER: 78.22 | dev-clean-WER: 103.42 | dev-other-loss: 131.78997 | dev-other-TER: 78.78 | dev-other-WER: 101.08 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 249.96 epoch: 194 | nupdates: 6785111 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:38 | bch(ms): 393.62 | smp(ms): 0.86 | fwd(ms): 111.43 | crit-fwd(ms): 7.24 | bwd(ms): 191.48 | optim(ms): 69.34 | loss: 256.26593 | train-TER: 102.91 | train-WER: 171.82 | dev-clean-loss: 150.95000 | dev-clean-TER: 78.00 | dev-clean-WER: 103.94 | dev-other-loss: 131.65181 | dev-other-TER: 78.48 | dev-other-WER: 101.11 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 249.87 epoch: 195 | nupdates: 6820267 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:48 | bch(ms): 393.91 | smp(ms): 0.91 | fwd(ms): 111.34 | crit-fwd(ms): 7.19 | bwd(ms): 191.41 | optim(ms): 69.28 | loss: 256.24342 | train-TER: 105.64 | train-WER: 176.11 | dev-clean-loss: 151.08256 | dev-clean-TER: 78.55 | dev-clean-WER: 105.50 | dev-other-loss: 131.90320 | dev-other-TER: 78.74 | dev-other-WER: 102.23 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 249.68 epoch: 196 | nupdates: 6855423 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:58 | bch(ms): 392.50 | smp(ms): 0.85 | fwd(ms): 111.24 | crit-fwd(ms): 7.18 | bwd(ms): 191.11 | optim(ms): 68.72 | loss: 256.21514 | train-TER: 104.34 | train-WER: 173.55 | dev-clean-loss: 150.69532 | dev-clean-TER: 78.97 | dev-clean-WER: 106.78 | dev-other-loss: 131.88351 | dev-other-TER: 78.77 | dev-other-WER: 102.85 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 250.58 epoch: 197 | nupdates: 6890579 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:23 | bch(ms): 393.21 | smp(ms): 0.84 | fwd(ms): 111.44 | crit-fwd(ms): 7.27 | bwd(ms): 191.34 | optim(ms): 68.94 | loss: 256.20526 | train-TER: 104.40 | train-WER: 174.00 | dev-clean-loss: 150.84328 | dev-clean-TER: 78.83 | dev-clean-WER: 106.41 | dev-other-loss: 131.70985 | dev-other-TER: 78.52 | dev-other-WER: 102.17 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 250.12 epoch: 198 | nupdates: 6925735 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:46 | bch(ms): 392.15 | smp(ms): 0.84 | fwd(ms): 111.49 | crit-fwd(ms): 7.30 | bwd(ms): 190.93 | optim(ms): 68.17 | loss: 256.15593 | train-TER: 105.53 | train-WER: 175.09 | dev-clean-loss: 150.82220 | dev-clean-TER: 79.21 | dev-clean-WER: 107.93 | dev-other-loss: 131.79739 | dev-other-TER: 78.64 | dev-other-WER: 103.23 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.80 epoch: 199 | nupdates: 6960891 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:03 | bch(ms): 392.63 | smp(ms): 0.84 | fwd(ms): 111.54 | crit-fwd(ms): 7.32 | bwd(ms): 191.08 | optim(ms): 68.43 | loss: 256.12402 | train-TER: 105.42 | train-WER: 175.16 | dev-clean-loss: 150.81605 | dev-clean-TER: 79.23 | dev-clean-WER: 107.50 | dev-other-loss: 132.01930 | dev-other-TER: 78.57 | dev-other-WER: 102.88 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 250.50 epoch: 200 | nupdates: 6996047 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:59 | bch(ms): 389.10 | smp(ms): 0.82 | fwd(ms): 111.25 | crit-fwd(ms): 7.22 | bwd(ms): 189.75 | optim(ms): 67.01 | loss: 256.17794 | train-TER: 104.53 | train-WER: 172.70 | dev-clean-loss: 150.66513 | dev-clean-TER: 79.81 | dev-clean-WER: 109.19 | dev-other-loss: 131.79386 | dev-other-TER: 78.96 | dev-other-WER: 104.25 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 106 | hrs: 960.47 | thrpt(sec/sec): 252.77 epoch: 201 | nupdates: 7031203 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:33 | bch(ms): 390.09 | smp(ms): 0.82 | fwd(ms): 111.18 | crit-fwd(ms): 7.17 | bwd(ms): 190.01 | optim(ms): 67.59 | loss: 256.09032 | train-TER: 105.38 | train-WER: 173.95 | dev-clean-loss: 150.58352 | dev-clean-TER: 80.31 | dev-clean-WER: 110.42 | dev-other-loss: 131.61675 | dev-other-TER: 79.00 | dev-other-WER: 104.56 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.13 epoch: 202 | nupdates: 7066359 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:52 | bch(ms): 392.31 | smp(ms): 0.84 | fwd(ms): 111.38 | crit-fwd(ms): 7.22 | bwd(ms): 190.85 | optim(ms): 68.52 | loss: 256.03822 | train-TER: 106.46 | train-WER: 175.50 | dev-clean-loss: 150.85309 | dev-clean-TER: 80.53 | dev-clean-WER: 111.42 | dev-other-loss: 131.66152 | dev-other-TER: 79.06 | dev-other-WER: 105.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 250.70 epoch: 203 | nupdates: 7101515 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:26 | bch(ms): 393.29 | smp(ms): 0.84 | fwd(ms): 111.52 | crit-fwd(ms): 7.25 | bwd(ms): 191.36 | optim(ms): 69.02 | loss: 256.02067 | train-TER: 105.62 | train-WER: 173.91 | dev-clean-loss: 150.50949 | dev-clean-TER: 80.46 | dev-clean-WER: 111.08 | dev-other-loss: 131.65813 | dev-other-TER: 79.23 | dev-other-WER: 105.36 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.08 epoch: 204 | nupdates: 7136671 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:29 | bch(ms): 393.38 | smp(ms): 0.89 | fwd(ms): 111.51 | crit-fwd(ms): 7.24 | bwd(ms): 191.33 | optim(ms): 69.10 | loss: 256.01354 | train-TER: 106.04 | train-WER: 174.12 | dev-clean-loss: 150.71239 | dev-clean-TER: 80.90 | dev-clean-WER: 112.89 | dev-other-loss: 131.59816 | dev-other-TER: 79.32 | dev-other-WER: 106.25 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.02 epoch: 205 | nupdates: 7171827 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:53:12 | bch(ms): 398.02 | smp(ms): 0.87 | fwd(ms): 111.99 | crit-fwd(ms): 7.39 | bwd(ms): 193.01 | optim(ms): 70.78 | loss: 255.96684 | train-TER: 107.71 | train-WER: 176.33 | dev-clean-loss: 150.60150 | dev-clean-TER: 80.92 | dev-clean-WER: 112.78 | dev-other-loss: 131.68930 | dev-other-TER: 79.17 | dev-other-WER: 105.90 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 105 | hrs: 960.47 | thrpt(sec/sec): 247.10 epoch: 206 | nupdates: 7206983 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:52:22 | bch(ms): 396.58 | smp(ms): 0.87 | fwd(ms): 111.98 | crit-fwd(ms): 7.39 | bwd(ms): 192.55 | optim(ms): 70.46 | loss: 255.96888 | train-TER: 105.52 | train-WER: 172.60 | dev-clean-loss: 150.67628 | dev-clean-TER: 81.37 | dev-clean-WER: 114.22 | dev-other-loss: 131.66807 | dev-other-TER: 79.54 | dev-other-WER: 106.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 248.00 epoch: 207 | nupdates: 7242139 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:18 | bch(ms): 393.07 | smp(ms): 0.88 | fwd(ms): 111.47 | crit-fwd(ms): 7.24 | bwd(ms): 191.12 | optim(ms): 69.10 | loss: 255.95677 | train-TER: 106.31 | train-WER: 174.10 | dev-clean-loss: 150.48214 | dev-clean-TER: 81.09 | dev-clean-WER: 112.65 | dev-other-loss: 131.86220 | dev-other-TER: 79.31 | dev-other-WER: 106.12 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 250.21 epoch: 208 | nupdates: 7277295 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:25 | bch(ms): 393.25 | smp(ms): 0.85 | fwd(ms): 111.70 | crit-fwd(ms): 7.38 | bwd(ms): 191.38 | optim(ms): 68.87 | loss: 255.85990 | train-TER: 105.59 | train-WER: 172.79 | dev-clean-loss: 150.49601 | dev-clean-TER: 81.48 | dev-clean-WER: 113.54 | dev-other-loss: 131.58462 | dev-other-TER: 79.85 | dev-other-WER: 107.34 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 250.10 epoch: 209 | nupdates: 7312451 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:40 | bch(ms): 393.70 | smp(ms): 0.85 | fwd(ms): 111.73 | crit-fwd(ms): 7.35 | bwd(ms): 191.59 | optim(ms): 69.18 | loss: 255.89797 | train-TER: 106.01 | train-WER: 172.68 | dev-clean-loss: 150.62065 | dev-clean-TER: 81.27 | dev-clean-WER: 113.44 | dev-other-loss: 131.50762 | dev-other-TER: 79.58 | dev-other-WER: 107.05 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 249.82 epoch: 210 | nupdates: 7347607 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:55 | bch(ms): 392.40 | smp(ms): 0.84 | fwd(ms): 111.65 | crit-fwd(ms): 7.35 | bwd(ms): 191.34 | optim(ms): 68.33 | loss: 255.82242 | train-TER: 106.32 | train-WER: 172.78 | dev-clean-loss: 150.64223 | dev-clean-TER: 81.76 | dev-clean-WER: 114.82 | dev-other-loss: 131.40095 | dev-other-TER: 79.81 | dev-other-WER: 107.81 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 250.64 epoch: 211 | nupdates: 7382763 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:43 | bch(ms): 390.36 | smp(ms): 0.83 | fwd(ms): 111.43 | crit-fwd(ms): 7.32 | bwd(ms): 190.63 | optim(ms): 67.52 | loss: 255.79918 | train-TER: 105.53 | train-WER: 171.46 | dev-clean-loss: 150.22513 | dev-clean-TER: 81.78 | dev-clean-WER: 114.75 | dev-other-loss: 131.50691 | dev-other-TER: 79.60 | dev-other-WER: 107.43 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 100 | hrs: 960.47 | thrpt(sec/sec): 251.95 epoch: 212 | nupdates: 7417919 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:40 | bch(ms): 388.56 | smp(ms): 0.81 | fwd(ms): 111.37 | crit-fwd(ms): 7.29 | bwd(ms): 189.94 | optim(ms): 66.79 | loss: 255.80021 | train-TER: 105.14 | train-WER: 169.54 | dev-clean-loss: 150.74661 | dev-clean-TER: 81.55 | dev-clean-WER: 113.91 | dev-other-loss: 131.45786 | dev-other-TER: 79.48 | dev-other-WER: 107.12 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 253.12 epoch: 213 | nupdates: 7453075 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:47:46 | bch(ms): 388.74 | smp(ms): 0.82 | fwd(ms): 111.24 | crit-fwd(ms): 7.21 | bwd(ms): 190.05 | optim(ms): 67.21 | loss: 255.75393 | train-TER: 105.00 | train-WER: 168.42 | dev-clean-loss: 150.52045 | dev-clean-TER: 81.87 | dev-clean-WER: 114.88 | dev-other-loss: 131.59659 | dev-other-TER: 80.05 | dev-other-WER: 108.29 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 253.00 epoch: 214 | nupdates: 7488231 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:20 | bch(ms): 389.70 | smp(ms): 0.83 | fwd(ms): 111.23 | crit-fwd(ms): 7.23 | bwd(ms): 190.33 | optim(ms): 67.58 | loss: 255.74193 | train-TER: 105.97 | train-WER: 170.24 | dev-clean-loss: 150.56458 | dev-clean-TER: 82.05 | dev-clean-WER: 114.92 | dev-other-loss: 131.43926 | dev-other-TER: 79.81 | dev-other-WER: 107.54 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 252.38 epoch: 215 | nupdates: 7523387 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:48:13 | bch(ms): 389.52 | smp(ms): 0.86 | fwd(ms): 111.40 | crit-fwd(ms): 7.27 | bwd(ms): 190.44 | optim(ms): 67.42 | loss: 255.67924 | train-TER: 104.24 | train-WER: 167.38 | dev-clean-loss: 150.34528 | dev-clean-TER: 82.10 | dev-clean-WER: 115.49 | dev-other-loss: 131.50913 | dev-other-TER: 79.87 | dev-other-WER: 108.03 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 252.50 epoch: 216 | nupdates: 7558543 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:12 | bch(ms): 391.19 | smp(ms): 0.84 | fwd(ms): 111.56 | crit-fwd(ms): 7.32 | bwd(ms): 191.11 | optim(ms): 68.13 | loss: 255.68077 | train-TER: 104.94 | train-WER: 167.63 | dev-clean-loss: 150.02326 | dev-clean-TER: 81.98 | dev-clean-WER: 115.16 | dev-other-loss: 131.48409 | dev-other-TER: 79.89 | dev-other-WER: 108.03 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 251.42 epoch: 217 | nupdates: 7593699 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:49:57 | bch(ms): 392.45 | smp(ms): 0.97 | fwd(ms): 111.75 | crit-fwd(ms): 7.44 | bwd(ms): 191.80 | optim(ms): 68.69 | loss: 255.66223 | train-TER: 103.21 | train-WER: 164.61 | dev-clean-loss: 150.20454 | dev-clean-TER: 82.49 | dev-clean-WER: 116.26 | dev-other-loss: 131.41589 | dev-other-TER: 80.19 | dev-other-WER: 108.61 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 104 | hrs: 960.47 | thrpt(sec/sec): 250.61 epoch: 218 | nupdates: 7628855 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:45 | bch(ms): 393.83 | smp(ms): 0.87 | fwd(ms): 112.06 | crit-fwd(ms): 7.53 | bwd(ms): 192.35 | optim(ms): 69.07 | loss: 255.59491 | train-TER: 104.04 | train-WER: 165.62 | dev-clean-loss: 150.15699 | dev-clean-TER: 82.44 | dev-clean-WER: 116.10 | dev-other-loss: 131.49055 | dev-other-TER: 80.47 | dev-other-WER: 109.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 102 | hrs: 960.47 | thrpt(sec/sec): 249.73 epoch: 219 | nupdates: 7664011 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:50:59 | bch(ms): 394.23 | smp(ms): 0.86 | fwd(ms): 111.88 | crit-fwd(ms): 7.44 | bwd(ms): 192.48 | optim(ms): 69.63 | loss: 255.60636 | train-TER: 103.20 | train-WER: 163.97 | dev-clean-loss: 150.35325 | dev-clean-TER: 82.36 | dev-clean-WER: 115.73 | dev-other-loss: 131.46223 | dev-other-TER: 80.45 | dev-other-WER: 109.16 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 101 | hrs: 960.47 | thrpt(sec/sec): 249.48 epoch: 220 | nupdates: 7699167 | lr: 0.000015 | lrcriterion: 0.000015 | runtime: 03:51:03 | bch(ms): 394.33 | smp(ms): 0.87 | fwd(ms): 112.31 | crit-fwd(ms): 7.61 | bwd(ms): 192.68 | optim(ms): 69.19 | loss: 255.56517 | train-TER: 102.77 | train-WER: 162.56 | dev-clean-loss: 150.29470 | dev-clean-TER: 82.99 | dev-clean-WER: 117.02 | dev-other-loss: 131.46274 | dev-other-TER: 80.77 | dev-other-WER: 109.97 | avg-isz: 1229 | avg-tsz: 041 | max-tsz: 103 | hrs: 960.47 | thrpt(sec/sec): 249.42

AlexandderGorodetski commented 4 years ago

date time epoch nupdates lr lrcriterion runtime bch(ms) smp(ms) fwd(ms) crit-fwd(ms) bwd(ms) optim(ms) loss train-TER train-WER dev-clean-loss dev-clean-TER dev-clean-WER dev-other-loss dev-other-TER dev-other-WER avg-isz avg-tsz max-tsz hrs thrpt(sec/sec)

2020-05-26 18:16:17 1 4 0.000015 0.000015 00:00:15 3913.51 112.27 1820.82 635.73 1153.31 493.75 474.52074 0.00 0.00 236.83615 407.98 589.89 210.84624 474.91 669.18 1483 049 061 0.13 30.33 2020-05-26 23:03:02 2 35159 0.000015 0.000015 03:57:28 405.30 0.89 112.79 8.02 189.02 65.77 321.18602 217.97 332.97 182.20035 407.91 589.86 160.49133 474.85 669.14 1229 041 102 960.47 242.66 2020-05-27 03:51:46 3 70315 0.000015 0.000015 03:59:26 408.66 0.81 111.97 7.64 189.56 66.00 296.58603 231.24 356.60 174.73162 407.80 589.80 153.43288 474.68 669.01 1229 041 104 960.47 240.67 2020-05-27 08:37:57 4 105471 0.000015 0.000015 03:59:35 408.92 0.81 111.95 7.65 189.82 66.09 288.46082 210.69 353.08 171.21415 339.16 548.01 150.11066 385.46 605.89 1229 041 100 960.47 240.52 2020-05-27 13:12:58 5 140627 0.000015 0.000015 03:59:58 409.55 0.83 111.72 7.49 190.20 67.52 283.93068 189.01 343.27 168.98421 241.70 413.02 147.96650 262.60 435.70 1229 041 100 960.47 240.15 2020-05-27 17:43:50 6 175783 0.000015 0.000015 03:58:57 407.84 0.83 111.83 7.61 189.99 66.81 280.91889 182.03 335.07 167.06601 215.92 373.11 146.56506 231.11 386.52 1229 041 102 960.47 241.16 2020-05-27 22:13:52 7 210939 0.000015 0.000015 04:00:50 411.04 2.11 112.99 8.17 190.67 66.80 278.78642 181.12 334.50 166.01244 197.06 340.50 145.41342 208.79 347.56 1229 041 100 960.47 239.28 2020-05-28 02:44:17 8 246095 0.000015 0.000015 04:00:00 409.62 0.97 112.19 7.74 191.06 67.01 277.06449 180.49 333.80 164.91638 205.70 354.55 144.29309 216.86 360.08 1229 041 104 960.47 240.11 2020-05-28 07:12:19 9 281251 0.000015 0.000015 03:58:40 407.34 0.82 111.43 7.38 190.10 66.91 275.67091 180.19 333.59 164.14041 206.74 357.39 143.83753 218.28 363.58 1229 041 102 960.47 241.45 2020-05-28 11:40:52 10 316407 0.000015 0.000015 03:58:27 406.98 0.81 111.55 7.46 190.12 66.40 274.56721 179.35 332.38 163.38458 211.50 365.98 143.14340 222.45 371.31 1229 041 100 960.47 241.67 2020-05-28 16:09:42 11 351563 0.000015 0.000015 03:58:39 407.31 0.91 111.46 7.41 189.98 66.82 273.55285 179.89 333.18 162.81614 216.88 375.51 142.69722 227.90 381.10 1229 041 103 960.47 241.47 2020-05-28 20:40:59 12 386719 0.000015 0.000015 03:59:52 409.39 0.84 111.66 7.42 190.92 67.83 272.77044 178.82 331.49 162.13990 222.48 385.41 142.38879 233.06 390.72 1229 041 107 960.47 240.24 2020-05-29 01:14:28 13 421875 0.000015 0.000015 03:59:55 409.49 0.84 111.67 7.45 190.95 67.70 272.03696 179.71 333.26 161.56650 235.14 407.74 142.02924 248.08 417.02 1229 041 102 960.47 240.19 2020-05-29 05:47:37 14 457031 0.000015 0.000015 04:00:38 410.69 0.85 111.66 7.39 191.33 68.41 271.37681 179.71 333.30 161.41158 227.06 393.53 141.45859 236.68 397.38 1229 041 100 960.47 239.48 2020-05-29 10:21:41 15 492187 0.000015 0.000015 04:00:13 409.99 0.85 111.72 7.45 191.06 68.18 270.76164 178.83 331.85 160.79378 237.07 411.48 140.96830 249.30 420.08 1229 041 103 960.47 239.89 2020-05-29 14:55:24 16 527343 0.000015 0.000015 03:59:47 409.25 0.83 111.45 7.32 190.88 68.17 270.29345 177.79 330.12 160.59392 237.15 411.61 140.85086 248.86 419.42 1229 041 102 960.47 240.32 2020-05-29 19:30:01 17 562499 0.000015 0.000015 04:00:03 409.71 0.88 111.50 7.33 190.94 68.51 269.78664 177.31 329.13 160.38124 243.65 422.71 140.47210 256.32 432.16 1229 041 100 960.47 240.05 2020-05-30 00:05:38 18 597655 0.000015 0.000015 03:59:55 409.47 0.84 111.48 7.28 190.74 68.65 269.40423 177.00 328.78 160.02742 250.20 433.70 140.13095 264.97 446.57 1229 041 103 960.47 240.20 2020-05-30 04:41:21 19 632811 0.000015 0.000015 04:00:04 409.73 0.99 111.67 7.42 191.08 68.30 268.99767 176.54 328.07 159.64364 255.31 442.22 140.17385 271.30 457.02 1229 041 103 960.47 240.04 2020-05-30 09:16:43 20 667967 0.000015 0.000015 03:58:37 407.27 0.82 111.73 7.48 190.41 66.70 268.62477 176.39 327.60 159.58057 255.53 442.62 139.75895 270.28 455.52 1229 041 103 960.47 241.49 2020-05-30 13:51:37 21 703123 0.000015 0.000015 03:58:59 407.89 0.83 111.75 7.49 190.58 67.36 268.28157 175.00 325.05 159.39702 254.49 440.89 139.72042 267.11 450.61 1229 041 101 960.47 241.13 2020-05-30 18:27:29 22 738279 0.000015 0.000015 03:59:16 408.38 1.19 111.54 7.38 190.58 67.74 267.99973 174.86 324.96 159.09193 255.34 442.33 139.37328 267.21 450.94 1229 041 101 960.47 240.84 2020-05-30 23:03:43 23 773435 0.000015 0.000015 03:58:54 407.74 0.94 111.36 7.29 190.49 67.78 267.71163 173.55 322.70 158.95212 260.97 451.94 139.14714 273.32 461.28 1229 041 107 960.47 241.21 2020-05-31 03:39:53 24 808591 0.000015 0.000015 03:58:45 407.50 0.83 111.26 7.24 190.56 68.17 267.35995 171.47 319.09 159.02618 265.78 459.61 139.18588 280.13 472.16 1229 041 101 960.47 241.36 2020-05-31 08:15:03 25 843747 0.000015 0.000015 03:58:55 407.78 0.84 111.29 7.24 190.61 68.51 267.17583 170.99 318.09 158.31644 258.21 447.22 139.10440 267.74 452.26 1229 041 102 960.47 241.19 2020-05-31 12:54:33 26 878903 0.000015 0.000015 04:01:57 412.94 5.17 111.92 7.48 190.90 68.40 266.89739 169.91 316.00 158.25573 264.41 457.33 138.90373 276.28 466.05 1229 041 100 960.47 238.18 2020-05-31 17:30:26 27 914059 0.000015 0.000015 03:59:12 408.26 0.85 111.74 7.39 190.93 68.64 266.65983 168.98 314.41 158.35988 258.02 446.75 138.84695 266.04 449.36 1229 041 101 960.47 240.91 2020-05-31 22:05:08 28 949215 0.000015 0.000015 03:58:07 406.40 0.83 111.01 7.08 190.62 68.18 266.41760 167.68 311.99 158.33944 262.87 454.44 138.38775 272.09 458.76 1229 041 105 960.47 242.01 2020-06-01 02:39:08 29 984371 0.000015 0.000015 03:57:48 405.87 0.83 111.21 7.18 190.57 67.90 266.27566 165.56 308.10 158.13176 257.40 445.37 138.57675 262.73 443.50 1229 041 104 960.47 242.33 2020-06-01 07:14:13 30 1019527 0.000015 0.000015 03:57:58 406.14 0.84 111.45 7.31 190.59 67.99 266.02431 164.53 306.15 157.83917 262.45 453.90 138.30048 268.87 453.87 1229 041 103 960.47 242.16 2020-06-01 11:49:39 31 1054683 0.000015 0.000015 03:58:06 406.37 0.83 111.64 7.39 190.85 68.03 265.81333 163.17 303.63 157.69846 267.31 461.75 138.26481 276.02 465.29 1229 041 104 960.47 242.03 2020-06-01 16:24:13 32 1089839 0.000015 0.000015 03:57:33 405.42 0.84 111.31 7.22 190.84 68.12 265.65036 160.21 297.97 157.25775 262.68 453.95 138.11651 267.16 450.61 1229 041 105 960.47 242.59 2020-06-01 20:57:59 33 1124995 0.000015 0.000015 03:57:48 405.87 1.40 111.49 7.33 191.01 67.83 265.44579 158.58 295.08 157.44991 259.66 448.73 137.96345 261.35 440.85 1229 041 104 960.47 242.33 2020-06-02 01:30:49 34 1160151 0.000015 0.000015 03:57:15 404.93 0.83 111.60 7.36 190.85 67.97 265.26384 156.59 291.12 157.15497 255.27 441.08 138.09927 255.32 430.40 1229 041 100 960.47 242.89 2020-06-02 06:03:18 35 1195307 0.000015 0.000015 03:57:12 404.84 0.84 111.55 7.33 190.93 68.26 265.08748 154.37 286.52 157.30106 253.43 438.50 137.63830 251.99 425.76 1229 041 101 960.47 242.94 2020-06-02 10:36:21 36 1230463 0.000015 0.000015 03:56:52 404.27 1.34 111.58 7.38 190.74 67.97 264.94989 152.04 282.44 157.00974 259.10 447.15 137.64722 260.30 438.31 1229 041 105 960.47 243.29 2020-06-02 15:06:57 37 1265619 0.000015 0.000015 03:55:59 402.76 0.82 111.01 7.09 190.63 68.02 264.72618 149.94 278.68 157.24842 251.44 434.27 137.49073 248.66 419.03 1229 041 100 960.47 244.20 2020-06-02 19:38:01 38 1300775 0.000015 0.000015 03:56:03 402.88 0.89 111.27 7.22 190.93 68.28 264.64282 146.83 272.31 157.04727 251.75 435.09 137.50750 249.57 420.91 1229 041 105 960.47 244.12 2020-06-03 00:09:03 39 1335931 0.000015 0.000015 03:56:22 403.41 0.85 111.50 7.28 191.27 68.50 264.50003 144.91 269.09 156.94164 249.14 430.23 137.41813 245.09 412.66 1229 041 104 960.47 243.80 2020-06-03 04:40:05 40 1371087 0.000015 0.000015 03:56:24 403.48 0.85 111.59 7.30 191.31 68.80 264.32941 143.21 265.24 156.82633 246.52 425.50 137.44124 242.35 407.97 1229 041 100 960.47 243.76 2020-06-03 09:09:26 41 1406243 0.000015 0.000015 03:56:11 403.11 1.56 111.66 7.41 191.23 68.39 264.20440 139.47 258.69 156.76834 238.48 411.80 137.15032 231.17 388.78 1229 041 106 960.47 243.99 2020-06-03 13:37:53 42 1441399 0.000015 0.000015 03:55:09 401.34 0.85 111.43 7.27 191.12 68.43 264.05314 136.28 252.33 156.25451 236.36 407.17 137.35367 230.01 386.04 1229 041 102 960.47 245.06 2020-06-03 18:05:01 43 1476555 0.000015 0.000015 03:55:03 401.18 0.85 111.39 7.26 191.21 68.52 263.92446 134.62 249.15 156.34083 222.77 385.20 137.07056 211.87 356.63 1229 041 102 960.47 245.16 2020-06-03 22:32:00 44 1511711 0.000015 0.000015 03:55:20 401.64 1.40 111.57 7.29 191.25 68.87 263.81132 131.60 243.56 156.13673 224.64 386.34 137.01737 215.89 360.75 1229 041 108 960.47 244.88 2020-06-04 02:57:04 45 1546867 0.000015 0.000015 03:54:30 400.24 0.84 111.35 7.20 191.13 68.73 263.68357 130.06 240.18 156.15754 213.24 367.43 136.83997 201.39 336.82 1229 041 101 960.47 245.73 2020-06-04 07:22:37 46 1582023 0.000015 0.000015 03:54:09 399.62 0.84 111.49 7.31 191.09 68.64 263.54657 126.79 234.65 155.99286 216.07 371.32 136.92398 205.17 342.33 1229 041 100 960.47 246.11 2020-06-04 11:44:27 47 1617179 0.000015 0.000015 03:53:33 398.60 0.85 111.32 7.23 190.81 68.36 263.46954 125.35 231.10 156.05933 196.14 337.92 136.52624 183.28 305.61 1229 041 109 960.47 246.75 2020-06-04 16:07:30 48 1652335 0.000015 0.000015 03:53:08 397.91 1.16 111.46 7.30 190.68 68.28 263.35012 120.88 223.31 155.96741 198.54 341.13 136.54581 184.13 306.76 1229 041 103 960.47 247.17 2020-06-04 20:26:31 49 1687491 0.000015 0.000015 03:53:28 398.46 1.30 111.38 7.27 191.14 68.33 263.22798 120.83 222.15 155.83255 175.22 302.32 136.39798 159.83 265.14 1229 041 103 960.47 246.83 2020-06-05 00:46:21 50 1722647 0.000015 0.000015 03:53:22 398.31 0.84 111.96 7.46 191.25 68.73 263.09735 116.49 214.18 155.51932 166.92 287.96 136.25914 151.05 249.51 1229 041 102 960.47 246.92 2020-06-05 05:04:43 51 1757803 0.000015 0.000015 03:53:10 397.95 0.85 111.81 7.41 191.19 69.01 263.03153 115.00 210.58 155.49137 162.18 278.34 136.26025 145.05 238.48 1229 041 102 960.47 247.15 2020-06-05 09:20:53 52 1792959 0.000015 0.000015 03:52:39 397.08 0.89 111.93 7.54 191.03 68.42 262.91839 113.68 208.25 155.11989 150.45 258.30 136.21301 134.09 219.06 1229 041 103 960.47 247.69 2020-06-05 13:34:35 53 1828115 0.000015 0.000015 03:52:21 396.55 1.01 111.94 7.59 191.08 67.87 262.80065 112.19 205.74 155.48579 138.46 236.92 136.03214 123.06 198.55 1229 041 100 960.47 248.02 2020-06-05 17:44:45 54 1863271 0.000015 0.000015 03:51:16 394.72 1.20 111.91 7.57 190.58 67.03 262.70432 110.28 201.13 155.40727 120.43 202.53 136.24333 108.92 170.62 1229 041 107 960.47 249.17 2020-06-05 21:54:22 55 1898427 0.000015 0.000015 03:51:02 394.32 0.84 111.91 7.54 190.63 67.21 262.66040 109.39 198.59 155.32088 121.19 204.33 136.07195 109.23 171.03 1229 041 102 960.47 249.42 2020-06-06 02:02:10 56 1933583 0.000015 0.000015 03:51:15 394.69 0.83 111.46 7.28 190.78 68.15 262.54934 108.17 195.99 155.33469 115.00 191.11 136.31209 104.57 160.46 1229 041 102 960.47 249.19 2020-06-06 06:08:27 57 1968739 0.000015 0.000015 03:51:07 394.45 0.83 111.79 7.46 190.66 67.89 262.49295 107.05 194.12 155.23888 101.85 163.77 136.47731 95.33 140.18 1229 041 105 960.47 249.34 2020-06-06 10:13:20 58 2003895 0.000015 0.000015 03:50:24 393.22 0.84 112.09 7.64 190.48 67.03 262.37333 105.10 189.56 155.56919 99.54 157.12 136.81476 94.01 135.09 1229 041 100 960.47 250.12 2020-06-06 14:15:42 59 2039051 0.000015 0.000015 03:50:15 392.99 0.89 111.84 7.49 190.52 67.19 262.29969 104.72 187.61 155.69976 89.20 132.42 137.19604 87.89 118.95 1229 041 100 960.47 250.27 2020-06-06 18:16:41 60 2074207 0.000015 0.000015 03:50:12 392.89 0.82 111.61 7.36 190.52 67.89 262.15188 102.63 183.59 156.00474 86.63 122.56 138.01695 87.13 112.43 1229 041 105 960.47 250.33 2020-06-06 22:17:47 61 2109363 0.000015 0.000015 03:50:21 393.14 0.83 111.70 7.35 190.69 68.28 262.08771 100.49 179.25 156.11602 87.14 122.78 138.19123 88.16 113.35 1229 041 105 960.47 250.17 2020-06-07 02:15:43 62 2144519 0.000015 0.000015 03:49:45 392.11 0.83 111.32 7.18 190.70 68.19 262.00544 98.73 175.00 157.03518 84.37 111.11 139.27559 87.09 105.77 1229 041 100 960.47 250.83 2020-06-07 06:13:26 63 2179675 0.000015 0.000015 03:49:42 392.04 0.83 111.45 7.25 190.45 68.30 261.88611 99.04 174.97 158.18390 84.73 105.09 141.32088 87.92 101.99 1229 041 102 960.47 250.88 2020-06-07 10:10:28 64 2214831 0.000015 0.000015 03:49:29 391.66 0.83 111.75 7.43 190.39 67.80 261.86054 97.88 172.75 159.23077 85.01 102.95 142.36364 88.55 101.02 1229 041 102 960.47 251.12 2020-06-07 14:06:45 65 2249987 0.000015 0.000015 03:49:15 391.27 0.84 111.92 7.59 190.52 67.03 261.79236 98.74 173.88 159.85490 85.65 100.48 143.88864 89.25 99.70 1229 041 102 960.47 251.37 2020-06-07 18:02:35 66 2285143 0.000015 0.000015 03:49:17 391.33 0.91 111.86 7.54 190.68 67.26 261.65671 97.31 170.93 161.21840 86.98 98.59 145.18651 90.39 98.54 1229 041 103 960.47 251.33 2020-06-07 21:58:09 67 2320299 0.000015 0.000015 03:49:18 391.36 0.84 111.80 7.45 190.66 67.85 261.62688 95.86 167.45 162.51443 87.96 98.55 146.84209 91.10 98.46 1229 041 106 960.47 251.31 2020-06-08 01:52:53 68 2355455 0.000015 0.000015 03:49:00 390.84 0.83 111.72 7.42 190.46 67.78 261.53104 95.69 166.96 163.97196 89.29 97.16 148.08402 92.17 97.80 1229 041 100 960.47 251.64 2020-06-08 05:47:06 69 2390611 0.000015 0.000015 03:48:44 390.40 0.83 111.89 7.59 190.53 67.30 261.45357 94.26 164.21 165.44775 90.43 97.06 149.57780 93.06 97.91 1229 041 105 960.47 251.93 2020-06-08 09:40:17 70 2425767 0.000015 0.000015 03:48:30 390.00 0.82 111.99 7.60 190.59 66.83 261.34970 94.62 163.62 166.61415 91.37 96.85 150.78956 93.70 97.71 1229 041 105 960.47 252.19 2020-06-08 13:34:26 71 2460923 0.000015 0.000015 03:48:57 390.75 0.93 111.94 7.55 190.62 67.66 261.29525 94.23 162.09 166.95464 91.81 97.21 150.47135 94.06 97.91 1229 041 102 960.47 251.70 2020-06-08 17:27:45 72 2496079 0.000015 0.000015 03:49:16 391.31 0.84 111.62 7.35 190.85 68.57 261.25106 93.41 160.65 167.29638 92.49 96.98 150.46426 94.63 97.89 1229 041 103 960.47 251.34 2020-06-08 21:21:25 73 2531235 0.000015 0.000015 03:49:37 391.90 0.85 111.67 7.35 191.13 68.96 261.18468 92.29 159.03 167.77998 92.77 97.01 150.82855 94.85 97.88 1229 041 102 960.47 250.96 2020-06-09 01:14:12 74 2566391 0.000015 0.000015 03:48:57 390.77 0.85 111.47 7.28 190.69 68.52 261.13077 92.54 159.11 167.71907 93.32 96.97 150.43553 95.27 97.92 1229 041 103 960.47 251.69 2020-06-09 05:06:19 75 2601547 0.000015 0.000015 03:48:13 389.51 0.84 111.38 7.25 190.45 68.23 261.06694 90.13 154.19 166.59511 92.94 97.10 148.80614 94.97 98.00 1229 041 101 960.47 252.50 2020-06-09 08:57:42 76 2636703 0.000015 0.000015 03:47:40 388.57 0.82 111.52 7.37 190.10 67.46 260.98578 90.73 154.36 167.49420 93.68 97.07 149.74775 95.55 98.05 1229 041 104 960.47 253.11 2020-06-09 12:48:51 77 2671859 0.000015 0.000015 03:47:28 388.23 0.82 111.31 7.21 190.04 67.54 260.88555 90.14 153.49 167.35766 93.69 97.10 149.37206 95.61 98.09 1229 041 100 960.47 253.34 2020-06-09 16:45:58 78 2707015 0.000015 0.000015 03:53:26 398.40 9.98 112.48 7.75 189.96 67.65 260.86193 88.94 150.51 167.09488 93.73 97.06 148.49668 95.62 98.07 1229 041 104 960.47 246.87 2020-06-09 20:37:22 79 2742171 0.000015 0.000015 03:47:31 388.31 0.82 111.38 7.26 190.16 67.66 260.78765 89.50 151.32 166.98801 94.00 97.19 148.57394 95.88 98.14 1229 041 105 960.47 253.28 2020-06-10 00:28:18 80 2777327 0.000015 0.000015 03:47:15 387.85 0.82 111.23 7.14 190.25 67.76 260.72257 87.57 147.22 165.71631 93.88 97.16 147.62807 95.80 98.15 1229 041 102 960.47 253.59 2020-06-10 04:18:53 81 2812483 0.000015 0.000015 03:46:54 387.25 0.82 111.22 7.22 190.43 67.04 260.66833 87.82 147.18 165.80330 93.88 97.02 146.97341 95.77 98.12 1229 041 101 960.47 253.97 2020-06-10 08:09:48 82 2847639 0.000015 0.000015 03:47:18 387.95 1.19 111.78 7.50 190.55 66.64 260.59017 87.50 146.58 165.30953 93.99 96.94 146.64897 95.89 98.12 1229 041 100 960.47 253.52 2020-06-10 11:59:56 83 2882795 0.000015 0.000015 03:46:34 386.70 0.82 111.56 7.39 190.12 66.61 260.52372 86.82 145.48 165.39087 94.19 97.05 146.77371 95.99 98.17 1229 041 102 960.47 254.34 2020-06-10 15:49:38 84 2917951 0.000015 0.000015 03:46:09 385.97 0.80 111.35 7.29 189.82 66.72 260.43491 85.85 143.12 164.90279 94.12 97.02 145.50106 95.97 98.16 1229 041 107 960.47 254.82 2020-06-10 19:39:27 85 2953107 0.000015 0.000015 03:46:20 386.29 0.81 111.47 7.30 190.04 66.79 260.41393 85.62 142.22 164.30638 94.09 96.77 145.04400 95.92 98.04 1229 041 104 960.47 254.61 2020-06-10 23:30:58 86 2988263 0.000015 0.000015 03:47:30 388.28 0.83 111.80 7.45 190.71 67.44 260.34703 86.28 143.44 165.07110 94.65 96.90 145.75688 96.33 98.11 1229 041 103 960.47 253.30 2020-06-11 03:22:43 87 3023419 0.000015 0.000015 03:48:00 389.14 0.87 111.85 7.46 191.29 68.07 260.28478 84.43 139.54 163.27011 93.86 96.71 144.23765 95.82 98.05 1229 041 104 960.47 252.75 2020-06-11 07:14:44 88 3058575 0.000015 0.000015 03:47:58 389.07 1.18 111.88 7.50 191.18 67.84 260.22713 84.46 139.56 163.29060 94.25 96.73 144.52239 96.07 98.07 1229 041 102 960.47 252.79 2020-06-11 11:06:06 89 3093731 0.000015 0.000015 03:47:43 388.66 0.85 111.65 7.39 191.00 68.18 260.15900 84.96 139.74 162.92082 94.09 96.65 143.75726 95.94 98.02 1229 041 106 960.47 253.06 2020-06-11 14:58:05 90 3128887 0.000015 0.000015 03:47:46 388.75 0.84 111.66 7.38 190.96 68.38 260.15623 84.63 139.22 162.62590 93.97 96.56 143.44386 95.86 97.97 1229 041 102 960.47 253.00 2020-06-11 18:49:58 91 3164043 0.000015 0.000015 03:47:47 388.76 0.84 111.90 7.48 191.03 68.21 260.08566 84.14 137.79 162.29835 94.05 96.41 143.09313 95.93 97.95 1229 041 102 960.47 252.99 2020-06-11 22:42:23 92 3199199 0.000015 0.000015 03:48:14 389.54 0.86 111.94 7.46 191.34 68.86 260.01662 82.87 134.86 162.04375 94.08 96.46 142.87217 95.92 97.98 1229 041 100 960.47 252.48 2020-06-12 02:33:42 93 3234355 0.000015 0.000015 03:47:41 388.60 0.84 111.51 7.26 191.13 68.63 260.00366 82.93 135.21 162.12996 94.15 96.39 142.69295 96.01 97.96 1229 041 100 960.47 253.10 2020-06-12 06:24:58 94 3269511 0.000015 0.000015 03:47:11 387.74 0.84 111.58 7.32 190.74 68.28 259.92158 81.81 132.99 161.31622 93.75 96.31 141.85798 95.66 97.82 1229 041 100 960.47 253.66 2020-06-12 10:14:56 95 3304667 0.000015 0.000015 03:45:48 385.40 0.82 111.60 7.37 189.85 67.27 259.85186 80.94 130.38 161.72460 94.00 96.39 142.38152 95.84 97.92 1229 041 102 960.47 255.20 2020-06-12 14:06:30 96 3339823 0.000015 0.000015 03:47:54 388.98 1.58 111.36 7.24 190.92 68.78 259.83224 82.50 134.06 161.25779 94.01 96.29 142.05668 95.85 97.88 1229 041 102 960.47 252.85 2020-06-12 17:58:14 97 3374979 0.000015 0.000015 03:47:28 388.23 0.85 111.81 7.44 191.04 68.45 259.74877 81.38 131.66 160.57660 93.40 96.10 141.10853 95.38 97.71 1229 041 100 960.47 253.33 2020-06-12 21:50:11 98 3410135 0.000015 0.000015 03:47:28 388.23 0.84 112.02 7.53 191.05 68.24 259.72790 81.14 131.21 161.12000 93.87 96.12 141.24080 95.76 97.82 1229 041 102 960.47 253.34 2020-06-13 01:40:13 99 3445291 0.000015 0.000015 03:46:25 386.43 0.85 111.22 7.30 190.23 68.01 259.67564 82.00 132.53 160.31366 93.48 95.93 140.73836 95.40 97.63 1229 041 115 960.47 254.52 2020-06-13 05:31:20 100 3480447 0.000015 0.000015 03:46:29 386.56 0.85 111.75 7.53 190.41 67.69 259.59484 80.73 129.88 160.14576 93.51 95.90 140.57077 95.44 97.61 1229 041 101 960.47 254.43 2020-06-13 09:22:53 101 3515603 0.000015 0.000015 03:46:53 387.23 0.84 112.20 7.70 190.82 67.22 259.59996 81.46 131.64 160.19224 93.61 95.98 140.77480 95.50 97.71 1229 041 100 960.47 253.99 2020-06-13 13:13:16 102 3550759 0.000015 0.000015 03:46:41 386.89 0.84 111.63 7.38 190.73 67.90 259.49187 80.64 129.66 159.54582 93.27 95.78 140.12226 95.22 97.55 1229 041 102 960.47 254.22 2020-06-13 17:04:51 103 3585915 0.000015 0.000015 03:46:57 387.34 0.84 111.79 7.46 190.83 68.11 259.44423 80.32 128.80 159.33418 93.15 95.69 140.03028 95.04 97.46 1229 041 100 960.47 253.92 2020-06-13 20:56:35 104 3621071 0.000015 0.000015 03:47:01 387.46 0.85 111.83 7.43 191.01 68.14 259.37726 79.75 127.66 159.42352 93.10 95.66 139.54446 94.96 97.44 1229 041 103 960.47 253.84 2020-06-14 00:48:05 105 3656227 0.000015 0.000015 03:46:50 387.14 0.85 111.66 7.37 190.75 68.29 259.38499 79.94 127.68 159.12320 93.08 95.65 139.55264 94.96 97.44 1229 041 104 960.47 254.05 2020-06-14 04:39:30 106 3691383 0.000015 0.000015 03:47:15 387.87 1.09 111.52 7.29 190.91 68.62 259.35301 80.80 128.80 158.86960 93.02 95.56 139.62160 94.85 97.31 1229 041 102 960.47 253.57 2020-06-14 08:31:06 107 3726539 0.000015 0.000015 03:46:56 387.32 0.84 111.91 7.52 190.90 68.19 259.28721 79.28 126.50 159.04308 93.26 95.57 139.75505 95.05 97.33 1229 041 102 960.47 253.93 2020-06-14 12:21:45 108 3761695 0.000015 0.000015 03:46:54 387.27 0.85 111.40 7.25 190.87 68.65 259.26876 79.27 126.39 158.61104 92.85 95.47 139.03558 94.71 97.22 1229 041 102 960.47 253.97 2020-06-14 16:13:37 109 3796851 0.000015 0.000015 03:47:06 387.60 0.98 111.54 7.29 190.78 68.73 259.22552 79.64 127.18 158.71878 92.78 95.41 139.18106 94.56 97.16 1229 041 103 960.47 253.75 2020-06-14 20:05:32 110 3832007 0.000015 0.000015 03:47:16 387.87 0.85 111.53 7.27 191.06 68.85 259.18394 79.89 127.55 158.49532 92.69 95.38 139.02144 94.51 97.11 1229 041 103 960.47 253.57 2020-06-14 23:57:24 111 3867163 0.000015 0.000015 03:47:09 387.67 0.84 111.58 7.29 190.88 68.84 259.10165 79.07 126.22 158.58265 92.96 95.43 138.68741 94.66 97.10 1229 041 104 960.47 253.70 2020-06-15 03:48:01 112 3902319 0.000015 0.000015 03:46:19 386.27 0.83 111.40 7.24 190.58 68.06 259.05985 79.28 126.14 157.63804 92.07 95.12 138.23064 94.03 96.90 1229 041 100 960.47 254.62 2020-06-15 07:38:40 113 3937475 0.000015 0.000015 03:45:56 385.62 0.83 111.55 7.38 190.32 67.64 258.99092 78.42 125.14 158.16593 92.43 95.17 138.53234 94.26 96.93 1229 041 103 960.47 255.05 2020-06-15 11:28:16 114 3972631 0.000015 0.000015 03:45:48 385.38 0.82 111.33 7.26 190.20 67.70 258.99155 79.17 125.78 157.84247 92.08 95.03 137.97870 93.94 96.83 1229 041 102 960.47 255.21 2020-06-15 15:19:23 115 4007787 0.000015 0.000015 03:46:21 386.33 0.84 111.60 7.37 190.38 68.03 258.92076 79.04 126.08 157.71754 91.96 94.97 137.97817 93.85 96.79 1229 041 110 960.47 254.58 2020-06-15 19:10:29 116 4042943 0.000015 0.000015 03:46:16 386.17 0.84 112.06 7.63 190.61 67.33 258.89442 78.44 124.78 157.52986 91.80 94.91 137.74699 93.69 96.72 1229 041 103 960.47 254.69 2020-06-15 23:02:09 117 4078099 0.000015 0.000015 03:46:39 386.84 0.84 112.16 7.65 190.65 67.63 258.84706 79.46 126.34 157.41823 91.91 94.83 137.80123 93.72 96.64 1229 041 100 960.47 254.25 2020-06-16 02:53:22 118 4113255 0.000015 0.000015 03:46:16 386.18 0.83 112.08 7.65 190.42 67.09 258.83961 80.18 128.12 156.99349 91.67 94.83 137.72929 93.53 96.61 1229 041 102 960.47 254.68 2020-06-16 06:44:03 119 4148411 0.000015 0.000015 03:46:18 386.25 0.83 111.72 7.44 190.44 67.61 258.78283 79.88 127.08 157.02879 91.45 94.82 137.52548 93.37 96.63 1229 041 103 960.47 254.64 2020-06-16 10:34:07 120 4183567 0.000015 0.000015 03:45:43 385.23 0.82 111.65 7.39 189.95 67.21 258.72236 79.76 127.18 156.98670 91.46 94.73 137.49244 93.34 96.52 1229 041 102 960.47 255.31 2020-06-16 14:24:47 121 4218723 0.000015 0.000015 03:45:44 385.27 1.18 111.47 7.31 189.84 67.32 258.72927 79.44 126.46 156.54345 90.93 94.58 137.10734 92.84 96.47 1229 041 105 960.47 255.29 2020-06-16 18:15:13 122 4253879 0.000015 0.000015 03:45:38 385.10 0.81 111.63 7.40 189.83 67.13 258.65746 80.03 127.80 156.85398 91.13 94.59 137.05342 92.99 96.43 1229 041 104 960.47 255.39 2020-06-16 22:04:59 123 4289035 0.000015 0.000015 03:45:20 384.59 0.81 111.51 7.31 189.68 66.98 258.59703 79.68 127.37 156.36957 90.62 94.49 136.70222 92.58 96.41 1229 041 103 960.47 255.73 2020-06-17 01:55:55 124 4324191 0.000015 0.000015 03:45:51 385.46 0.83 111.71 7.41 190.07 67.40 258.52723 78.74 125.55 156.69372 90.81 94.47 136.93204 92.59 96.29 1229 041 105 960.47 255.16 2020-06-17 05:47:19 125 4359347 0.000015 0.000015 03:46:23 386.38 0.83 111.73 7.43 190.46 67.81 258.52271 79.04 126.59 156.15160 90.68 94.45 136.96404 92.61 96.33 1229 041 106 960.47 254.55 2020-06-17 09:38:51 126 4394503 0.000015 0.000015 03:46:58 387.38 1.20 111.76 7.45 190.68 67.99 258.49495 79.49 127.62 156.10806 90.48 94.35 136.68027 92.30 96.20 1229 041 100 960.47 253.89 2020-06-17 13:30:57 127 4429659 0.000015 0.000015 03:46:58 387.38 0.84 111.67 7.40 190.61 68.46 258.45029 79.95 128.36 156.08215 90.22 94.29 136.59479 92.07 96.24 1229 041 102 960.47 253.89 2020-06-17 17:22:30 128 4464815 0.000015 0.000015 03:46:56 387.31 0.84 111.58 7.36 190.75 68.29 258.42800 80.26 128.91 156.05849 90.03 94.28 136.35127 91.92 96.12 1229 041 100 960.47 253.94 2020-06-17 21:15:02 129 4499971 0.000015 0.000015 03:47:19 387.98 0.84 111.83 7.44 190.97 68.49 258.36758 79.71 128.31 156.13134 90.23 94.22 136.45945 92.05 96.17 1229 041 102 960.47 253.50 2020-06-18 01:06:57 130 4535127 0.000015 0.000015 03:47:16 387.90 0.85 111.63 7.33 190.92 68.61 258.33606 80.20 128.83 156.21171 90.26 94.28 136.72820 92.12 96.18 1229 041 102 960.47 253.55 2020-06-18 04:58:02 131 4570283 0.000015 0.000015 03:46:53 387.23 0.84 111.43 7.25 190.74 68.39 258.30001 79.73 128.51 155.69105 89.62 94.15 136.11993 91.54 96.10 1229 041 104 960.47 253.99 2020-06-18 08:50:34 132 4605439 0.000015 0.000015 03:47:19 387.98 1.18 111.96 7.58 190.75 67.97 258.28554 80.78 130.36 155.79097 89.42 94.09 136.26015 91.29 95.97 1229 041 103 960.47 253.50 2020-06-18 12:43:11 133 4640595 0.000015 0.000015 03:47:21 388.03 0.84 111.95 7.54 190.94 68.16 258.20357 80.64 130.37 155.58804 89.45 94.07 136.00232 91.27 96.01 1229 041 103 960.47 253.46 2020-06-18 16:35:50 134 4675751 0.000015 0.000015 03:47:48 388.80 0.86 111.84 7.45 191.12 68.68 258.15152 81.45 131.73 155.33767 89.09 94.00 135.72141 90.95 95.95 1229 041 103 960.47 252.96 2020-06-18 20:28:52 135 4710907 0.000015 0.000015 03:47:39 388.54 0.84 112.00 7.51 191.08 68.21 258.18005 81.61 132.45 155.46899 89.17 94.01 135.80036 90.98 95.94 1229 041 101 960.47 253.13 2020-06-19 00:21:09 136 4746063 0.000015 0.000015 03:47:23 388.09 0.84 111.71 7.38 190.89 68.30 258.12683 81.74 132.70 155.32771 88.76 93.98 135.82437 90.65 95.95 1229 041 103 960.47 253.43 2020-06-19 04:13:23 137 4781219 0.000015 0.000015 03:47:23 388.09 1.03 111.63 7.33 190.84 68.31 258.06615 81.43 131.69 155.19278 88.74 93.96 135.81947 90.58 95.88 1229 041 101 960.47 253.43 2020-06-19 08:06:18 138 4816375 0.000015 0.000015 03:47:32 388.35 0.89 111.94 7.54 190.94 68.31 258.11559 81.12 131.50 154.93656 88.38 93.95 135.55102 90.32 95.97 1229 041 104 960.47 253.26 2020-06-19 11:58:36 139 4851531 0.000015 0.000015 03:47:14 387.84 0.84 111.65 7.38 190.72 68.21 258.00978 81.49 132.92 154.47822 87.93 93.86 135.32953 89.82 95.90 1229 041 105 960.47 253.59 2020-06-19 15:50:50 140 4886687 0.000015 0.000015 03:47:18 387.94 0.84 111.54 7.29 190.74 68.43 257.98566 81.66 132.74 154.90387 88.27 93.90 135.54692 90.15 95.87 1229 041 102 960.47 253.53 2020-06-19 19:43:22 141 4921843 0.000015 0.000015 03:47:32 388.35 0.84 111.60 7.30 190.93 68.48 257.93742 82.22 134.02 154.53823 87.87 93.85 135.15855 89.70 95.85 1229 041 101 960.47 253.26 2020-06-19 23:36:28 142 4956999 0.000015 0.000015 03:47:37 388.49 0.84 111.78 7.38 190.91 68.42 257.92868 82.19 134.04 154.31551 87.68 93.76 135.12497 89.56 95.80 1229 041 102 960.47 253.17 2020-06-20 03:29:20 143 4992155 0.000015 0.000015 03:47:20 387.99 0.83 111.75 7.41 190.70 68.05 257.91130 82.68 135.26 154.53828 87.58 93.84 135.29013 89.58 96.11 1229 041 102 960.47 253.49 2020-06-20 07:20:29 144 5027311 0.000015 0.000015 03:46:29 386.56 0.83 111.55 7.34 190.20 67.41 257.82463 82.31 134.48 154.29588 87.13 93.76 135.01481 89.23 96.09 1229 041 102 960.47 254.43 2020-06-20 11:12:44 145 5062467 0.000015 0.000015 03:47:07 387.62 1.31 111.85 7.47 190.31 67.50 257.77365 82.21 134.63 154.08671 86.96 93.71 134.89036 89.05 95.94 1229 041 102 960.47 253.74 2020-06-20 15:05:09 146 5097623 0.000015 0.000015 03:47:11 387.75 0.83 111.59 7.30 190.56 68.12 257.79496 82.14 134.55 154.14262 86.94 93.66 134.88979 89.02 96.02 1229 041 102 960.47 253.65 2020-06-20 18:57:22 147 5132779 0.000015 0.000015 03:46:59 387.40 0.83 111.52 7.27 190.47 67.95 257.76017 82.31 134.64 154.10222 86.47 93.63 134.67872 88.67 96.11 1229 041 100 960.47 253.88 2020-06-20 22:49:33 148 5167935 0.000015 0.000015 03:46:57 387.36 0.82 111.53 7.29 190.37 67.86 257.74441 82.63 135.72 153.90700 86.05 93.63 134.46129 88.34 96.10 1229 041 101 960.47 253.91 2020-06-21 02:41:14 149 5203091 0.000015 0.000015 03:46:57 387.33 1.16 111.34 7.20 190.26 67.77 257.70611 82.99 136.52 153.81132 86.24 93.66 134.47911 88.29 95.96 1229 041 106 960.47 253.92 2020-06-21 06:32:48 150 5238247 0.000015 0.000015 03:46:52 387.20 0.82 111.27 7.17 190.29 67.90 257.68441 83.54 137.33 153.86353 86.28 93.56 134.70497 88.41 95.98 1229 041 101 960.47 254.01 2020-06-21 10:25:00 151 5273403 0.000015 0.000015 03:46:59 387.40 0.83 111.58 7.33 190.13 67.82 257.59082 83.65 138.04 153.91617 86.07 93.62 134.79177 88.16 95.97 1229 041 102 960.47 253.88 2020-06-21 14:17:03 152 5308559 0.000015 0.000015 03:47:08 387.65 0.83 111.24 7.17 190.37 68.10 257.53004 83.89 138.84 153.68358 85.55 93.57 134.23774 87.78 95.89 1229 041 107 960.47 253.71 2020-06-21 18:10:24 153 5343715 0.000015 0.000015 03:48:26 389.86 1.20 111.80 7.24 191.23 68.38 257.54859 84.31 139.16 153.62729 85.27 93.57 134.28350 87.42 95.79 1229 041 102 960.47 252.28 2020-06-21 22:04:36 154 5378871 0.000015 0.000015 03:48:39 390.26 0.83 112.40 7.41 191.46 68.32 257.54569 84.40 139.85 153.08680 84.67 93.49 134.00631 86.88 95.70 1229 041 101 960.47 252.02 2020-06-22 01:58:24 155 5414027 0.000015 0.000015 03:47:48 388.81 0.83 111.94 7.45 190.66 68.12 257.48899 84.30 139.57 153.09680 84.34 93.51 133.80229 86.49 95.62 1229 041 100 960.47 252.96 2020-06-22 05:50:53 156 5449183 0.000015 0.000015 03:47:00 387.42 0.82 111.35 7.20 190.31 67.71 257.45502 84.88 140.53 153.14582 84.31 93.47 134.03258 86.45 95.56 1229 041 103 960.47 253.86 2020-06-22 09:53:39 157 5484339 0.000015 0.000015 03:56:45 404.06 16.93 113.09 8.01 190.05 66.41 257.39802 85.21 141.87 153.08824 84.22 93.52 133.78610 86.39 95.62 1229 041 104 960.47 243.41 2020-06-22 13:44:44 158 5519495 0.000015 0.000015 03:46:00 385.71 0.81 112.34 7.83 189.49 65.47 257.35797 86.13 143.31 153.15055 83.76 93.56 133.74750 85.98 95.57 1229 041 103 960.47 254.99 2020-06-22 17:35:21 159 5554651 0.000015 0.000015 03:45:35 385.01 0.81 111.33 7.22 189.09 66.36 257.33948 85.39 142.42 152.97234 83.60 93.53 133.79695 85.78 95.60 1229 041 100 960.47 255.46 2020-06-22 21:27:32 160 5589807 0.000015 0.000015 03:46:54 387.25 0.83 111.41 7.33 189.81 67.29 257.31320 87.08 145.36 152.86047 83.31 93.59 133.82100 85.55 95.59 1229 041 112 960.47 253.98 2020-06-23 01:20:31 161 5624963 0.000015 0.000015 03:47:41 388.61 0.84 111.62 7.33 190.48 67.96 257.28598 86.09 143.63 153.11426 83.35 93.63 133.73761 85.60 95.71 1229 041 100 960.47 253.09 2020-06-23 05:13:50 162 5660119 0.000015 0.000015 03:47:58 389.08 0.83 111.43 7.22 190.66 68.24 257.21608 87.10 145.72 152.84156 83.08 93.61 133.58709 85.26 95.56 1229 041 101 960.47 252.78 2020-06-23 09:07:41 163 5695275 0.000015 0.000015 03:47:56 389.02 1.30 111.76 7.45 190.41 67.62 257.23799 86.89 145.51 152.82700 82.54 93.69 133.43517 84.74 95.58 1229 041 102 960.47 252.82 2020-06-23 13:02:16 164 5730431 0.000015 0.000015 03:48:54 390.67 1.20 111.57 7.30 191.03 68.70 257.22058 87.99 147.77 152.72781 82.19 93.87 133.49828 84.46 95.70 1229 041 106 960.47 251.75 2020-06-23 16:56:59 165 5765587 0.000015 0.000015 03:49:00 390.85 0.84 111.49 7.26 191.21 68.96 257.17852 89.09 148.94 152.50482 82.07 93.88 133.26595 84.33 95.68 1229 041 100 960.47 251.64 2020-06-23 20:51:27 166 5800743 0.000015 0.000015 03:48:43 390.35 0.85 111.47 7.34 190.92 68.74 257.12018 88.68 148.78 152.45420 81.61 93.98 133.23615 83.82 95.56 1229 041 103 960.47 251.96 2020-06-24 00:46:15 167 5835899 0.000015 0.000015 03:48:55 390.71 0.85 111.29 7.36 190.73 69.23 257.08309 89.78 150.81 152.30927 81.50 93.90 133.35587 83.74 95.59 1229 041 104 960.47 251.73 2020-06-24 04:41:15 168 5871055 0.000015 0.000015 03:49:08 391.07 0.83 111.65 7.30 191.38 68.66 257.09030 89.51 150.36 152.37439 81.20 93.90 133.04801 83.50 95.51 1229 041 104 960.47 251.50 2020-06-24 08:37:07 169 5906211 0.000015 0.000015 03:49:20 391.42 0.84 112.08 7.57 191.47 68.12 257.01927 90.92 153.28 152.48178 80.86 93.90 133.15829 83.08 95.46 1229 041 105 960.47 251.27 2020-06-24 12:32:58 170 5941367 0.000015 0.000015 03:49:13 391.21 0.84 112.10 7.60 191.44 67.77 257.00670 91.56 154.54 152.15277 80.31 94.23 132.94506 82.65 95.71 1229 041 101 960.47 251.41 2020-06-24 16:28:00 171 5976523 0.000015 0.000015 03:48:48 390.50 0.83 111.54 7.28 191.17 68.19 256.97052 91.21 153.75 152.12403 79.83 94.51 132.88262 82.08 95.79 1229 041 102 960.47 251.86 2020-06-24 20:22:43 172 6011679 0.000015 0.000015 03:48:34 390.09 0.84 111.46 7.24 190.88 68.11 256.90750 91.58 154.05 152.16270 80.04 94.36 133.01482 82.24 95.73 1229 041 102 960.47 252.13 2020-06-25 00:16:50 173 6046835 0.000015 0.000015 03:47:58 389.07 0.83 111.60 7.36 190.52 67.04 256.87018 93.30 157.30 152.02816 79.75 94.29 132.82660 81.91 95.58 1229 041 103 960.47 252.79 2020-06-25 04:11:05 174 6081991 0.000015 0.000015 03:47:50 388.84 0.83 111.52 7.31 190.33 67.17 256.89640 92.87 156.69 151.68679 79.08 94.78 132.55885 81.29 95.97 1229 041 103 960.47 252.94 2020-06-25 08:06:22 175 6117147 0.000015 0.000015 03:48:54 390.66 2.61 111.59 7.39 190.33 66.97 256.81383 94.17 158.73 151.87699 79.17 94.79 132.67529 81.33 95.94 1229 041 100 960.47 251.76 2020-06-25 12:02:23 176 6152303 0.000015 0.000015 03:49:42 392.03 3.64 111.54 7.36 190.21 67.30 256.81824 94.31 159.55 151.71694 79.11 94.67 132.70313 81.23 95.89 1229 041 102 960.47 250.88 2020-06-25 15:59:40 177 6187459 0.000015 0.000015 03:50:39 393.66 4.95 111.50 7.35 190.38 67.47 256.79767 95.22 160.59 151.57764 78.60 95.62 132.44590 80.80 96.66 1229 041 104 960.47 249.84 2020-06-25 19:54:28 178 6222615 0.000015 0.000015 03:48:15 389.58 0.82 111.52 7.30 190.39 67.44 256.74600 95.90 161.68 151.31898 78.43 95.51 132.48649 80.32 95.98 1229 041 100 960.47 252.46 2020-06-25 23:49:44 179 6257771 0.000015 0.000015 03:48:30 389.98 0.82 111.60 7.37 190.78 67.16 256.70996 96.51 162.67 151.48194 78.12 96.26 132.31816 80.21 96.92 1229 041 100 960.47 252.20 2020-06-26 03:45:15 180 6292927 0.000015 0.000015 03:48:28 389.94 0.82 111.59 7.38 190.70 67.10 256.67377 97.37 163.72 151.58034 77.81 96.15 132.11481 79.96 96.95 1229 041 102 960.47 252.22 2020-06-26 07:41:22 181 6328083 0.000015 0.000015 03:49:03 390.93 0.83 111.79 7.46 191.08 67.34 256.69717 97.62 164.68 151.32011 77.82 97.21 132.41029 79.85 97.71 1229 041 105 960.47 251.59 2020-06-26 11:37:51 182 6363239 0.000015 0.000015 03:49:22 391.47 0.84 111.74 7.43 191.09 67.83 256.64246 97.80 165.07 151.45288 77.71 96.85 132.11498 79.74 97.37 1229 041 101 960.47 251.24 2020-06-26 15:34:29 183 6398395 0.000015 0.000015 03:49:30 391.70 0.84 111.52 7.29 191.10 68.31 256.57251 98.19 165.39 151.32559 77.56 97.69 132.15515 79.50 97.80 1229 041 107 960.47 251.09 2020-06-26 19:31:00 184 6433551 0.000015 0.000015 03:49:15 391.27 0.83 111.49 7.26 190.87 68.17 256.58194 98.48 165.61 151.27784 77.87 98.51 132.22312 79.66 98.41 1229 041 100 960.47 251.37 2020-06-26 23:27:59 185 6468707 0.000015 0.000015 03:49:36 391.86 0.84 111.76 7.42 191.14 67.79 256.56800 100.21 168.75 151.13067 77.27 98.46 132.07044 79.23 98.33 1229 041 102 960.47 250.99 2020-06-27 03:24:36 186 6503863 0.000015 0.000015 03:49:13 391.22 0.84 111.72 7.44 190.96 67.43 256.52746 100.70 169.45 151.00694 77.21 98.54 131.98985 78.89 97.88 1229 041 102 960.47 251.40 2020-06-27 07:20:32 187 6539019 0.000015 0.000015 03:48:10 389.43 0.83 111.17 7.47 189.89 67.39 256.51928 99.82 167.80 151.00005 77.78 100.62 132.17019 79.27 99.77 1229 041 102 960.47 252.55 2020-06-27 11:16:29 188 6574175 0.000015 0.000015 03:48:12 389.47 0.84 111.05 7.43 189.82 67.54 256.47559 100.82 169.37 151.00809 77.58 100.62 131.99761 79.06 99.56 1229 041 106 960.47 252.53 2020-06-27 15:13:04 189 6609331 0.000015 0.000015 03:48:50 390.56 0.83 111.12 7.32 190.21 68.05 256.42094 101.38 170.24 151.10533 77.89 100.94 132.06297 79.16 99.95 1229 041 103 960.47 251.83 2020-06-27 19:10:30 190 6644487 0.000015 0.000015 03:49:31 391.73 0.83 111.39 7.20 190.89 68.25 256.39708 102.00 171.11 151.01147 77.78 102.10 131.93125 78.74 100.16 1229 041 101 960.47 251.07 2020-06-27 23:08:09 191 6679643 0.000015 0.000015 03:49:33 391.77 0.84 111.28 7.14 191.00 68.48 256.38072 101.26 169.69 151.00810 77.91 103.47 131.86704 78.44 100.63 1229 041 100 960.47 251.05 2020-06-28 03:05:55 192 6714799 0.000015 0.000015 03:49:39 391.95 0.83 111.22 7.14 191.00 68.45 256.33752 102.69 171.88 150.87421 77.98 103.28 131.86192 78.59 101.01 1229 041 100 960.47 250.93 2020-06-28 07:04:45 193 6749955 0.000015 0.000015 03:50:32 393.47 0.85 111.30 7.16 191.40 69.36 256.31688 102.92 172.10 150.82075 78.22 103.42 131.78997 78.78 101.08 1229 041 102 960.47 249.96 2020-06-28 11:03:48 194 6785111 0.000015 0.000015 03:50:38 393.62 0.86 111.43 7.24 191.48 69.34 256.26593 102.91 171.82 150.95000 78.00 103.94 131.65181 78.48 101.11 1229 041 102 960.47 249.87 2020-06-28 15:03:04 195 6820267 0.000015 0.000015 03:50:48 393.91 0.91 111.34 7.19 191.41 69.28 256.24342 105.64 176.11 151.08256 78.55 105.50 131.90320 78.74 102.23 1229 041 103 960.47 249.68 2020-06-28 19:01:39 196 6855423 0.000015 0.000015 03:49:58 392.50 0.85 111.24 7.18 191.11 68.72 256.21514 104.34 173.55 150.69532 78.97 106.78 131.88351 78.77 102.85 1229 041 104 960.47 250.58 2020-06-28 23:00:34 197 6890579 0.000015 0.000015 03:50:23 393.21 0.84 111.44 7.27 191.34 68.94 256.20526 104.40 174.00 150.84328 78.83 106.41 131.70985 78.52 102.17 1229 041 103 960.47 250.12 2020-06-29 02:59:03 198 6925735 0.000015 0.000015 03:49:46 392.15 0.84 111.49 7.30 190.93 68.17 256.15593 105.53 175.09 150.82220 79.21 107.93 131.79739 78.64 103.23 1229 041 102 960.47 250.80 2020-06-29 06:57:45 199 6960891 0.000015 0.000015 03:50:03 392.63 0.84 111.54 7.32 191.08 68.43 256.12402 105.42 175.16 150.81605 79.23 107.50 132.01930 78.57 102.88 1229 041 100 960.47 250.50 2020-06-29 10:54:21 200 6996047 0.000015 0.000015 03:47:59 389.10 0.82 111.25 7.22 189.75 67.01 256.17794 104.53 172.70 150.66513 79.81 109.19 131.79386 78.96 104.25 1229 041 106 960.47 252.77 2020-06-29 14:51:51 201 7031203 0.000015 0.000015 03:48:33 390.09 0.82 111.18 7.17 190.01 67.59 256.09032 105.38 173.95 150.58352 80.31 110.42 131.61675 79.00 104.56 1229 041 102 960.47 252.13 2020-06-29 18:50:43 202 7066359 0.000015 0.000015 03:49:52 392.31 0.84 111.38 7.22 190.85 68.52 256.03822 106.46 175.50 150.85309 80.53 111.42 131.66152 79.06 105.16 1229 041 103 960.47 250.70 2020-06-29 22:50:18 203 7101515 0.000015 0.000015 03:50:26 393.29 0.84 111.52 7.25 191.36 69.02 256.02067 105.62 173.91 150.50949 80.46 111.08 131.65813 79.23 105.36 1229 041 102 960.47 250.08 2020-06-30 02:50:09 204 7136671 0.000015 0.000015 03:50:29 393.38 0.89 111.51 7.24 191.33 69.10 256.01354 106.04 174.12 150.71239 80.90 112.89 131.59816 79.32 106.25 1229 041 102 960.47 250.02 2020-06-30 06:53:10 205 7171827 0.000015 0.000015 03:53:12 398.02 0.87 111.99 7.39 193.01 70.78 255.96684 107.71 176.33 150.60150 80.92 112.78 131.68930 79.17 105.90 1229 041 105 960.47 247.10 2020-06-30 10:55:02 206 7206983 0.000015 0.000015 03:52:22 396.58 0.87 111.98 7.39 192.55 70.46 255.96888 105.52 172.60 150.67628 81.37 114.22 131.66807 79.54 106.97 1229 041 100 960.47 248.00 2020-06-30 14:54:29 207 7242139 0.000015 0.000015 03:50:18 393.07 0.88 111.47 7.24 191.12 69.10 255.95677 106.31 174.10 150.48214 81.09 112.65 131.86220 79.31 106.12 1229 041 102 960.47 250.21 2020-06-30 18:54:23 208 7277295 0.000015 0.000015 03:50:25 393.25 0.85 111.70 7.38 191.38 68.87 255.85990 105.59 172.79 150.49601 81.48 113.54 131.58462 79.85 107.34 1229 041 101 960.47 250.10 2020-06-30 22:54:31 209 7312451 0.000015 0.000015 03:50:40 393.70 0.85 111.73 7.35 191.59 69.18 255.89797 106.01 172.68 150.62065 81.27 113.44 131.50762 79.58 107.05 1229 041 100 960.47 249.82 2020-07-01 02:53:54 210 7347607 0.000015 0.000015 03:49:55 392.40 0.84 111.65 7.35 191.34 68.33 255.82242 106.32 172.78 150.64223 81.76 114.82 131.40095 79.81 107.81 1229 041 100 960.47 250.64 2020-07-01 06:52:00 211 7382763 0.000015 0.000015 03:48:43 390.36 0.83 111.43 7.32 190.63 67.52 255.79918 105.53 171.46 150.22513 81.78 114.75 131.50691 79.60 107.43 1229 041 100 960.47 251.95 2020-07-01 10:48:47 212 7417919 0.000015 0.000015 03:47:40 388.56 0.81 111.37 7.29 189.94 66.79 255.80021 105.14 169.54 150.74661 81.55 113.91 131.45786 79.48 107.12 1229 041 102 960.47 253.12 2020-07-01 14:45:48 213 7453075 0.000015 0.000015 03:47:46 388.74 0.82 111.24 7.21 190.05 67.21 255.75393 105.00 168.42 150.52045 81.87 114.88 131.59659 80.05 108.29 1229 041 104 960.47 253.00 2020-07-01 18:43:31 214 7488231 0.000015 0.000015 03:48:20 389.70 0.83 111.23 7.23 190.33 67.58 255.74193 105.97 170.24 150.56458 82.05 114.92 131.43926 79.81 107.54 1229 041 102 960.47 252.38 2020-07-01 22:41:10 215 7523387 0.000015 0.000015 03:48:13 389.52 0.86 111.40 7.27 190.44 67.42 255.67924 104.24 167.38 150.34528 82.10 115.49 131.50913 79.87 108.03 1229 041 103 960.47 252.50 2020-07-02 02:40:22 216 7558543 0.000015 0.000015 03:49:12 391.19 0.84 111.56 7.32 191.11 68.13 255.68077 104.94 167.63 150.02326 81.98 115.16 131.48409 79.89 108.03 1229 041 103 960.47 251.42 2020-07-02 06:40:05 217 7593699 0.000015 0.000015 03:49:57 392.45 0.97 111.75 7.44 191.80 68.69 255.66223 103.21 164.61 150.20454 82.49 116.26 131.41589 80.19 108.61 1229 041 104 960.47 250.61 2020-07-02 10:40:39 218 7628855 0.000015 0.000015 03:50:45 393.83 0.87 112.06 7.53 192.35 69.07 255.59491 104.04 165.62 150.15699 82.44 116.10 131.49055 80.47 109.16 1229 041 102 960.47 249.73 2020-07-02 14:41:28 219 7664011 0.000015 0.000015 03:50:59 394.23 0.86 111.88 7.44 192.48 69.63 255.60636 103.20 163.97 150.35325 82.36 115.73 131.46223 80.45 109.16 1229 041 101 960.47 249.48 2020-07-02 18:42:26 220 7699167 0.000015 0.000015 03:51:03 394.33 0.87 112.31 7.61 192.68 69.19 255.56517 102.77 162.56 150.29470 82.99 117.02 131.46274 80.77 109.97 1229 041 103 960.47 249.42

tlikhomanenko commented 4 years ago

@AlexandderGorodetski

Are you using the latest versions of w2l? Please pull because we recently fixed some problems with lr schedule. Then please add to the config --warmup=64000 (we didn't fix the config when we released the code related to warmup). Also add --saug_start_update=64000, --lr_decay_step=40, --lr_decay=200. Check then in the log that lr and lrcrit is changing at the beginning from 0 to 0.03 during 64k updates linearly.

tlikhomanenko commented 4 years ago

I see that you are running on 1 GPU only, is this right? Then lr, warmup possible should be tuned.

xiaosdawn commented 4 years ago

Maybe my problem is the same as @AlexandderGorodetski . I did run the transformer_s2s model on 1 GPU only. So if lr, warmup should be tuned, how should I set them? @tlikhomanenko BTW, I had pulled the lastest w2l, and add to config --warmup=64000 , saug_start_update=64000, --lr_decay_step=40, --lr_decay=200, details as:

--runname=am_transformer_seq2seq_librispeech
--rundir=/root/wav2letter-release-20200729/tutorials/1-librispeech_clean/librispeech_rundir
--archdir=/root/wav2letter-release-20200729/recipes/models/sota/2019
--arch=am_arch/am_transformer_s2s.arch
--tokensdir=/root/wav2letter-release-20200729/recipes/models/sota/2019/model_dst/am
--tokens=librispeech-train-all-unigram-10000.tokens
--lexicon=/root/wav2letter-release-20200729/recipes/models/sota/2019/model_dst/am/librispeech-train+dev-unigram-10000-nbest10.lexicon
--train=/app/lists/train-clean-100.lst,/app/lists/train-clean-360.lst,/app/lists/train-other-500.lst
--valid=dev-clean:/app/lists/dev-clean.lst,dev-other:/app/lists/dev-other.lst
--criterion=transformer
--mfsc
--usewordpiece=true
--wordseparator=_
--am_decoder_tr_dropout=0.1
--am_decoder_tr_layerdrop=0.1
--am_decoder_tr_layers=6
--maxdecoderoutputlen=120
--labelsmooth=0.05
--dataorder=output_spiral
--inputbinsize=25
--attnWindow=softPretrain
--softwstd=4
--trainWithWindow=true
--pretrainWindow=3
--attention=keyvalue
--encoderdim=256
--memstepsize=5000000
--eostoken=true
--pcttraineval=1
--pctteacherforcing=99
--sampletarget=0.01
--netoptim=adagrad
--critoptim=adagrad
--lr=0.03
--lrcrit=0.03
--linseg=0
--momentum=0.0
--maxgradnorm=0.1
--onorm=target
--sqnorm
--nthread=6
--batchsize=8
--filterbanks=80
--enable_distributed
--warmup=64000
--saug_start_update=64000
--lr_decay=200
--lr_decay_step=40
tlikhomanenko commented 4 years ago

you can set specaug to be the same as warmup. I would start with warmup 16k, 32k, 64k, 80k to see if this helps. For the best then tune lr: 0.03, 0.01, 0.005.

xiaosdawn commented 4 years ago

So my config should be fine I think. Maybe tune lr: 0.01, 0.005. Is it the bigger warmup is, the smaller lr is ?

tlikhomanenko commented 4 years ago

this could depend, warmup actually is more to prevent blowing up, so if you see blowing up then just increase warmup

xiaosdawn commented 4 years ago

Got it. Thanks @tlikhomanenko

xiaosdawn commented 4 years ago

Hi, I tune lr to 0.01, then train transformer_s2s, 001_log as:

epoch:        1 | nupdates:            4 | lr: 0.000001 | lrcriterion: 0.000001 | runtime: 00:00:15 | bch(ms): 3780.05 | smp(ms): 77.14 | fwd(ms): 1888.72 | crit-fwd(ms): 629.75 | bwd(ms): 1179.83 | optim(ms): 454.46 | loss:  546.34261 | train-TER:  0.00 | train-WER:  0.00 | dev-clean-loss:  289.16442 | dev-clean-TER: 407.98 | dev-clean-WER: 589.89 | avg-isz: 1550 | avg-tsz: 054 | max-tsz: 057 | hrs:    0.14 | thrpt(sec/sec): 32.82
epoch:        2 | nupdates:         3571 | lr: 0.000558 | lrcriterion: 0.000558 | runtime: 00:21:36 | bch(ms): 363.23 | smp(ms): 0.85 | fwd(ms): 110.99 | crit-fwd(ms): 4.57 | bwd(ms): 184.23 | optim(ms): 39.39 | loss:  305.62201 | train-TER: 194.43 | train-WER: 321.84 | dev-clean-loss:  159.46351 | dev-clean-TER: 133.06 | dev-clean-WER: 236.95 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 279.37
epoch:        3 | nupdates:         7139 | lr: 0.001115 | lrcriterion: 0.001115 | runtime: 00:21:17 | bch(ms): 358.12 | smp(ms): 0.86 | fwd(ms): 110.66 | crit-fwd(ms): 4.40 | bwd(ms): 183.95 | optim(ms): 39.01 | loss:  272.27505 | train-TER: 156.20 | train-WER: 263.89 | dev-clean-loss:  152.90058 | dev-clean-TER: 243.12 | dev-clean-WER: 387.42 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 283.36
epoch:        4 | nupdates:        10707 | lr: 0.001673 | lrcriterion: 0.001673 | runtime: 00:20:30 | bch(ms): 344.92 | smp(ms): 0.90 | fwd(ms): 110.62 | crit-fwd(ms): 4.38 | bwd(ms): 184.00 | optim(ms): 38.94 | loss:  263.07869 | train-TER: 87.84 | train-WER: 126.91 | dev-clean-loss:  148.36372 | dev-clean-TER: 129.10 | dev-clean-WER: 177.17 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 294.20
epoch:        5 | nupdates:        14275 | lr: 0.002230 | lrcriterion: 0.002230 | runtime: 00:20:23 | bch(ms): 342.79 | smp(ms): 0.89 | fwd(ms): 110.67 | crit-fwd(ms): 4.42 | bwd(ms): 184.02 | optim(ms): 39.12 | loss:  254.92954 | train-TER: 76.05 | train-WER: 104.70 | dev-clean-loss:  144.33253 | dev-clean-TER: 95.22 | dev-clean-WER: 136.78 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 296.03
epoch:        6 | nupdates:        17843 | lr: 0.002788 | lrcriterion: 0.002788 | runtime: 00:20:24 | bch(ms): 343.28 | smp(ms): 0.88 | fwd(ms): 110.56 | crit-fwd(ms): 4.38 | bwd(ms): 183.95 | optim(ms): 38.99 | loss:  248.74795 | train-TER: 79.35 | train-WER: 111.32 | dev-clean-loss:  142.41985 | dev-clean-TER: 138.12 | dev-clean-WER: 196.13 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.60
epoch:        7 | nupdates:        21411 | lr: 0.003345 | lrcriterion: 0.003345 | runtime: 00:20:44 | bch(ms): 348.86 | smp(ms): 0.89 | fwd(ms): 111.05 | crit-fwd(ms): 4.69 | bwd(ms): 184.57 | optim(ms): 40.18 | loss:  244.07397 | train-TER: 90.25 | train-WER: 132.60 | dev-clean-loss:  139.96105 | dev-clean-TER: 122.26 | dev-clean-WER: 179.79 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 290.88

the config details: image

Then I tune lr to 0.005 there are still problems. Would you guide me to modify the config to ensure normal transformer_s2s training on 1 GPU. sota/2019/librispeech BTW, I have pulled the lastest w2l. THX.

tlikhomanenko commented 4 years ago

You need to wait more time to be sure there is no convergence. Your loss looks good, it goes down, blowing up I meant after warmup stage.

xiaosdawn commented 4 years ago

I notice that the lr is increasing. I guess it is warmup stage which is doing linear increase of lr from 0 to value specific in --lr. If I'm right, I'll wait more time. THX.

tlikhomanenko commented 4 years ago

it is increasing during warmup, it is ok.

xiaosdawn commented 4 years ago

Thanks, I'll wait more time.

xiaosdawn commented 4 years ago

Hi, @tlikhomanenko , the transformer_s2s model has been training for a fewer days. here is 001_log

epoch:        1 | nupdates:            4 | lr: 0.000001 | lrcriterion: 0.000001 | runtime: 00:00:14 | bch(ms): 3635.95 | smp(ms): 89.80 | fwd(ms): 1774.30 | crit-fwd(ms): 596.94 | bwd(ms): 1113.14 | optim(ms): 489.57 | loss:  542.55881 | train-TER:  0.00 | train-WER:  0.00 | dev-clean-loss:  289.05373 | dev-clean-TER: 407.98 | dev-clean-WER: 589.89 | avg-isz: 1550 | avg-tsz: 054 | max-tsz: 057 | hrs:    0.14 | thrpt(sec/sec): 34.12
epoch:        2 | nupdates:         3571 | lr: 0.000558 | lrcriterion: 0.000558 | runtime: 00:22:08 | bch(ms): 372.33 | smp(ms): 1.05 | fwd(ms): 111.88 | crit-fwd(ms): 5.04 | bwd(ms): 186.09 | optim(ms): 42.57 | loss:  305.57316 | train-TER: 197.39 | train-WER: 327.84 | dev-clean-loss:  159.49892 | dev-clean-TER: 149.98 | dev-clean-WER: 268.14 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 272.54
epoch:        3 | nupdates:         7139 | lr: 0.001115 | lrcriterion: 0.001115 | runtime: 00:21:33 | bch(ms): 362.45 | smp(ms): 1.01 | fwd(ms): 111.43 | crit-fwd(ms): 4.81 | bwd(ms): 185.54 | optim(ms): 41.64 | loss:  272.25188 | train-TER: 142.33 | train-WER: 239.09 | dev-clean-loss:  152.96557 | dev-clean-TER: 214.49 | dev-clean-WER: 340.05 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 279.97
epoch:        4 | nupdates:        10707 | lr: 0.001673 | lrcriterion: 0.001673 | runtime: 00:20:31 | bch(ms): 345.26 | smp(ms): 0.86 | fwd(ms): 110.38 | crit-fwd(ms): 4.31 | bwd(ms): 183.61 | optim(ms): 38.82 | loss:  263.06951 | train-TER: 99.23 | train-WER: 144.80 | dev-clean-loss:  148.12723 | dev-clean-TER: 114.66 | dev-clean-WER: 164.19 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 293.91
epoch:        5 | nupdates:        14275 | lr: 0.002230 | lrcriterion: 0.002230 | runtime: 00:20:20 | bch(ms): 342.05 | smp(ms): 0.84 | fwd(ms): 110.41 | crit-fwd(ms): 4.32 | bwd(ms): 183.65 | optim(ms): 38.82 | loss:  254.89768 | train-TER: 77.55 | train-WER: 107.56 | dev-clean-loss:  144.46799 | dev-clean-TER: 108.96 | dev-clean-WER: 158.93 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 296.67
epoch:        6 | nupdates:        17843 | lr: 0.002788 | lrcriterion: 0.002788 | runtime: 00:20:24 | bch(ms): 343.30 | smp(ms): 0.82 | fwd(ms): 110.51 | crit-fwd(ms): 4.34 | bwd(ms): 183.77 | optim(ms): 38.93 | loss:  248.69334 | train-TER: 81.75 | train-WER: 115.31 | dev-clean-loss:  142.44636 | dev-clean-TER: 144.36 | dev-clean-WER: 205.02 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.58
epoch:        7 | nupdates:        21411 | lr: 0.003345 | lrcriterion: 0.003345 | runtime: 00:20:27 | bch(ms): 344.13 | smp(ms): 0.83 | fwd(ms): 110.00 | crit-fwd(ms): 4.32 | bwd(ms): 183.08 | optim(ms): 38.83 | loss:  244.09262 | train-TER: 90.48 | train-WER: 137.07 | dev-clean-loss:  140.10024 | dev-clean-TER: 126.72 | dev-clean-WER: 192.76 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.87
epoch:        8 | nupdates:        24979 | lr: 0.003903 | lrcriterion: 0.003903 | runtime: 00:20:22 | bch(ms): 342.53 | smp(ms): 0.84 | fwd(ms): 109.97 | crit-fwd(ms): 4.31 | bwd(ms): 182.98 | optim(ms): 38.79 | loss:  240.41240 | train-TER: 85.17 | train-WER: 122.27 | dev-clean-loss:  138.17301 | dev-clean-TER: 108.58 | dev-clean-WER: 158.15 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.25
epoch:        9 | nupdates:        28547 | lr: 0.004460 | lrcriterion: 0.004460 | runtime: 00:20:31 | bch(ms): 345.14 | smp(ms): 0.78 | fwd(ms): 109.98 | crit-fwd(ms): 4.35 | bwd(ms): 183.02 | optim(ms): 38.94 | loss:  237.26765 | train-TER: 95.93 | train-WER: 145.54 | dev-clean-loss:  136.75084 | dev-clean-TER: 119.24 | dev-clean-WER: 183.63 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.01
epoch:       10 | nupdates:        32115 | lr: 0.005018 | lrcriterion: 0.005018 | runtime: 00:20:30 | bch(ms): 344.83 | smp(ms): 0.79 | fwd(ms): 109.96 | crit-fwd(ms): 4.32 | bwd(ms): 182.99 | optim(ms): 38.84 | loss:  234.49187 | train-TER: 92.59 | train-WER: 142.13 | dev-clean-loss:  135.94455 | dev-clean-TER: 151.87 | dev-clean-WER: 228.50 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 294.28
epoch:       11 | nupdates:        35683 | lr: 0.005575 | lrcriterion: 0.005575 | runtime: 00:20:36 | bch(ms): 346.50 | smp(ms): 0.80 | fwd(ms): 109.92 | crit-fwd(ms): 4.31 | bwd(ms): 182.86 | optim(ms): 38.78 | loss:  232.13157 | train-TER: 105.66 | train-WER: 162.18 | dev-clean-loss:  135.33027 | dev-clean-TER: 392.21 | dev-clean-WER: 579.16 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 292.86
epoch:       12 | nupdates:        39251 | lr: 0.006133 | lrcriterion: 0.006133 | runtime: 00:20:52 | bch(ms): 351.13 | smp(ms): 0.84 | fwd(ms): 109.80 | crit-fwd(ms): 4.30 | bwd(ms): 182.69 | optim(ms): 38.75 | loss:  229.54482 | train-TER: 140.64 | train-WER: 220.80 | dev-clean-loss:  135.00185 | dev-clean-TER: 130.21 | dev-clean-WER: 203.52 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 289.00
epoch:       13 | nupdates:        42819 | lr: 0.006690 | lrcriterion: 0.006690 | runtime: 00:20:47 | bch(ms): 349.66 | smp(ms): 0.81 | fwd(ms): 109.99 | crit-fwd(ms): 4.45 | bwd(ms): 182.94 | optim(ms): 39.22 | loss:  227.18786 | train-TER: 115.10 | train-WER: 183.34 | dev-clean-loss:  134.34021 | dev-clean-TER: 151.42 | dev-clean-WER: 233.80 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 290.21
epoch:       14 | nupdates:        46387 | lr: 0.007248 | lrcriterion: 0.007248 | runtime: 00:20:50 | bch(ms): 350.35 | smp(ms): 0.75 | fwd(ms): 110.01 | crit-fwd(ms): 4.47 | bwd(ms): 182.96 | optim(ms): 39.27 | loss:  225.08032 | train-TER: 123.99 | train-WER: 194.85 | dev-clean-loss:  133.59888 | dev-clean-TER: 108.90 | dev-clean-WER: 181.81 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 289.64
epoch:       15 | nupdates:        49955 | lr: 0.007805 | lrcriterion: 0.007805 | runtime: 00:20:35 | bch(ms): 346.33 | smp(ms): 0.84 | fwd(ms): 109.87 | crit-fwd(ms): 4.33 | bwd(ms): 182.79 | optim(ms): 38.83 | loss:  222.56216 | train-TER: 103.89 | train-WER: 160.51 | dev-clean-loss:  133.34383 | dev-clean-TER: 139.95 | dev-clean-WER: 211.29 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.00
epoch:       16 | nupdates:        53523 | lr: 0.008363 | lrcriterion: 0.008363 | runtime: 00:20:44 | bch(ms): 348.92 | smp(ms): 0.81 | fwd(ms): 109.72 | crit-fwd(ms): 4.31 | bwd(ms): 182.63 | optim(ms): 38.77 | loss:  220.42952 | train-TER: 123.25 | train-WER: 193.46 | dev-clean-loss:  133.46815 | dev-clean-TER: 165.69 | dev-clean-WER: 254.93 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 290.82
epoch:       17 | nupdates:        57091 | lr: 0.008920 | lrcriterion: 0.008920 | runtime: 00:20:37 | bch(ms): 346.83 | smp(ms): 0.80 | fwd(ms): 109.71 | crit-fwd(ms): 4.31 | bwd(ms): 182.67 | optim(ms): 38.78 | loss:  218.33361 | train-TER: 110.72 | train-WER: 169.60 | dev-clean-loss:  133.23524 | dev-clean-TER: 128.21 | dev-clean-WER: 184.87 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 292.57
epoch:       18 | nupdates:        60659 | lr: 0.009478 | lrcriterion: 0.009478 | runtime: 00:20:35 | bch(ms): 346.30 | smp(ms): 0.87 | fwd(ms): 109.82 | crit-fwd(ms): 4.38 | bwd(ms): 182.72 | optim(ms): 38.99 | loss:  216.21758 | train-TER: 98.61 | train-WER: 152.91 | dev-clean-loss:  132.90344 | dev-clean-TER: 122.01 | dev-clean-WER: 191.58 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.02
epoch:       19 | nupdates:        64227 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:28 | bch(ms): 344.40 | smp(ms): 0.81 | fwd(ms): 109.77 | crit-fwd(ms): 4.31 | bwd(ms): 182.61 | optim(ms): 38.78 | loss:  214.13100 | train-TER: 91.13 | train-WER: 137.95 | dev-clean-loss:  132.81396 | dev-clean-TER: 135.43 | dev-clean-WER: 206.39 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 294.64
epoch:       20 | nupdates:        67795 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:31 | bch(ms): 345.09 | smp(ms): 0.81 | fwd(ms): 109.86 | crit-fwd(ms): 4.32 | bwd(ms): 182.71 | optim(ms): 38.84 | loss:  212.05076 | train-TER: 97.52 | train-WER: 147.36 | dev-clean-loss:  133.04026 | dev-clean-TER: 148.82 | dev-clean-WER: 239.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 294.06
epoch:       21 | nupdates:        71363 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:28 | bch(ms): 344.34 | smp(ms): 0.83 | fwd(ms): 109.93 | crit-fwd(ms): 4.37 | bwd(ms): 182.79 | optim(ms): 39.00 | loss:  209.71874 | train-TER: 89.92 | train-WER: 132.64 | dev-clean-loss:  133.18914 | dev-clean-TER: 164.78 | dev-clean-WER: 228.53 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.70
epoch:       22 | nupdates:        74931 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:29 | bch(ms): 344.65 | smp(ms): 0.83 | fwd(ms): 109.84 | crit-fwd(ms): 4.33 | bwd(ms): 182.65 | optim(ms): 38.84 | loss:  207.52258 | train-TER: 90.78 | train-WER: 139.91 | dev-clean-loss:  133.38231 | dev-clean-TER: 124.20 | dev-clean-WER: 188.11 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 294.43
epoch:       23 | nupdates:        78499 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:33 | bch(ms): 345.79 | smp(ms): 0.82 | fwd(ms): 109.80 | crit-fwd(ms): 4.30 | bwd(ms): 182.61 | optim(ms): 38.74 | loss:  205.48626 | train-TER: 102.19 | train-WER: 154.95 | dev-clean-loss:  134.06632 | dev-clean-TER: 407.28 | dev-clean-WER: 582.99 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.46
epoch:       24 | nupdates:        82067 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:27 | bch(ms): 344.11 | smp(ms): 0.85 | fwd(ms): 109.73 | crit-fwd(ms): 4.28 | bwd(ms): 182.53 | optim(ms): 38.73 | loss:  203.55546 | train-TER: 93.75 | train-WER: 139.76 | dev-clean-loss:  134.34406 | dev-clean-TER: 137.70 | dev-clean-WER: 200.38 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.89
epoch:       25 | nupdates:        85635 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:24 | bch(ms): 343.32 | smp(ms): 0.85 | fwd(ms): 109.71 | crit-fwd(ms): 4.27 | bwd(ms): 182.49 | optim(ms): 38.70 | loss:  201.54203 | train-TER: 90.66 | train-WER: 134.88 | dev-clean-loss:  134.69770 | dev-clean-TER: 131.58 | dev-clean-WER: 205.26 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.57
epoch:       26 | nupdates:        89203 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:27 | bch(ms): 344.07 | smp(ms): 0.80 | fwd(ms): 109.68 | crit-fwd(ms): 4.27 | bwd(ms): 182.42 | optim(ms): 38.68 | loss:  199.63130 | train-TER: 95.25 | train-WER: 147.17 | dev-clean-loss:  135.44977 | dev-clean-TER: 142.70 | dev-clean-WER: 224.19 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.93
epoch:       27 | nupdates:        92771 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:31 | bch(ms): 345.03 | smp(ms): 0.73 | fwd(ms): 109.73 | crit-fwd(ms): 4.29 | bwd(ms): 182.52 | optim(ms): 38.77 | loss:  197.79597 | train-TER: 99.80 | train-WER: 152.56 | dev-clean-loss:  135.94322 | dev-clean-TER: 318.84 | dev-clean-WER: 583.59 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.11
epoch:       28 | nupdates:        96339 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:31 | bch(ms): 345.05 | smp(ms): 0.81 | fwd(ms): 109.69 | crit-fwd(ms): 4.28 | bwd(ms): 182.51 | optim(ms): 38.72 | loss:  196.17866 | train-TER: 93.98 | train-WER: 152.50 | dev-clean-loss:  135.97586 | dev-clean-TER: 141.18 | dev-clean-WER: 200.02 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 294.09
epoch:       29 | nupdates:        99907 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:26 | bch(ms): 343.79 | smp(ms): 0.80 | fwd(ms): 109.69 | crit-fwd(ms): 4.27 | bwd(ms): 182.44 | optim(ms): 38.67 | loss:  194.58659 | train-TER: 93.35 | train-WER: 140.54 | dev-clean-loss:  136.92299 | dev-clean-TER: 93.92 | dev-clean-WER: 122.77 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.17
epoch:       30 | nupdates:       103475 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:37 | bch(ms): 346.84 | smp(ms): 0.88 | fwd(ms): 110.04 | crit-fwd(ms): 4.53 | bwd(ms): 182.90 | optim(ms): 39.44 | loss:  192.74155 | train-TER: 99.43 | train-WER: 148.19 | dev-clean-loss:  137.50296 | dev-clean-TER: 160.72 | dev-clean-WER: 206.65 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 292.57
epoch:       31 | nupdates:       107043 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:32 | bch(ms): 345.38 | smp(ms): 0.83 | fwd(ms): 109.69 | crit-fwd(ms): 4.28 | bwd(ms): 182.52 | optim(ms): 38.73 | loss:  191.22567 | train-TER: 93.96 | train-WER: 155.17 | dev-clean-loss:  137.95059 | dev-clean-TER: 138.42 | dev-clean-WER: 216.83 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.81
epoch:       32 | nupdates:       110611 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:34 | bch(ms): 346.07 | smp(ms): 0.72 | fwd(ms): 109.68 | crit-fwd(ms): 4.27 | bwd(ms): 182.44 | optim(ms): 38.66 | loss:  189.74221 | train-TER: 103.30 | train-WER: 168.49 | dev-clean-loss:  138.76265 | dev-clean-TER: 100.36 | dev-clean-WER: 143.39 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.22
epoch:       33 | nupdates:       114179 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:26 | bch(ms): 343.67 | smp(ms): 0.76 | fwd(ms): 109.73 | crit-fwd(ms): 4.27 | bwd(ms): 182.56 | optim(ms): 38.68 | loss:  188.43351 | train-TER: 91.47 | train-WER: 139.59 | dev-clean-loss:  139.19705 | dev-clean-TER: 101.32 | dev-clean-WER: 151.93 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 097 | hrs:  100.57 | thrpt(sec/sec): 295.27
epoch:       34 | nupdates:       117747 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:33 | bch(ms): 345.59 | smp(ms): 0.81 | fwd(ms): 109.70 | crit-fwd(ms): 4.27 | bwd(ms): 182.48 | optim(ms): 38.66 | loss:  186.98903 | train-TER: 104.38 | train-WER: 162.80 | dev-clean-loss:  139.65809 | dev-clean-TER: 148.84 | dev-clean-WER: 270.25 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 293.63
epoch:       35 | nupdates:       121315 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:31 | bch(ms): 345.13 | smp(ms): 0.88 | fwd(ms): 109.68 | crit-fwd(ms): 4.26 | bwd(ms): 182.18 | optim(ms): 38.55 | loss:  185.66078 | train-TER: 98.91 | train-WER: 160.05 | dev-clean-loss:  140.45970 | dev-clean-TER: 118.42 | dev-clean-WER: 176.45 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.02
epoch:       36 | nupdates:       124883 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:13 | bch(ms): 340.23 | smp(ms): 0.83 | fwd(ms): 109.69 | crit-fwd(ms): 4.25 | bwd(ms): 182.03 | optim(ms): 38.48 | loss:  184.23861 | train-TER: 81.65 | train-WER: 117.33 | dev-clean-loss:  140.62293 | dev-clean-TER: 124.72 | dev-clean-WER: 193.53 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 298.25
epoch:       37 | nupdates:       128451 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.45 | smp(ms): 0.79 | fwd(ms): 109.65 | crit-fwd(ms): 4.24 | bwd(ms): 181.97 | optim(ms): 38.40 | loss:  182.81692 | train-TER: 94.03 | train-WER: 149.29 | dev-clean-loss:  141.34250 | dev-clean-TER: 134.28 | dev-clean-WER: 201.33 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.46
epoch:       38 | nupdates:       132019 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:29 | bch(ms): 344.66 | smp(ms): 0.82 | fwd(ms): 109.65 | crit-fwd(ms): 4.24 | bwd(ms): 181.96 | optim(ms): 38.43 | loss:  181.53103 | train-TER: 99.98 | train-WER: 156.23 | dev-clean-loss:  141.65211 | dev-clean-TER: 169.19 | dev-clean-WER: 229.41 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.42
epoch:       39 | nupdates:       135587 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 342.80 | smp(ms): 0.84 | fwd(ms): 109.68 | crit-fwd(ms): 4.26 | bwd(ms): 182.08 | optim(ms): 38.53 | loss:  180.59272 | train-TER: 91.33 | train-WER: 135.95 | dev-clean-loss:  142.55995 | dev-clean-TER: 162.90 | dev-clean-WER: 228.41 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.02
epoch:       40 | nupdates:       139155 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 342.78 | smp(ms): 0.85 | fwd(ms): 109.72 | crit-fwd(ms): 4.25 | bwd(ms): 182.08 | optim(ms): 38.48 | loss:  179.30319 | train-TER: 90.71 | train-WER: 135.92 | dev-clean-loss:  142.82189 | dev-clean-TER: 154.45 | dev-clean-WER: 237.88 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.04
epoch:       41 | nupdates:       142723 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:19 | bch(ms): 341.85 | smp(ms): 0.81 | fwd(ms): 109.68 | crit-fwd(ms): 4.26 | bwd(ms): 182.06 | optim(ms): 38.53 | loss:  178.41700 | train-TER: 92.70 | train-WER: 130.82 | dev-clean-loss:  143.78451 | dev-clean-TER: 186.03 | dev-clean-WER: 264.07 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.84
epoch:       42 | nupdates:       146291 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:18 | bch(ms): 341.45 | smp(ms): 0.76 | fwd(ms): 109.73 | crit-fwd(ms): 4.25 | bwd(ms): 182.07 | optim(ms): 38.45 | loss:  177.33186 | train-TER: 89.57 | train-WER: 127.78 | dev-clean-loss:  143.84129 | dev-clean-TER: 126.46 | dev-clean-WER: 173.77 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.19
epoch:       43 | nupdates:       149859 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:36 | bch(ms): 346.49 | smp(ms): 0.77 | fwd(ms): 110.35 | crit-fwd(ms): 4.71 | bwd(ms): 182.84 | optim(ms): 39.82 | loss:  176.07572 | train-TER: 93.46 | train-WER: 133.80 | dev-clean-loss:  144.47340 | dev-clean-TER: 330.77 | dev-clean-WER: 588.24 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 292.87
epoch:       44 | nupdates:       153427 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 342.81 | smp(ms): 0.82 | fwd(ms): 109.66 | crit-fwd(ms): 4.25 | bwd(ms): 182.02 | optim(ms): 38.48 | loss:  175.00870 | train-TER: 91.66 | train-WER: 133.83 | dev-clean-loss:  145.05405 | dev-clean-TER: 93.74 | dev-clean-WER: 131.50 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.01
epoch:       45 | nupdates:       156995 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:19 | bch(ms): 341.71 | smp(ms): 0.81 | fwd(ms): 109.63 | crit-fwd(ms): 4.24 | bwd(ms): 181.89 | optim(ms): 38.43 | loss:  174.08165 | train-TER: 88.27 | train-WER: 125.71 | dev-clean-loss:  145.73759 | dev-clean-TER: 182.36 | dev-clean-WER: 251.29 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 296.96
epoch:       46 | nupdates:       160563 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 342.89 | smp(ms): 0.78 | fwd(ms): 109.70 | crit-fwd(ms): 4.27 | bwd(ms): 182.26 | optim(ms): 38.61 | loss:  173.14638 | train-TER: 91.68 | train-WER: 131.39 | dev-clean-loss:  146.29957 | dev-clean-TER: 129.11 | dev-clean-WER: 189.77 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.94
epoch:       47 | nupdates:       164131 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:27 | bch(ms): 343.91 | smp(ms): 0.75 | fwd(ms): 109.71 | crit-fwd(ms): 4.26 | bwd(ms): 182.29 | optim(ms): 38.61 | loss:  172.45187 | train-TER: 99.33 | train-WER: 148.46 | dev-clean-loss:  146.18781 | dev-clean-TER: 131.71 | dev-clean-WER: 206.84 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.06
epoch:       48 | nupdates:       167699 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:21 | bch(ms): 342.38 | smp(ms): 0.82 | fwd(ms): 109.66 | crit-fwd(ms): 4.23 | bwd(ms): 181.94 | optim(ms): 38.40 | loss:  171.23553 | train-TER: 97.03 | train-WER: 143.21 | dev-clean-loss:  147.33217 | dev-clean-TER: 171.39 | dev-clean-WER: 257.52 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.38
epoch:       49 | nupdates:       171267 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:20 | bch(ms): 342.18 | smp(ms): 0.75 | fwd(ms): 109.71 | crit-fwd(ms): 4.25 | bwd(ms): 182.11 | optim(ms): 38.48 | loss:  170.30662 | train-TER: 86.45 | train-WER: 126.11 | dev-clean-loss:  147.64729 | dev-clean-TER: 154.95 | dev-clean-WER: 214.33 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.55
epoch:       50 | nupdates:       174835 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:21 | bch(ms): 342.45 | smp(ms): 0.84 | fwd(ms): 109.77 | crit-fwd(ms): 4.26 | bwd(ms): 182.25 | optim(ms): 38.54 | loss:  169.80076 | train-TER: 86.21 | train-WER: 123.07 | dev-clean-loss:  147.87325 | dev-clean-TER: 95.43 | dev-clean-WER: 129.41 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 097 | hrs:  100.57 | thrpt(sec/sec): 296.32
epoch:       51 | nupdates:       178403 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 342.88 | smp(ms): 0.83 | fwd(ms): 109.70 | crit-fwd(ms): 4.27 | bwd(ms): 182.27 | optim(ms): 38.38 | loss:  168.84181 | train-TER: 91.85 | train-WER: 133.52 | dev-clean-loss:  148.07291 | dev-clean-TER: 146.02 | dev-clean-WER: 182.75 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 100 | hrs:  100.57 | thrpt(sec/sec): 295.95
epoch:       52 | nupdates:       181971 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:22 | bch(ms): 342.58 | smp(ms): 0.81 | fwd(ms): 109.66 | crit-fwd(ms): 4.25 | bwd(ms): 182.02 | optim(ms): 38.26 | loss:  167.69585 | train-TER: 95.13 | train-WER: 136.58 | dev-clean-loss:  148.21641 | dev-clean-TER: 155.95 | dev-clean-WER: 223.45 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.20
epoch:       53 | nupdates:       185539 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:17 | bch(ms): 341.26 | smp(ms): 0.76 | fwd(ms): 109.60 | crit-fwd(ms): 4.24 | bwd(ms): 181.84 | optim(ms): 38.19 | loss:  167.34793 | train-TER: 87.41 | train-WER: 125.72 | dev-clean-loss:  148.97657 | dev-clean-TER: 133.15 | dev-clean-WER: 194.84 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.36
epoch:       54 | nupdates:       189107 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:32 | bch(ms): 345.38 | smp(ms): 0.83 | fwd(ms): 110.35 | crit-fwd(ms): 4.79 | bwd(ms): 182.75 | optim(ms): 39.69 | loss:  166.27398 | train-TER: 84.72 | train-WER: 120.43 | dev-clean-loss:  149.48425 | dev-clean-TER: 85.14 | dev-clean-WER: 111.03 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.81
epoch:       55 | nupdates:       192675 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:20 | bch(ms): 341.96 | smp(ms): 0.77 | fwd(ms): 109.79 | crit-fwd(ms): 4.27 | bwd(ms): 182.22 | optim(ms): 38.53 | loss:  165.58569 | train-TER: 90.05 | train-WER: 127.27 | dev-clean-loss:  149.90463 | dev-clean-TER: 136.16 | dev-clean-WER: 193.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.74
epoch:       56 | nupdates:       196243 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.47 | smp(ms): 0.84 | fwd(ms): 109.90 | crit-fwd(ms): 4.29 | bwd(ms): 182.54 | optim(ms): 38.66 | loss:  164.67441 | train-TER: 90.20 | train-WER: 131.30 | dev-clean-loss:  150.56553 | dev-clean-TER: 151.08 | dev-clean-WER: 204.83 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 295.44
epoch:       57 | nupdates:       199811 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:21 | bch(ms): 342.28 | smp(ms): 0.83 | fwd(ms): 109.91 | crit-fwd(ms): 4.29 | bwd(ms): 182.55 | optim(ms): 38.65 | loss:  164.19569 | train-TER: 87.42 | train-WER: 123.21 | dev-clean-loss:  150.97664 | dev-clean-TER: 171.93 | dev-clean-WER: 242.06 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 296.47
epoch:       58 | nupdates:       203379 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:20 | bch(ms): 342.19 | smp(ms): 0.80 | fwd(ms): 109.92 | crit-fwd(ms): 4.29 | bwd(ms): 182.57 | optim(ms): 38.66 | loss:  163.44549 | train-TER: 84.60 | train-WER: 119.74 | dev-clean-loss:  151.46788 | dev-clean-TER: 134.81 | dev-clean-WER: 221.41 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.55
epoch:       59 | nupdates:       206947 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:19 | bch(ms): 341.65 | smp(ms): 0.82 | fwd(ms): 109.87 | crit-fwd(ms): 4.28 | bwd(ms): 182.45 | optim(ms): 38.63 | loss:  162.64097 | train-TER: 84.92 | train-WER: 122.45 | dev-clean-loss:  151.30337 | dev-clean-TER: 121.72 | dev-clean-WER: 188.95 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 297.01
epoch:       60 | nupdates:       210515 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:20 | bch(ms): 342.14 | smp(ms): 0.70 | fwd(ms): 109.88 | crit-fwd(ms): 4.28 | bwd(ms): 182.52 | optim(ms): 38.63 | loss:  162.30657 | train-TER: 87.57 | train-WER: 125.70 | dev-clean-loss:  152.27026 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.59
epoch:       61 | nupdates:       214083 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:15 | bch(ms): 340.69 | smp(ms): 0.84 | fwd(ms): 109.98 | crit-fwd(ms): 4.30 | bwd(ms): 182.68 | optim(ms): 38.71 | loss:  161.44300 | train-TER: 81.71 | train-WER: 112.01 | dev-clean-loss:  152.11278 | dev-clean-TER: 84.93 | dev-clean-WER: 117.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 297.85
epoch:       62 | nupdates:       217651 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.58 | smp(ms): 0.86 | fwd(ms): 109.92 | crit-fwd(ms): 4.28 | bwd(ms): 182.51 | optim(ms): 38.60 | loss:  161.07355 | train-TER: 90.04 | train-WER: 130.34 | dev-clean-loss:  152.54457 | dev-clean-TER: 139.33 | dev-clean-WER: 201.45 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 295.34
epoch:       63 | nupdates:       221219 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:24 | bch(ms): 343.16 | smp(ms): 0.83 | fwd(ms): 109.93 | crit-fwd(ms): 4.28 | bwd(ms): 182.52 | optim(ms): 38.60 | loss:  160.36102 | train-TER: 88.38 | train-WER: 128.48 | dev-clean-loss:  152.95352 | dev-clean-TER: 188.04 | dev-clean-WER: 240.31 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 295.71
epoch:       64 | nupdates:       224787 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 342.96 | smp(ms): 0.77 | fwd(ms): 109.95 | crit-fwd(ms): 4.30 | bwd(ms): 182.63 | optim(ms): 38.69 | loss:  159.57476 | train-TER: 88.17 | train-WER: 122.60 | dev-clean-loss:  153.11353 | dev-clean-TER: 142.04 | dev-clean-WER: 200.31 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 295.88
epoch:       65 | nupdates:       228355 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:12 | bch(ms): 339.91 | smp(ms): 0.71 | fwd(ms): 109.98 | crit-fwd(ms): 4.29 | bwd(ms): 182.58 | optim(ms): 38.66 | loss:  158.95271 | train-TER: 91.13 | train-WER: 119.80 | dev-clean-loss:  154.09927 | dev-clean-TER: 123.74 | dev-clean-WER: 187.67 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.53
epoch:       66 | nupdates:       231923 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:22 | bch(ms): 342.68 | smp(ms): 0.87 | fwd(ms): 109.95 | crit-fwd(ms): 4.30 | bwd(ms): 182.61 | optim(ms): 38.69 | loss:  158.84053 | train-TER: 90.31 | train-WER: 127.58 | dev-clean-loss:  154.59831 | dev-clean-TER: 160.20 | dev-clean-WER: 225.71 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 296.12
epoch:       67 | nupdates:       235491 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:19 | bch(ms): 341.79 | smp(ms): 0.76 | fwd(ms): 109.95 | crit-fwd(ms): 4.28 | bwd(ms): 182.55 | optim(ms): 38.61 | loss:  158.00905 | train-TER: 92.93 | train-WER: 123.97 | dev-clean-loss:  155.27557 | dev-clean-TER: 132.07 | dev-clean-WER: 162.77 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.89
epoch:       68 | nupdates:       239059 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:22 | bch(ms): 342.58 | smp(ms): 0.83 | fwd(ms): 109.99 | crit-fwd(ms): 4.29 | bwd(ms): 182.61 | optim(ms): 38.61 | loss:  157.37001 | train-TER: 87.83 | train-WER: 122.27 | dev-clean-loss:  155.13058 | dev-clean-TER: 103.13 | dev-clean-WER: 140.57 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.21
epoch:       69 | nupdates:       242627 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:14 | bch(ms): 340.36 | smp(ms): 0.78 | fwd(ms): 109.95 | crit-fwd(ms): 4.28 | bwd(ms): 182.59 | optim(ms): 38.63 | loss:  156.55204 | train-TER: 88.47 | train-WER: 116.73 | dev-clean-loss:  154.99159 | dev-clean-TER: 151.68 | dev-clean-WER: 207.95 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 298.14
epoch:       70 | nupdates:       246195 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:14 | bch(ms): 340.51 | smp(ms): 0.85 | fwd(ms): 109.87 | crit-fwd(ms): 4.28 | bwd(ms): 182.48 | optim(ms): 38.64 | loss:  156.86179 | train-TER: 86.60 | train-WER: 117.11 | dev-clean-loss:  154.94811 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.01
epoch:       71 | nupdates:       249763 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:18 | bch(ms): 341.61 | smp(ms): 0.83 | fwd(ms): 109.91 | crit-fwd(ms): 4.28 | bwd(ms): 182.54 | optim(ms): 38.65 | loss:  155.55935 | train-TER: 86.86 | train-WER: 119.78 | dev-clean-loss:  155.16727 | dev-clean-TER: 83.24 | dev-clean-WER: 100.38 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 297.05
epoch:       72 | nupdates:       253331 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:17 | bch(ms): 341.27 | smp(ms): 0.82 | fwd(ms): 109.90 | crit-fwd(ms): 4.30 | bwd(ms): 182.66 | optim(ms): 38.71 | loss:  155.33844 | train-TER: 88.20 | train-WER: 121.28 | dev-clean-loss:  156.18725 | dev-clean-TER: 83.24 | dev-clean-WER: 100.38 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 297.34
epoch:       73 | nupdates:       256899 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:12 | bch(ms): 339.94 | smp(ms): 0.82 | fwd(ms): 110.11 | crit-fwd(ms): 4.40 | bwd(ms): 182.81 | optim(ms): 39.00 | loss:  153.76100 | train-TER: 82.34 | train-WER: 107.20 | dev-clean-loss:  156.19769 | dev-clean-TER: 164.28 | dev-clean-WER: 224.24 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.51
epoch:       74 | nupdates:       260467 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:07 | bch(ms): 338.33 | smp(ms): 0.79 | fwd(ms): 109.93 | crit-fwd(ms): 4.30 | bwd(ms): 182.59 | optim(ms): 38.73 | loss:  153.45101 | train-TER: 81.52 | train-WER: 104.27 | dev-clean-loss:  157.56766 | dev-clean-TER: 83.24 | dev-clean-WER: 100.38 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 299.93
epoch:       75 | nupdates:       264035 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:05 | bch(ms): 337.81 | smp(ms): 0.80 | fwd(ms): 109.94 | crit-fwd(ms): 4.30 | bwd(ms): 182.58 | optim(ms): 38.68 | loss:  153.40627 | train-TER: 82.36 | train-WER: 103.31 | dev-clean-loss:  157.60326 | dev-clean-TER: 83.24 | dev-clean-WER: 100.38 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 300.39
epoch:       76 | nupdates:       267603 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.56 | smp(ms): 0.80 | fwd(ms): 109.95 | crit-fwd(ms): 4.31 | bwd(ms): 182.56 | optim(ms): 38.69 | loss:  153.55262 | train-TER: 84.42 | train-WER: 94.71 | dev-clean-loss:  157.91365 | dev-clean-TER: 81.74 | dev-clean-WER: 96.48 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 303.31
epoch:       77 | nupdates:       271171 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.88 | smp(ms): 0.83 | fwd(ms): 109.94 | crit-fwd(ms): 4.29 | bwd(ms): 182.57 | optim(ms): 38.66 | loss:  153.19554 | train-TER: 83.42 | train-WER: 96.89 | dev-clean-loss:  157.56810 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.12
epoch:       78 | nupdates:       274739 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:59 | bch(ms): 336.08 | smp(ms): 0.88 | fwd(ms): 109.93 | crit-fwd(ms): 4.30 | bwd(ms): 182.57 | optim(ms): 38.69 | loss:  153.49981 | train-TER: 82.33 | train-WER: 97.79 | dev-clean-loss:  157.85903 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 301.94
epoch:       79 | nupdates:       278307 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:03 | bch(ms): 337.36 | smp(ms): 0.85 | fwd(ms): 109.90 | crit-fwd(ms): 4.30 | bwd(ms): 182.55 | optim(ms): 38.70 | loss:  152.96562 | train-TER: 84.68 | train-WER: 104.58 | dev-clean-loss:  157.93562 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 300.79
epoch:       80 | nupdates:       281875 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:03 | bch(ms): 337.41 | smp(ms): 0.81 | fwd(ms): 109.92 | crit-fwd(ms): 4.30 | bwd(ms): 182.55 | optim(ms): 38.68 | loss:  152.47149 | train-TER: 80.86 | train-WER: 99.10 | dev-clean-loss:  158.49520 | dev-clean-TER: 128.12 | dev-clean-WER: 184.83 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 300.74
epoch:       81 | nupdates:       285443 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:03 | bch(ms): 337.36 | smp(ms): 0.85 | fwd(ms): 110.13 | crit-fwd(ms): 4.45 | bwd(ms): 182.86 | optim(ms): 39.24 | loss:  152.96693 | train-TER: 82.20 | train-WER: 97.29 | dev-clean-loss:  158.95157 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 300.80
epoch:       82 | nupdates:       289011 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:14 | bch(ms): 340.52 | smp(ms): 0.80 | fwd(ms): 109.86 | crit-fwd(ms): 4.29 | bwd(ms): 182.44 | optim(ms): 38.65 | loss:  152.86233 | train-TER: 83.99 | train-WER: 111.32 | dev-clean-loss:  158.99447 | dev-clean-TER: 80.02 | dev-clean-WER: 101.94 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.00
epoch:       83 | nupdates:       292579 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:04 | bch(ms): 337.60 | smp(ms): 0.79 | fwd(ms): 109.90 | crit-fwd(ms): 4.30 | bwd(ms): 182.57 | optim(ms): 38.71 | loss:  152.57697 | train-TER: 81.23 | train-WER: 99.01 | dev-clean-loss:  160.90335 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 300.58
epoch:       84 | nupdates:       296147 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 334.98 | smp(ms): 0.87 | fwd(ms): 109.98 | crit-fwd(ms): 4.30 | bwd(ms): 182.64 | optim(ms): 38.70 | loss:  153.24202 | train-TER: 85.83 | train-WER: 95.26 | dev-clean-loss:  160.50108 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.93
epoch:       85 | nupdates:       299715 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:05 | bch(ms): 337.90 | smp(ms): 0.80 | fwd(ms): 109.99 | crit-fwd(ms): 4.32 | bwd(ms): 182.66 | optim(ms): 38.80 | loss:  153.37814 | train-TER: 84.77 | train-WER: 102.79 | dev-clean-loss:  161.58790 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 097 | hrs:  100.57 | thrpt(sec/sec): 300.31
epoch:       86 | nupdates:       303283 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 334.00 | smp(ms): 0.80 | fwd(ms): 109.99 | crit-fwd(ms): 4.30 | bwd(ms): 182.69 | optim(ms): 38.70 | loss:  153.32155 | train-TER: 88.02 | train-WER: 94.68 | dev-clean-loss:  161.49743 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.82
epoch:       87 | nupdates:       306851 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.76 | smp(ms): 0.82 | fwd(ms): 109.99 | crit-fwd(ms): 4.30 | bwd(ms): 182.66 | optim(ms): 38.71 | loss:  154.16143 | train-TER: 85.85 | train-WER: 95.25 | dev-clean-loss:  161.91502 | dev-clean-TER: 81.98 | dev-clean-WER: 100.81 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.12
epoch:       88 | nupdates:       310419 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.24 | smp(ms): 0.82 | fwd(ms): 109.98 | crit-fwd(ms): 4.30 | bwd(ms): 182.62 | optim(ms): 38.71 | loss:  154.19888 | train-TER: 87.26 | train-WER: 95.10 | dev-clean-loss:  162.48530 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.60
epoch:       89 | nupdates:       313987 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.96 | smp(ms): 0.88 | fwd(ms): 110.01 | crit-fwd(ms): 4.31 | bwd(ms): 182.77 | optim(ms): 38.76 | loss:  154.51493 | train-TER: 83.67 | train-WER: 95.09 | dev-clean-loss:  161.40838 | dev-clean-TER: 123.46 | dev-clean-WER: 175.30 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.05
epoch:       90 | nupdates:       317555 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.69 | smp(ms): 0.81 | fwd(ms): 109.95 | crit-fwd(ms): 4.30 | bwd(ms): 182.59 | optim(ms): 38.71 | loss:  154.46733 | train-TER: 87.11 | train-WER: 96.30 | dev-clean-loss:  163.06164 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 303.19
epoch:       91 | nupdates:       321123 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.33 | smp(ms): 0.76 | fwd(ms): 109.98 | crit-fwd(ms): 4.29 | bwd(ms): 182.65 | optim(ms): 38.69 | loss:  154.39079 | train-TER: 86.67 | train-WER: 94.87 | dev-clean-loss:  161.77164 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.52
epoch:       92 | nupdates:       324691 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.84 | smp(ms): 0.80 | fwd(ms): 109.95 | crit-fwd(ms): 4.29 | bwd(ms): 182.59 | optim(ms): 38.68 | loss:  154.24363 | train-TER: 88.02 | train-WER: 94.68 | dev-clean-loss:  160.72886 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.96
epoch:       93 | nupdates:       328259 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.46 | smp(ms): 0.83 | fwd(ms): 110.05 | crit-fwd(ms): 4.39 | bwd(ms): 182.70 | optim(ms): 38.97 | loss:  153.62046 | train-TER: 87.74 | train-WER: 94.55 | dev-clean-loss:  161.40628 | dev-clean-TER: 83.48 | dev-clean-WER: 94.90 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.40
epoch:       94 | nupdates:       331827 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:05 | bch(ms): 337.72 | smp(ms): 0.87 | fwd(ms): 110.62 | crit-fwd(ms): 4.77 | bwd(ms): 183.43 | optim(ms): 40.14 | loss:  154.43271 | train-TER: 86.45 | train-WER: 94.53 | dev-clean-loss:  161.42775 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 300.47
epoch:       95 | nupdates:       335395 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.35 | smp(ms): 0.82 | fwd(ms): 109.97 | crit-fwd(ms): 4.30 | bwd(ms): 182.61 | optim(ms): 38.70 | loss:  154.29831 | train-TER: 86.25 | train-WER: 94.40 | dev-clean-loss:  162.09740 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.50
epoch:       96 | nupdates:       338963 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 334.96 | smp(ms): 0.75 | fwd(ms): 109.93 | crit-fwd(ms): 4.30 | bwd(ms): 182.56 | optim(ms): 38.70 | loss:  154.56103 | train-TER: 83.58 | train-WER: 94.48 | dev-clean-loss:  161.17520 | dev-clean-TER: 81.57 | dev-clean-WER: 101.34 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 302.94
epoch:       97 | nupdates:       342531 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:49 | bch(ms): 333.35 | smp(ms): 0.81 | fwd(ms): 109.72 | crit-fwd(ms): 4.27 | bwd(ms): 182.09 | optim(ms): 38.28 | loss:  154.43085 | train-TER: 85.66 | train-WER: 94.29 | dev-clean-loss:  162.22355 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.41
epoch:       98 | nupdates:       346099 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:49 | bch(ms): 333.44 | smp(ms): 0.79 | fwd(ms): 109.71 | crit-fwd(ms): 4.26 | bwd(ms): 182.12 | optim(ms): 38.30 | loss:  154.56332 | train-TER: 85.08 | train-WER: 94.28 | dev-clean-loss:  161.79685 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.33
epoch:       99 | nupdates:       349667 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.38 | smp(ms): 0.78 | fwd(ms): 109.69 | crit-fwd(ms): 4.27 | bwd(ms): 182.17 | optim(ms): 38.36 | loss:  154.73157 | train-TER: 83.11 | train-WER: 94.60 | dev-clean-loss:  162.06919 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.47
epoch:      100 | nupdates:       353235 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:50 | bch(ms): 333.63 | smp(ms): 0.82 | fwd(ms): 109.65 | crit-fwd(ms): 4.24 | bwd(ms): 182.00 | optim(ms): 38.23 | loss:  155.31400 | train-TER: 83.94 | train-WER: 94.74 | dev-clean-loss:  161.82853 | dev-clean-TER: 81.57 | dev-clean-WER: 101.34 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.15
epoch:      101 | nupdates:       356803 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.21 | smp(ms): 0.80 | fwd(ms): 109.84 | crit-fwd(ms): 4.40 | bwd(ms): 182.33 | optim(ms): 38.73 | loss:  154.61344 | train-TER: 82.08 | train-WER: 94.87 | dev-clean-loss:  162.43304 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.72
epoch:      102 | nupdates:       360371 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:03 | bch(ms): 337.17 | smp(ms): 0.82 | fwd(ms): 110.40 | crit-fwd(ms): 4.79 | bwd(ms): 183.00 | optim(ms): 39.82 | loss:  155.12835 | train-TER: 85.21 | train-WER: 94.35 | dev-clean-loss:  162.40635 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 300.96
epoch:      103 | nupdates:       363939 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.58 | smp(ms): 0.80 | fwd(ms): 110.03 | crit-fwd(ms): 4.53 | bwd(ms): 182.50 | optim(ms): 39.10 | loss:  155.50791 | train-TER: 84.14 | train-WER: 94.62 | dev-clean-loss:  162.67707 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 302.38
epoch:      104 | nupdates:       367507 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:47 | bch(ms): 332.83 | smp(ms): 0.84 | fwd(ms): 109.68 | crit-fwd(ms): 4.25 | bwd(ms): 182.09 | optim(ms): 38.30 | loss:  156.07733 | train-TER: 87.51 | train-WER: 94.90 | dev-clean-loss:  163.63833 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.89
epoch:      105 | nupdates:       371075 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:47 | bch(ms): 332.78 | smp(ms): 0.80 | fwd(ms): 109.64 | crit-fwd(ms): 4.25 | bwd(ms): 182.05 | optim(ms): 38.29 | loss:  156.35874 | train-TER: 87.49 | train-WER: 94.69 | dev-clean-loss:  164.31446 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 304.93
epoch:      106 | nupdates:       374643 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:48 | bch(ms): 333.13 | smp(ms): 0.84 | fwd(ms): 109.66 | crit-fwd(ms): 4.25 | bwd(ms): 182.06 | optim(ms): 38.32 | loss:  155.83097 | train-TER: 85.25 | train-WER: 93.98 | dev-clean-loss:  163.34330 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 304.61
epoch:      107 | nupdates:       378211 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:48 | bch(ms): 333.22 | smp(ms): 0.80 | fwd(ms): 109.66 | crit-fwd(ms): 4.25 | bwd(ms): 182.05 | optim(ms): 38.30 | loss:  155.07300 | train-TER: 85.56 | train-WER: 94.80 | dev-clean-loss:  161.08573 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.53
epoch:      108 | nupdates:       381779 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.80 | smp(ms): 0.71 | fwd(ms): 110.09 | crit-fwd(ms): 4.23 | bwd(ms): 182.52 | optim(ms): 38.23 | loss:  155.13161 | train-TER: 85.41 | train-WER: 93.93 | dev-clean-loss:  160.79763 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.00
epoch:      109 | nupdates:       385347 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.93 | smp(ms): 0.78 | fwd(ms): 109.95 | crit-fwd(ms): 4.23 | bwd(ms): 182.31 | optim(ms): 38.22 | loss:  154.90495 | train-TER: 84.70 | train-WER: 94.20 | dev-clean-loss:  161.20922 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 303.88
epoch:      110 | nupdates:       388915 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:49 | bch(ms): 333.26 | smp(ms): 0.79 | fwd(ms): 109.96 | crit-fwd(ms): 4.22 | bwd(ms): 182.22 | optim(ms): 38.15 | loss:  154.76890 | train-TER: 87.08 | train-WER: 94.66 | dev-clean-loss:  161.31529 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.49
epoch:      111 | nupdates:       392483 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.86 | smp(ms): 0.78 | fwd(ms): 109.99 | crit-fwd(ms): 4.24 | bwd(ms): 182.44 | optim(ms): 38.24 | loss:  154.70887 | train-TER: 86.44 | train-WER: 94.72 | dev-clean-loss:  161.74030 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 303.95
epoch:      112 | nupdates:       396051 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.46 | smp(ms): 0.79 | fwd(ms): 110.04 | crit-fwd(ms): 4.25 | bwd(ms): 182.77 | optim(ms): 38.34 | loss:  154.46265 | train-TER: 86.12 | train-WER: 94.94 | dev-clean-loss:  161.57226 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.40
epoch:      113 | nupdates:       399619 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.57 | smp(ms): 0.80 | fwd(ms): 109.96 | crit-fwd(ms): 4.24 | bwd(ms): 182.65 | optim(ms): 38.27 | loss:  154.35467 | train-TER: 84.76 | train-WER: 94.57 | dev-clean-loss:  162.55857 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 303.30
epoch:      114 | nupdates:       403187 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.84 | smp(ms): 0.76 | fwd(ms): 109.91 | crit-fwd(ms): 4.25 | bwd(ms): 182.60 | optim(ms): 38.31 | loss:  154.71760 | train-TER: 86.77 | train-WER: 94.77 | dev-clean-loss:  161.90953 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.96
epoch:      115 | nupdates:       406755 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.57 | smp(ms): 0.76 | fwd(ms): 110.10 | crit-fwd(ms): 4.41 | bwd(ms): 182.82 | optim(ms): 38.77 | loss:  154.52847 | train-TER: 84.58 | train-WER: 94.47 | dev-clean-loss:  161.62992 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.40
epoch:      116 | nupdates:       410323 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.17 | smp(ms): 0.81 | fwd(ms): 109.96 | crit-fwd(ms): 4.29 | bwd(ms): 182.57 | optim(ms): 38.42 | loss:  154.08250 | train-TER: 85.96 | train-WER: 94.55 | dev-clean-loss:  161.70801 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.67
epoch:      117 | nupdates:       413891 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.15 | smp(ms): 0.75 | fwd(ms): 109.90 | crit-fwd(ms): 4.25 | bwd(ms): 182.56 | optim(ms): 38.32 | loss:  154.27710 | train-TER: 85.55 | train-WER: 94.65 | dev-clean-loss:  161.82331 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.68
epoch:      118 | nupdates:       417459 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.32 | smp(ms): 0.79 | fwd(ms): 110.05 | crit-fwd(ms): 4.33 | bwd(ms): 182.70 | optim(ms): 38.54 | loss:  154.68164 | train-TER: 87.07 | train-WER: 94.80 | dev-clean-loss:  161.52156 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 303.53
epoch:      119 | nupdates:       421027 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.99 | smp(ms): 0.77 | fwd(ms): 109.90 | crit-fwd(ms): 4.24 | bwd(ms): 182.49 | optim(ms): 38.28 | loss:  153.60212 | train-TER: 86.07 | train-WER: 94.82 | dev-clean-loss:  161.30502 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.83
epoch:      120 | nupdates:       424595 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.69 | smp(ms): 0.87 | fwd(ms): 110.06 | crit-fwd(ms): 4.37 | bwd(ms): 182.78 | optim(ms): 38.69 | loss:  154.23309 | train-TER: 83.70 | train-WER: 94.91 | dev-clean-loss:  159.99628 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.29
epoch:      121 | nupdates:       428163 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 334.01 | smp(ms): 0.83 | fwd(ms): 109.90 | crit-fwd(ms): 4.32 | bwd(ms): 182.49 | optim(ms): 38.49 | loss:  154.40798 | train-TER: 85.47 | train-WER: 94.37 | dev-clean-loss:  161.66519 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.81
epoch:      122 | nupdates:       431731 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:50 | bch(ms): 333.78 | smp(ms): 0.78 | fwd(ms): 109.90 | crit-fwd(ms): 4.28 | bwd(ms): 182.53 | optim(ms): 38.39 | loss:  153.47090 | train-TER: 86.08 | train-WER: 94.48 | dev-clean-loss:  161.42364 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 304.02
epoch:      123 | nupdates:       435299 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.82 | smp(ms): 0.78 | fwd(ms): 109.89 | crit-fwd(ms): 4.25 | bwd(ms): 182.50 | optim(ms): 38.31 | loss:  153.96813 | train-TER: 85.28 | train-WER: 94.45 | dev-clean-loss:  161.29903 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.98
epoch:      124 | nupdates:       438867 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.12 | smp(ms): 0.78 | fwd(ms): 109.95 | crit-fwd(ms): 4.24 | bwd(ms): 182.57 | optim(ms): 38.28 | loss:  154.42332 | train-TER: 85.32 | train-WER: 94.75 | dev-clean-loss:  162.01995 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 303.71
epoch:      125 | nupdates:       442435 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:51 | bch(ms): 333.97 | smp(ms): 0.73 | fwd(ms): 109.82 | crit-fwd(ms): 4.26 | bwd(ms): 182.45 | optim(ms): 38.33 | loss:  154.13882 | train-TER: 84.64 | train-WER: 94.04 | dev-clean-loss:  162.06003 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.84
epoch:      126 | nupdates:       446003 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:50 | bch(ms): 333.56 | smp(ms): 0.75 | fwd(ms): 109.56 | crit-fwd(ms): 4.23 | bwd(ms): 182.02 | optim(ms): 38.25 | loss:  154.30960 | train-TER: 83.82 | train-WER: 94.35 | dev-clean-loss:  160.98784 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 097 | hrs:  100.57 | thrpt(sec/sec): 304.21
epoch:      127 | nupdates:       449571 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.87 | smp(ms): 0.79 | fwd(ms): 109.88 | crit-fwd(ms): 4.27 | bwd(ms): 182.60 | optim(ms): 38.55 | loss:  154.56189 | train-TER: 81.83 | train-WER: 94.00 | dev-clean-loss:  160.86199 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.03
epoch:      128 | nupdates:       453139 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.21 | smp(ms): 0.86 | fwd(ms): 110.10 | crit-fwd(ms): 4.28 | bwd(ms): 182.91 | optim(ms): 38.69 | loss:  154.19063 | train-TER: 84.05 | train-WER: 94.39 | dev-clean-loss:  162.42944 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.73
epoch:      129 | nupdates:       456707 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.34 | smp(ms): 0.82 | fwd(ms): 110.04 | crit-fwd(ms): 4.27 | bwd(ms): 182.79 | optim(ms): 38.66 | loss:  154.23648 | train-TER: 81.89 | train-WER: 94.12 | dev-clean-loss:  162.58422 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.60
epoch:      130 | nupdates:       460275 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.15 | smp(ms): 0.83 | fwd(ms): 110.11 | crit-fwd(ms): 4.27 | bwd(ms): 182.85 | optim(ms): 38.62 | loss:  154.52280 | train-TER: 84.59 | train-WER: 95.07 | dev-clean-loss:  161.16557 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.78
epoch:      131 | nupdates:       463843 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.86 | smp(ms): 0.84 | fwd(ms): 110.13 | crit-fwd(ms): 4.28 | bwd(ms): 182.91 | optim(ms): 38.67 | loss:  155.04494 | train-TER: 85.23 | train-WER: 94.18 | dev-clean-loss:  162.77481 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.04
epoch:      132 | nupdates:       467411 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.38 | smp(ms): 0.80 | fwd(ms): 110.19 | crit-fwd(ms): 4.30 | bwd(ms): 183.03 | optim(ms): 38.77 | loss:  154.17915 | train-TER: 83.82 | train-WER: 94.12 | dev-clean-loss:  162.40305 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.57
epoch:      133 | nupdates:       470979 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.74 | smp(ms): 0.83 | fwd(ms): 110.18 | crit-fwd(ms): 4.27 | bwd(ms): 182.97 | optim(ms): 38.65 | loss:  154.87226 | train-TER: 81.46 | train-WER: 94.21 | dev-clean-loss:  162.43960 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.24
epoch:      134 | nupdates:       474547 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.83 | smp(ms): 0.76 | fwd(ms): 110.25 | crit-fwd(ms): 4.33 | bwd(ms): 183.20 | optim(ms): 38.89 | loss:  154.60203 | train-TER: 85.29 | train-WER: 94.67 | dev-clean-loss:  162.35641 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.16
epoch:      135 | nupdates:       478115 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.68 | smp(ms): 0.81 | fwd(ms): 110.09 | crit-fwd(ms): 4.28 | bwd(ms): 182.89 | optim(ms): 38.68 | loss:  155.14193 | train-TER: 84.52 | train-WER: 94.76 | dev-clean-loss:  163.51076 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 302.30
epoch:      136 | nupdates:       481683 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.36 | smp(ms): 0.85 | fwd(ms): 110.08 | crit-fwd(ms): 4.27 | bwd(ms): 182.90 | optim(ms): 38.67 | loss:  155.51227 | train-TER: 85.14 | train-WER: 94.69 | dev-clean-loss:  163.24225 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.59
epoch:      137 | nupdates:       485251 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.28 | smp(ms): 0.84 | fwd(ms): 110.12 | crit-fwd(ms): 4.27 | bwd(ms): 182.98 | optim(ms): 38.70 | loss:  155.11367 | train-TER: 85.57 | train-WER: 94.71 | dev-clean-loss:  163.96871 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 302.66
epoch:      138 | nupdates:       488819 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.95 | smp(ms): 0.80 | fwd(ms): 110.16 | crit-fwd(ms): 4.28 | bwd(ms): 183.07 | optim(ms): 38.74 | loss:  156.14757 | train-TER: 83.61 | train-WER: 94.54 | dev-clean-loss:  163.81721 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.06
epoch:      139 | nupdates:       492387 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.94 | smp(ms): 0.79 | fwd(ms): 110.20 | crit-fwd(ms): 4.28 | bwd(ms): 183.05 | optim(ms): 38.72 | loss:  156.00515 | train-TER: 84.30 | train-WER: 94.82 | dev-clean-loss:  164.70073 | dev-clean-TER: 83.55 | dev-clean-WER: 95.46 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.06
epoch:      140 | nupdates:       495955 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:00 | bch(ms): 336.50 | smp(ms): 0.80 | fwd(ms): 110.33 | crit-fwd(ms): 4.37 | bwd(ms): 183.31 | optim(ms): 39.02 | loss:  155.89516 | train-TER: 84.17 | train-WER: 94.97 | dev-clean-loss:  164.48078 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 301.56
epoch:      141 | nupdates:       499523 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.53 | smp(ms): 0.88 | fwd(ms): 110.16 | crit-fwd(ms): 4.29 | bwd(ms): 183.11 | optim(ms): 38.75 | loss:  156.37795 | train-TER: 84.48 | train-WER: 94.67 | dev-clean-loss:  164.81515 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 302.43
epoch:      142 | nupdates:       503091 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.95 | smp(ms): 0.79 | fwd(ms): 110.08 | crit-fwd(ms): 4.28 | bwd(ms): 182.98 | optim(ms): 38.72 | loss:  157.58943 | train-TER: 81.51 | train-WER: 94.64 | dev-clean-loss:  165.39143 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.06
epoch:      143 | nupdates:       506659 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.41 | smp(ms): 0.88 | fwd(ms): 110.11 | crit-fwd(ms): 4.28 | bwd(ms): 182.99 | optim(ms): 38.73 | loss:  157.55311 | train-TER: 85.15 | train-WER: 95.10 | dev-clean-loss:  165.66821 | dev-clean-TER: 82.39 | dev-clean-WER: 96.81 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.54
epoch:      144 | nupdates:       510227 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.20 | smp(ms): 0.81 | fwd(ms): 110.18 | crit-fwd(ms): 4.28 | bwd(ms): 183.07 | optim(ms): 38.72 | loss:  158.02096 | train-TER: 84.08 | train-WER: 95.30 | dev-clean-loss:  166.09003 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 302.73
epoch:      145 | nupdates:       513795 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.42 | smp(ms): 0.72 | fwd(ms): 110.19 | crit-fwd(ms): 4.27 | bwd(ms): 183.11 | optim(ms): 38.70 | loss:  158.40458 | train-TER: 85.42 | train-WER: 95.09 | dev-clean-loss:  165.56787 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.53
epoch:      146 | nupdates:       517363 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:59 | bch(ms): 336.15 | smp(ms): 0.82 | fwd(ms): 110.29 | crit-fwd(ms): 4.28 | bwd(ms): 183.26 | optim(ms): 38.75 | loss:  158.44356 | train-TER: 84.95 | train-WER: 95.17 | dev-clean-loss:  166.22395 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 301.87
epoch:      147 | nupdates:       520931 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.68 | smp(ms): 0.83 | fwd(ms): 110.20 | crit-fwd(ms): 4.27 | bwd(ms): 183.05 | optim(ms): 38.68 | loss:  158.78301 | train-TER: 84.48 | train-WER: 94.85 | dev-clean-loss:  165.87655 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.30
epoch:      148 | nupdates:       524499 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:03 | bch(ms): 337.40 | smp(ms): 0.82 | fwd(ms): 110.41 | crit-fwd(ms): 4.48 | bwd(ms): 183.31 | optim(ms): 39.29 | loss:  158.91239 | train-TER: 83.99 | train-WER: 94.77 | dev-clean-loss:  166.32557 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 300.75
epoch:      149 | nupdates:       528067 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:58 | bch(ms): 335.94 | smp(ms): 0.86 | fwd(ms): 110.18 | crit-fwd(ms): 4.27 | bwd(ms): 183.07 | optim(ms): 38.72 | loss:  159.08712 | train-TER: 84.33 | train-WER: 94.69 | dev-clean-loss:  165.85832 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.07
epoch:      150 | nupdates:       531635 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.61 | smp(ms): 0.84 | fwd(ms): 110.14 | crit-fwd(ms): 4.29 | bwd(ms): 182.99 | optim(ms): 38.73 | loss:  159.89116 | train-TER: 83.91 | train-WER: 94.38 | dev-clean-loss:  166.66715 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.36
epoch:      151 | nupdates:       535203 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:57 | bch(ms): 335.51 | smp(ms): 0.88 | fwd(ms): 110.09 | crit-fwd(ms): 4.27 | bwd(ms): 182.60 | optim(ms): 38.61 | loss:  159.37660 | train-TER: 84.00 | train-WER: 94.77 | dev-clean-loss:  165.89872 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.45
epoch:      152 | nupdates:       538771 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.37 | smp(ms): 0.82 | fwd(ms): 110.10 | crit-fwd(ms): 4.25 | bwd(ms): 182.57 | optim(ms): 38.53 | loss:  159.18712 | train-TER: 83.72 | train-WER: 94.74 | dev-clean-loss:  166.16447 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.57
epoch:      153 | nupdates:       542339 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.09 | smp(ms): 0.82 | fwd(ms): 110.11 | crit-fwd(ms): 4.25 | bwd(ms): 182.55 | optim(ms): 38.52 | loss:  159.37344 | train-TER: 84.64 | train-WER: 94.96 | dev-clean-loss:  166.35110 | dev-clean-TER: 83.55 | dev-clean-WER: 95.46 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.83
epoch:      154 | nupdates:       545907 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.00 | smp(ms): 0.87 | fwd(ms): 110.17 | crit-fwd(ms): 4.26 | bwd(ms): 182.70 | optim(ms): 38.57 | loss:  159.88270 | train-TER: 85.44 | train-WER: 95.43 | dev-clean-loss:  166.42316 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 302.91
epoch:      155 | nupdates:       549475 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.62 | smp(ms): 0.81 | fwd(ms): 110.08 | crit-fwd(ms): 4.25 | bwd(ms): 182.55 | optim(ms): 38.55 | loss:  159.64685 | train-TER: 84.50 | train-WER: 94.12 | dev-clean-loss:  166.31384 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.25
epoch:      156 | nupdates:       553043 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.83 | smp(ms): 0.83 | fwd(ms): 110.04 | crit-fwd(ms): 4.25 | bwd(ms): 182.44 | optim(ms): 38.52 | loss:  160.08803 | train-TER: 83.30 | train-WER: 94.72 | dev-clean-loss:  165.81207 | dev-clean-TER: 81.57 | dev-clean-WER: 101.34 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 303.07
epoch:      157 | nupdates:       556611 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.27 | smp(ms): 0.86 | fwd(ms): 110.04 | crit-fwd(ms): 4.26 | bwd(ms): 182.50 | optim(ms): 38.58 | loss:  159.32672 | train-TER: 83.51 | train-WER: 94.52 | dev-clean-loss:  166.05583 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 302.67
epoch:      158 | nupdates:       560179 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 334.96 | smp(ms): 0.87 | fwd(ms): 110.06 | crit-fwd(ms): 4.26 | bwd(ms): 182.49 | optim(ms): 38.56 | loss:  159.76073 | train-TER: 83.95 | train-WER: 94.43 | dev-clean-loss:  165.75618 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.95
epoch:      159 | nupdates:       563747 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.04 | smp(ms): 0.83 | fwd(ms): 110.13 | crit-fwd(ms): 4.27 | bwd(ms): 182.64 | optim(ms): 38.59 | loss:  159.82187 | train-TER: 84.46 | train-WER: 94.13 | dev-clean-loss:  165.97481 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.88
epoch:      160 | nupdates:       567315 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 334.96 | smp(ms): 0.78 | fwd(ms): 110.17 | crit-fwd(ms): 4.25 | bwd(ms): 182.63 | optim(ms): 38.54 | loss:  160.52518 | train-TER: 83.06 | train-WER: 94.15 | dev-clean-loss:  167.09910 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 096 | hrs:  100.57 | thrpt(sec/sec): 302.95
epoch:      161 | nupdates:       570883 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.03 | smp(ms): 0.86 | fwd(ms): 110.10 | crit-fwd(ms): 4.25 | bwd(ms): 182.53 | optim(ms): 38.52 | loss:  160.61119 | train-TER: 83.98 | train-WER: 94.29 | dev-clean-loss:  166.25240 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.89
epoch:      162 | nupdates:       574451 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.76 | smp(ms): 0.87 | fwd(ms): 110.05 | crit-fwd(ms): 4.25 | bwd(ms): 182.49 | optim(ms): 38.53 | loss:  160.40325 | train-TER: 84.56 | train-WER: 94.79 | dev-clean-loss:  166.68233 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.13
epoch:      163 | nupdates:       578019 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.43 | smp(ms): 0.85 | fwd(ms): 110.09 | crit-fwd(ms): 4.25 | bwd(ms): 182.55 | optim(ms): 38.55 | loss:  160.61654 | train-TER: 84.90 | train-WER: 94.00 | dev-clean-loss:  166.60610 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.43
epoch:      164 | nupdates:       581587 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 334.98 | smp(ms): 0.82 | fwd(ms): 110.09 | crit-fwd(ms): 4.26 | bwd(ms): 182.58 | optim(ms): 38.58 | loss:  160.11053 | train-TER: 83.43 | train-WER: 94.13 | dev-clean-loss:  166.55324 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.92
epoch:      165 | nupdates:       585155 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 334.92 | smp(ms): 0.83 | fwd(ms): 110.14 | crit-fwd(ms): 4.25 | bwd(ms): 182.62 | optim(ms): 38.54 | loss:  161.45298 | train-TER: 83.97 | train-WER: 94.08 | dev-clean-loss:  166.81902 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.98
epoch:      166 | nupdates:       588723 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.84 | smp(ms): 0.78 | fwd(ms): 110.13 | crit-fwd(ms): 4.24 | bwd(ms): 182.57 | optim(ms): 38.47 | loss:  161.25563 | train-TER: 84.33 | train-WER: 94.25 | dev-clean-loss:  166.76187 | dev-clean-TER: 83.32 | dev-clean-WER: 96.70 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.06
epoch:      167 | nupdates:       592291 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.12 | smp(ms): 0.84 | fwd(ms): 110.08 | crit-fwd(ms): 4.26 | bwd(ms): 182.55 | optim(ms): 38.53 | loss:  161.71987 | train-TER: 82.33 | train-WER: 94.10 | dev-clean-loss:  166.82693 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.80
epoch:      168 | nupdates:       595859 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.08 | smp(ms): 0.81 | fwd(ms): 110.11 | crit-fwd(ms): 4.25 | bwd(ms): 182.53 | optim(ms): 38.54 | loss:  161.47708 | train-TER: 81.67 | train-WER: 94.20 | dev-clean-loss:  166.92267 | dev-clean-TER: 80.56 | dev-clean-WER: 97.82 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 302.84
epoch:      169 | nupdates:       599427 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.90 | smp(ms): 0.85 | fwd(ms): 110.05 | crit-fwd(ms): 4.26 | bwd(ms): 182.49 | optim(ms): 38.55 | loss:  161.48364 | train-TER: 83.67 | train-WER: 94.77 | dev-clean-loss:  166.67452 | dev-clean-TER: 83.45 | dev-clean-WER: 96.07 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.00
epoch:      170 | nupdates:       602995 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:52 | bch(ms): 334.31 | smp(ms): 0.87 | fwd(ms): 110.10 | crit-fwd(ms): 4.25 | bwd(ms): 182.50 | optim(ms): 38.50 | loss:  161.60627 | train-TER: 84.67 | train-WER: 93.66 | dev-clean-loss:  166.82503 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.54
epoch:      171 | nupdates:       606563 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.67 | smp(ms): 0.79 | fwd(ms): 110.15 | crit-fwd(ms): 4.26 | bwd(ms): 182.64 | optim(ms): 38.55 | loss:  161.79084 | train-TER: 83.98 | train-WER: 93.67 | dev-clean-loss:  166.56082 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.21
epoch:      172 | nupdates:       610131 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.60 | smp(ms): 0.82 | fwd(ms): 110.20 | crit-fwd(ms): 4.25 | bwd(ms): 182.65 | optim(ms): 38.51 | loss:  161.94637 | train-TER: 84.44 | train-WER: 94.30 | dev-clean-loss:  166.67404 | dev-clean-TER: 82.39 | dev-clean-WER: 96.81 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.28
epoch:      173 | nupdates:       613699 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:56 | bch(ms): 335.47 | smp(ms): 0.84 | fwd(ms): 110.29 | crit-fwd(ms): 4.34 | bwd(ms): 182.89 | optim(ms): 38.84 | loss:  161.85668 | train-TER: 83.73 | train-WER: 94.26 | dev-clean-loss:  166.28433 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.49
epoch:      174 | nupdates:       617267 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.43 | smp(ms): 0.83 | fwd(ms): 110.10 | crit-fwd(ms): 4.26 | bwd(ms): 182.59 | optim(ms): 38.58 | loss:  161.81610 | train-TER: 84.76 | train-WER: 93.56 | dev-clean-loss:  167.11956 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.43
epoch:      175 | nupdates:       620835 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.91 | smp(ms): 0.78 | fwd(ms): 110.17 | crit-fwd(ms): 4.31 | bwd(ms): 182.85 | optim(ms): 38.75 | loss:  162.18653 | train-TER: 84.75 | train-WER: 93.56 | dev-clean-loss:  167.21651 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.99
epoch:      176 | nupdates:       624403 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.84 | smp(ms): 0.85 | fwd(ms): 110.17 | crit-fwd(ms): 4.28 | bwd(ms): 182.84 | optim(ms): 38.66 | loss:  162.17333 | train-TER: 84.81 | train-WER: 93.65 | dev-clean-loss:  166.43163 | dev-clean-TER: 83.14 | dev-clean-WER: 94.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.06
epoch:      177 | nupdates:       627971 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.04 | smp(ms): 0.86 | fwd(ms): 110.18 | crit-fwd(ms): 4.27 | bwd(ms): 182.74 | optim(ms): 38.62 | loss:  161.75354 | train-TER: 83.44 | train-WER: 95.01 | dev-clean-loss:  165.93003 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.88
epoch:      178 | nupdates:       631539 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:53 | bch(ms): 334.53 | smp(ms): 0.83 | fwd(ms): 110.19 | crit-fwd(ms): 4.27 | bwd(ms): 182.75 | optim(ms): 38.59 | loss:  161.76806 | train-TER: 86.49 | train-WER: 94.52 | dev-clean-loss:  165.73376 | dev-clean-TER: 82.22 | dev-clean-WER: 95.68 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.34
epoch:      179 | nupdates:       635107 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:54 | bch(ms): 334.88 | smp(ms): 0.85 | fwd(ms): 110.13 | crit-fwd(ms): 4.27 | bwd(ms): 182.64 | optim(ms): 38.62 | loss:  162.79040 | train-TER: 83.56 | train-WER: 95.27 | dev-clean-loss:  166.64503 | dev-clean-TER: 82.39 | dev-clean-WER: 96.81 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 303.02
epoch:      180 | nupdates:       638675 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:19:55 | bch(ms): 335.12 | smp(ms): 0.87 | fwd(ms): 110.09 | crit-fwd(ms): 4.27 | bwd(ms): 182.61 | optim(ms): 38.58 | loss:  161.87160 | train-TER: 84.29 | train-WER: 94.63 | dev-clean-loss:  165.83350 | dev-clean-TER: 85.36 | dev-clean-WER: 94.60 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 302.80

epoch 1-19, lr increases from 0 to value specific in --lr. It should be normal. epoch 20-160, lr keeps 0.01, loss goes down epoch 161-, it doesn't seem normal I'm not sure if I did the analysis right, or if I need to wait more time to be sure there is no convergence.

tlikhomanenko commented 4 years ago

Yep, definitely seems that these parameters doesn't work well. I didn't try with 1 GPU. Could you try to set --pretrainWindow say 2000?

xiaosdawn commented 4 years ago

OK, I'll try. Thanks.

xiaosdawn commented 4 years ago

I try to set --pretrainWindow=2000, 001_log as:

epoch:        1 | nupdates:         2001 | lr: 0.000313 | lrcriterion: 0.000313 | runtime: 00:12:19 | bch(ms): 369.35 | smp(ms): 1.16 | fwd(ms): 111.97 | crit-fwd(ms): 5.34 | bwd(ms): 182.64 | optim(ms): 38.58 | loss:  319.22716 | train-TER: 232.87 | train-WER: 360.73 | dev-clean-loss:  167.54987 | dev-clean-TER: 407.98 | dev-clean-WER: 589.89 | avg-isz: 1254 | avg-tsz: 043 | max-tsz: 093 | hrs:   55.76 | thrpt(sec/sec): 271.61
epoch:        2 | nupdates:         5568 | lr: 0.000870 | lrcriterion: 0.000870 | runtime: 00:21:23 | bch(ms): 359.85 | smp(ms): 1.01 | fwd(ms): 110.29 | crit-fwd(ms): 4.30 | bwd(ms): 183.48 | optim(ms): 38.20 | loss:  278.69952 | train-TER: 177.97 | train-WER: 322.16 | dev-clean-loss:  155.01459 | dev-clean-TER: 131.30 | dev-clean-WER: 214.57 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 281.99
epoch:        3 | nupdates:         9136 | lr: 0.001427 | lrcriterion: 0.001427 | runtime: 00:20:44 | bch(ms): 348.71 | smp(ms): 1.09 | fwd(ms): 110.18 | crit-fwd(ms): 4.28 | bwd(ms): 183.65 | optim(ms): 38.46 | loss:  266.83660 | train-TER: 118.17 | train-WER: 181.49 | dev-clean-loss:  150.57047 | dev-clean-TER: 114.52 | dev-clean-WER: 166.19 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 291.00
epoch:        4 | nupdates:        12704 | lr: 0.001985 | lrcriterion: 0.001985 | runtime: 00:20:14 | bch(ms): 340.51 | smp(ms): 1.12 | fwd(ms): 109.97 | crit-fwd(ms): 4.18 | bwd(ms): 183.31 | optim(ms): 37.92 | loss:  258.17459 | train-TER: 75.50 | train-WER: 108.09 | dev-clean-loss:  146.03059 | dev-clean-TER: 113.81 | dev-clean-WER: 164.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 298.00
epoch:        5 | nupdates:        16272 | lr: 0.002542 | lrcriterion: 0.002542 | runtime: 00:20:13 | bch(ms): 340.06 | smp(ms): 1.09 | fwd(ms): 109.85 | crit-fwd(ms): 4.18 | bwd(ms): 183.21 | optim(ms): 37.83 | loss:  251.10969 | train-TER: 76.67 | train-WER: 107.18 | dev-clean-loss:  142.91633 | dev-clean-TER: 124.16 | dev-clean-WER: 181.86 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.40
epoch:        6 | nupdates:        19840 | lr: 0.003100 | lrcriterion: 0.003100 | runtime: 00:20:15 | bch(ms): 340.65 | smp(ms): 1.08 | fwd(ms): 109.95 | crit-fwd(ms): 4.22 | bwd(ms): 183.30 | optim(ms): 37.97 | loss:  245.95887 | train-TER: 78.24 | train-WER: 108.86 | dev-clean-loss:  141.26636 | dev-clean-TER: 147.25 | dev-clean-WER: 221.44 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.89
epoch:        7 | nupdates:        23408 | lr: 0.003658 | lrcriterion: 0.003658 | runtime: 00:20:24 | bch(ms): 343.33 | smp(ms): 1.10 | fwd(ms): 109.86 | crit-fwd(ms): 4.17 | bwd(ms): 183.17 | optim(ms): 37.73 | loss:  241.94782 | train-TER: 94.16 | train-WER: 141.90 | dev-clean-loss:  139.26814 | dev-clean-TER: 123.29 | dev-clean-WER: 195.25 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.56
epoch:        8 | nupdates:        26976 | lr: 0.004215 | lrcriterion: 0.004215 | runtime: 00:20:19 | bch(ms): 341.85 | smp(ms): 1.10 | fwd(ms): 109.79 | crit-fwd(ms): 4.15 | bwd(ms): 183.08 | optim(ms): 37.66 | loss:  238.60524 | train-TER: 86.27 | train-WER: 127.34 | dev-clean-loss:  137.50114 | dev-clean-TER: 115.47 | dev-clean-WER: 178.28 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 296.84
epoch:        9 | nupdates:        30544 | lr: 0.004772 | lrcriterion: 0.004772 | runtime: 00:20:29 | bch(ms): 344.51 | smp(ms): 1.07 | fwd(ms): 109.75 | crit-fwd(ms): 4.18 | bwd(ms): 183.02 | optim(ms): 37.82 | loss:  235.67307 | train-TER: 101.53 | train-WER: 155.34 | dev-clean-loss:  136.13499 | dev-clean-TER: 114.81 | dev-clean-WER: 172.68 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.55
epoch:       10 | nupdates:        34112 | lr: 0.005330 | lrcriterion: 0.005330 | runtime: 00:20:27 | bch(ms): 344.16 | smp(ms): 1.09 | fwd(ms): 109.70 | crit-fwd(ms): 4.16 | bwd(ms): 183.01 | optim(ms): 37.75 | loss:  233.06011 | train-TER: 97.85 | train-WER: 152.22 | dev-clean-loss:  135.73419 | dev-clean-TER: 144.94 | dev-clean-WER: 214.47 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.85
epoch:       11 | nupdates:        37680 | lr: 0.005888 | lrcriterion: 0.005888 | runtime: 00:20:36 | bch(ms): 346.52 | smp(ms): 1.08 | fwd(ms): 109.76 | crit-fwd(ms): 4.16 | bwd(ms): 183.25 | optim(ms): 37.85 | loss:  230.60495 | train-TER: 113.51 | train-WER: 175.17 | dev-clean-loss:  134.91158 | dev-clean-TER: 118.21 | dev-clean-WER: 183.02 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 292.84
epoch:       12 | nupdates:        41248 | lr: 0.006445 | lrcriterion: 0.006445 | runtime: 00:20:44 | bch(ms): 348.67 | smp(ms): 1.09 | fwd(ms): 109.75 | crit-fwd(ms): 4.17 | bwd(ms): 182.97 | optim(ms): 37.74 | loss:  228.19558 | train-TER: 132.60 | train-WER: 207.65 | dev-clean-loss:  134.56208 | dev-clean-TER: 118.43 | dev-clean-WER: 178.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 291.03
epoch:       13 | nupdates:        44816 | lr: 0.007003 | lrcriterion: 0.007003 | runtime: 00:20:47 | bch(ms): 349.66 | smp(ms): 1.07 | fwd(ms): 109.67 | crit-fwd(ms): 4.18 | bwd(ms): 182.89 | optim(ms): 37.82 | loss:  226.12033 | train-TER: 136.80 | train-WER: 217.16 | dev-clean-loss:  133.87504 | dev-clean-TER: 156.27 | dev-clean-WER: 245.81 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 290.21
epoch:       14 | nupdates:        48384 | lr: 0.007560 | lrcriterion: 0.007560 | runtime: 00:20:35 | bch(ms): 346.32 | smp(ms): 1.00 | fwd(ms): 109.52 | crit-fwd(ms): 4.16 | bwd(ms): 182.64 | optim(ms): 37.69 | loss:  223.85356 | train-TER: 116.26 | train-WER: 184.57 | dev-clean-loss:  133.21993 | dev-clean-TER: 119.36 | dev-clean-WER: 197.97 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.01
epoch:       15 | nupdates:        51952 | lr: 0.008117 | lrcriterion: 0.008117 | runtime: 00:20:36 | bch(ms): 346.60 | smp(ms): 1.11 | fwd(ms): 109.70 | crit-fwd(ms): 4.16 | bwd(ms): 182.94 | optim(ms): 37.69 | loss:  221.49561 | train-TER: 117.56 | train-WER: 182.89 | dev-clean-loss:  133.36835 | dev-clean-TER: 143.92 | dev-clean-WER: 173.58 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 292.77
epoch:       16 | nupdates:        55520 | lr: 0.008675 | lrcriterion: 0.008675 | runtime: 00:20:38 | bch(ms): 347.05 | smp(ms): 1.07 | fwd(ms): 109.51 | crit-fwd(ms): 4.20 | bwd(ms): 182.58 | optim(ms): 37.88 | loss:  219.29123 | train-TER: 123.28 | train-WER: 191.73 | dev-clean-loss:  133.19510 | dev-clean-TER: 131.98 | dev-clean-WER: 207.09 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 292.39
epoch:       17 | nupdates:        59088 | lr: 0.009233 | lrcriterion: 0.009233 | runtime: 00:20:34 | bch(ms): 346.10 | smp(ms): 1.05 | fwd(ms): 109.69 | crit-fwd(ms): 4.14 | bwd(ms): 182.88 | optim(ms): 37.63 | loss:  217.21415 | train-TER: 116.87 | train-WER: 180.89 | dev-clean-loss:  133.59193 | dev-clean-TER: 181.51 | dev-clean-WER: 276.71 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 293.19
epoch:       18 | nupdates:        62656 | lr: 0.009790 | lrcriterion: 0.009790 | runtime: 00:20:25 | bch(ms): 343.42 | smp(ms): 1.12 | fwd(ms): 109.89 | crit-fwd(ms): 4.14 | bwd(ms): 183.23 | optim(ms): 37.70 | loss:  214.96007 | train-TER: 86.04 | train-WER: 132.88 | dev-clean-loss:  133.10148 | dev-clean-TER: 121.84 | dev-clean-WER: 188.85 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 295.49
epoch:       19 | nupdates:        66224 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:30 | bch(ms): 344.75 | smp(ms): 1.07 | fwd(ms): 110.06 | crit-fwd(ms): 4.24 | bwd(ms): 183.32 | optim(ms): 38.00 | loss:  212.97140 | train-TER: 91.66 | train-WER: 138.41 | dev-clean-loss:  132.92021 | dev-clean-TER: 142.54 | dev-clean-WER: 215.58 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.34
epoch:       20 | nupdates:        69792 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:27 | bch(ms): 344.01 | smp(ms): 1.06 | fwd(ms): 109.91 | crit-fwd(ms): 4.14 | bwd(ms): 183.06 | optim(ms): 37.59 | loss:  210.86179 | train-TER: 96.59 | train-WER: 145.13 | dev-clean-loss:  133.07213 | dev-clean-TER: 165.19 | dev-clean-WER: 264.55 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 294.98
epoch:       21 | nupdates:        73360 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:26 | bch(ms): 343.87 | smp(ms): 1.07 | fwd(ms): 109.97 | crit-fwd(ms): 4.19 | bwd(ms): 183.20 | optim(ms): 37.84 | loss:  208.53992 | train-TER: 92.68 | train-WER: 139.41 | dev-clean-loss:  133.26886 | dev-clean-TER: 181.73 | dev-clean-WER: 272.50 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.10
epoch:       22 | nupdates:        76928 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:28 | bch(ms): 344.24 | smp(ms): 1.09 | fwd(ms): 110.00 | crit-fwd(ms): 4.15 | bwd(ms): 183.23 | optim(ms): 37.69 | loss:  206.54754 | train-TER: 98.25 | train-WER: 144.29 | dev-clean-loss:  133.51797 | dev-clean-TER: 80.92 | dev-clean-WER: 99.32 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 294.78
epoch:       23 | nupdates:        80496 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:23 | bch(ms): 343.03 | smp(ms): 1.07 | fwd(ms): 110.02 | crit-fwd(ms): 4.16 | bwd(ms): 183.26 | optim(ms): 37.67 | loss:  204.41112 | train-TER: 88.31 | train-WER: 131.61 | dev-clean-loss:  134.13168 | dev-clean-TER: 138.43 | dev-clean-WER: 205.49 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.82
epoch:       24 | nupdates:        84064 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:22 | bch(ms): 342.68 | smp(ms): 1.10 | fwd(ms): 110.01 | crit-fwd(ms): 4.20 | bwd(ms): 183.21 | optim(ms): 37.83 | loss:  202.54595 | train-TER: 83.44 | train-WER: 125.84 | dev-clean-loss:  134.51623 | dev-clean-TER: 131.86 | dev-clean-WER: 205.23 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.12
epoch:       25 | nupdates:        87632 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:27 | bch(ms): 343.98 | smp(ms): 1.09 | fwd(ms): 109.89 | crit-fwd(ms): 4.13 | bwd(ms): 183.00 | optim(ms): 37.57 | loss:  200.61941 | train-TER: 91.24 | train-WER: 147.34 | dev-clean-loss:  135.06454 | dev-clean-TER: 154.29 | dev-clean-WER: 223.90 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.00
epoch:       26 | nupdates:        91200 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.46 | smp(ms): 1.07 | fwd(ms): 109.99 | crit-fwd(ms): 4.16 | bwd(ms): 183.24 | optim(ms): 37.72 | loss:  198.69316 | train-TER: 93.67 | train-WER: 141.74 | dev-clean-loss:  135.47330 | dev-clean-TER: 139.05 | dev-clean-WER: 215.45 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.45
epoch:       27 | nupdates:        94768 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.47 | smp(ms): 1.05 | fwd(ms): 109.92 | crit-fwd(ms): 4.15 | bwd(ms): 183.14 | optim(ms): 37.67 | loss:  196.95412 | train-TER: 90.62 | train-WER: 141.04 | dev-clean-loss:  136.39837 | dev-clean-TER: 172.66 | dev-clean-WER: 245.51 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.44
epoch:       28 | nupdates:        98336 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:24 | bch(ms): 343.19 | smp(ms): 1.06 | fwd(ms): 109.94 | crit-fwd(ms): 4.18 | bwd(ms): 183.16 | optim(ms): 37.81 | loss:  195.15938 | train-TER: 86.57 | train-WER: 129.51 | dev-clean-loss:  136.46274 | dev-clean-TER: 129.13 | dev-clean-WER: 189.61 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 295.68
epoch:       29 | nupdates:       101904 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:24 | bch(ms): 343.32 | smp(ms): 1.06 | fwd(ms): 109.85 | crit-fwd(ms): 4.12 | bwd(ms): 182.83 | optim(ms): 37.48 | loss:  193.62253 | train-TER: 93.60 | train-WER: 143.82 | dev-clean-loss:  137.18896 | dev-clean-TER: 314.98 | dev-clean-WER: 584.85 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.57
epoch:       30 | nupdates:       105472 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:26 | bch(ms): 343.67 | smp(ms): 1.13 | fwd(ms): 109.91 | crit-fwd(ms): 4.15 | bwd(ms): 182.94 | optim(ms): 37.62 | loss:  191.74782 | train-TER: 92.57 | train-WER: 141.05 | dev-clean-loss:  137.93664 | dev-clean-TER: 144.95 | dev-clean-WER: 207.11 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.27
epoch:       31 | nupdates:       109040 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.40 | smp(ms): 1.08 | fwd(ms): 109.90 | crit-fwd(ms): 4.13 | bwd(ms): 182.95 | optim(ms): 37.52 | loss:  190.43215 | train-TER: 89.28 | train-WER: 137.20 | dev-clean-loss:  138.19775 | dev-clean-TER: 159.71 | dev-clean-WER: 236.99 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.50
epoch:       32 | nupdates:       112608 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:22 | bch(ms): 342.71 | smp(ms): 0.98 | fwd(ms): 109.90 | crit-fwd(ms): 4.14 | bwd(ms): 182.88 | optim(ms): 37.54 | loss:  189.08973 | train-TER: 92.45 | train-WER: 136.19 | dev-clean-loss:  138.95170 | dev-clean-TER: 127.34 | dev-clean-WER: 179.34 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.09
epoch:       33 | nupdates:       116176 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:22 | bch(ms): 342.68 | smp(ms): 1.04 | fwd(ms): 110.07 | crit-fwd(ms): 4.25 | bwd(ms): 183.15 | optim(ms): 38.00 | loss:  187.76152 | train-TER: 82.99 | train-WER: 121.01 | dev-clean-loss:  139.45738 | dev-clean-TER: 141.05 | dev-clean-WER: 206.95 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 094 | hrs:  100.57 | thrpt(sec/sec): 296.12
epoch:       34 | nupdates:       119744 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:20 | bch(ms): 342.14 | smp(ms): 1.09 | fwd(ms): 109.87 | crit-fwd(ms): 4.11 | bwd(ms): 182.82 | optim(ms): 37.46 | loss:  186.16443 | train-TER: 86.27 | train-WER: 130.61 | dev-clean-loss:  140.03219 | dev-clean-TER: 135.51 | dev-clean-WER: 185.89 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.59
epoch:       35 | nupdates:       123312 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:25 | bch(ms): 343.34 | smp(ms): 1.12 | fwd(ms): 110.09 | crit-fwd(ms): 4.29 | bwd(ms): 182.90 | optim(ms): 38.12 | loss:  184.96225 | train-TER: 86.18 | train-WER: 125.37 | dev-clean-loss:  140.52871 | dev-clean-TER: 186.85 | dev-clean-WER: 266.27 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 295.55
epoch:       36 | nupdates:       126880 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:19 | bch(ms): 341.67 | smp(ms): 1.09 | fwd(ms): 109.80 | crit-fwd(ms): 4.10 | bwd(ms): 182.53 | optim(ms): 37.35 | loss:  183.54360 | train-TER: 89.06 | train-WER: 130.44 | dev-clean-loss:  141.00610 | dev-clean-TER: 150.79 | dev-clean-WER: 220.05 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 095 | hrs:  100.57 | thrpt(sec/sec): 297.00
epoch:       37 | nupdates:       130448 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:15 | bch(ms): 340.58 | smp(ms): 1.06 | fwd(ms): 109.73 | crit-fwd(ms): 4.09 | bwd(ms): 182.22 | optim(ms): 37.23 | loss:  182.29519 | train-TER: 90.36 | train-WER: 131.67 | dev-clean-loss:  141.83075 | dev-clean-TER: 136.19 | dev-clean-WER: 203.73 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.95
epoch:       38 | nupdates:       134016 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:16 | bch(ms): 341.00 | smp(ms): 1.06 | fwd(ms): 109.77 | crit-fwd(ms): 4.08 | bwd(ms): 182.06 | optim(ms): 37.17 | loss:  181.04652 | train-TER: 87.92 | train-WER: 128.16 | dev-clean-loss:  142.08505 | dev-clean-TER: 128.93 | dev-clean-WER: 188.03 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.58
epoch:       39 | nupdates:       137584 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:16 | bch(ms): 340.93 | smp(ms): 1.09 | fwd(ms): 109.79 | crit-fwd(ms): 4.07 | bwd(ms): 182.05 | optim(ms): 37.12 | loss:  179.60404 | train-TER: 87.42 | train-WER: 129.84 | dev-clean-loss:  142.89777 | dev-clean-TER: 145.29 | dev-clean-WER: 212.63 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.64
epoch:       40 | nupdates:       141152 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:12 | bch(ms): 339.75 | smp(ms): 1.13 | fwd(ms): 109.74 | crit-fwd(ms): 4.07 | bwd(ms): 182.02 | optim(ms): 37.13 | loss:  178.86776 | train-TER: 83.85 | train-WER: 120.03 | dev-clean-loss:  143.06155 | dev-clean-TER: 128.56 | dev-clean-WER: 194.99 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.68
epoch:       41 | nupdates:       144720 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:15 | bch(ms): 340.60 | smp(ms): 1.07 | fwd(ms): 109.79 | crit-fwd(ms): 4.07 | bwd(ms): 182.06 | optim(ms): 37.13 | loss:  177.45621 | train-TER: 86.16 | train-WER: 125.09 | dev-clean-loss:  144.39468 | dev-clean-TER: 142.47 | dev-clean-WER: 181.61 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 297.93
epoch:       42 | nupdates:       148288 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:13 | bch(ms): 340.01 | smp(ms): 1.04 | fwd(ms): 109.78 | crit-fwd(ms): 4.07 | bwd(ms): 182.02 | optim(ms): 37.11 | loss:  176.99047 | train-TER: 85.07 | train-WER: 124.87 | dev-clean-loss:  143.70708 | dev-clean-TER: 120.27 | dev-clean-WER: 176.39 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.45
epoch:       43 | nupdates:       151856 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:14 | bch(ms): 340.49 | smp(ms): 1.02 | fwd(ms): 109.78 | crit-fwd(ms): 4.07 | bwd(ms): 182.03 | optim(ms): 37.10 | loss:  175.50045 | train-TER: 85.83 | train-WER: 127.45 | dev-clean-loss:  144.58858 | dev-clean-TER: 166.61 | dev-clean-WER: 295.09 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 298.03
epoch:       44 | nupdates:       155424 | lr: 0.010000 | lrcriterion: 0.010000 | runtime: 00:20:19 | bch(ms): 341.83 | smp(ms): 1.09 | fwd(ms): 109.67 | crit-fwd(ms): 4.08 | bwd(ms): 181.94 | optim(ms): 37.17 | loss:  174.43614 | train-TER: 90.58 | train-WER: 138.35 | dev-clean-loss:  145.44580 | dev-clean-TER: 153.51 | dev-clean-WER: 227.93 | avg-isz: 1268 | avg-tsz: 043 | max-tsz: 093 | hrs:  100.57 | thrpt(sec/sec): 296.86

Seems that it doesn't work. I had trained transformer_ctc on 1 GPU successfully, but the transformer_s2s has never worked. Hope to help. THX. BTW, the successful transformer_ctc is followed by sota/2019/librispeech/train_am_transformer_ctc nothing changed.

tlikhomanenko commented 4 years ago

S2S is very sensitive and hard to train in general (to make it converge). I would still try the longer pre-train window then and also increase the warmup (at first just increase the pre-training window). Also make sure that saug is activated when you have at least 50% of train WER.

xiaosdawn commented 4 years ago

Thank @tlikhomanenko. I would try.