tumusudheer
commented
4 years ago I built wav2letter in another one of my ubuntu machine Ubuntu 18.04 cuda 10.1
After I typed this command:
./build/Train train --flagsfile tutorials/1-librispeech_clean/train.cfg
I'm getting the same issues as I mentioned. I get the console print as follows:
Memory Manager Stats
MemoryManager type: CachingMemoryManager
Number of allocated blocks:20
Size of free block pool (small):18
Size of free block pool (large):12
Total native mallocs:19
Total native frees:0And the the process ends, I don't get any errors and training process doesn't start at all.
My train.cfg looks as follows:
# Training config for Mini Librispeech
# Replace `[...]` with appropriate paths
--datadir=/home/user01/Self/Research/Work/facebook/data
--rundir=/data/user01/facebook/wav2letter/train_runs
--archdir=/data/user01/facebook/wav2letter/tutorials/1-librispeech_clean/
--train=lists/train-clean-100.lst
--valid=lists/dev-clean.lst
--input=flac
--arch=network.arch
--tokens=/home/user01/Self/Research/Work/facebook/data/am/tokens.txt
--lexicon=/home/user01/Self/Research/Work/facebook/data/am/lexicon.txt
--criterion=ctc
--lr=0.1
--maxgradnorm=1.0
--replabel=1
--surround=|
--onorm=target
--sqnorm=true
--mfsc=true
--filterbanks=40
--nthread=4
--batchsize=4
--runname=librispeech_clean_trainlogs
--iter=25
When I looked at my rundir, I saw the following files: subdirectory : librispeech_clean_trainlogs/ contains
total 30512
-rw-rw-r-- 1 user01 user01 4578 Jul 5 13:33 001_config
-rw-rw-r-- 1 user01 user01 428 Jul 5 13:33 001_perf
-rw-rw-r-- 1 user01 user01 460 Jul 5 13:33 001_log
-rw-rw-r-- 1 user01 user01 15610682 Jul 5 13:33 001_model_last.bin
-rw-rw-r-- 1 user01 user01 15610682 Jul 5 13:33 001_model_lists#dev-clean.lst.binContents of 001_perf are
# date time epoch nupdates lr lrcriterion runtime bch(ms) smp(ms) fwd(ms) crit-fwd(ms) bwd(ms) optim(ms) loss train-TER train-WER lists/dev-clean.lst-loss lists/dev-clean.lst-TER lists/dev-clean.lst-WER avg-isz avg-tsz max-tsz hrs thrpt(sec/sec)
2020-07-05 13:33:57 1 26 0.100000 0.000000 00:00:04 173.38 4.70 97.43 38.25 24.85 20.42 56.91331 99.95 100.00 40.77565 100.00 100.00 1301 217 299 0.38 300.30
With the following command
./build/Train train --flagsfile tutorials/1-librispeech_clean/train.cfg --logtostderr=1 --minloglevel=0
I get the following output
I0705 15:39:30.304139 8436 Train.cpp:59] Reading flags from file tutorials/1-librispeech_clean/train.cfg
I0705 15:39:30.309223 8436 Train.cpp:151] Gflags after parsing
--flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --adambeta1=0.90000000000000002; --adambeta2=0.999; --am=; --am_decoder_tr_dropout=0; --am_decoder_tr_layerdrop=0; --am_decoder_tr_layers=1; --arch=network.arch; --archdir=/data/user01/Self/facebook/wav2letter/tutorials/1-librispeech_clean/; --attention=content; --attentionthreshold=2147483647; --attnWindow=no; --attnconvchannel=0; --attnconvkernel=0; --attndim=0; --batchsize=4; --beamsize=2500; --beamsizetoken=250000; --beamthreshold=25; --blobdata=false; --channels=1; --criterion=ctc; --critoptim=sgd; --datadir=/home/user01/Suki/Research/Work/Self/facebook/data; --dataorder=input; --decoderattnround=1; --decoderdropout=0; --decoderrnnlayer=1; --decodertype=wrd; --devwin=0; --emission_dir=; --emission_queue_size=3000; --enable_distributed=false; --encoderdim=0; --eosscore=0; --eostoken=false; --everstoredb=false; --fftcachesize=1; --filterbanks=40; --flagsfile=tutorials/1-librispeech_clean/train.cfg; --framesizems=25; --framestridems=10; --gamma=1; --gumbeltemperature=1; --input=flac; --inputbinsize=100; --inputfeeding=false; --isbeamdump=false; --iter=25; --itersave=false; --labelsmooth=0; --leftWindowSize=50; --lexicon=/home/user01/Suki/Research/Work/Self/facebook/data/am/lexicon.txt; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lm_memory=5000; --lm_vocab=; --lmtype=kenlm; --lmweight=0; --localnrmlleftctx=0; --localnrmlrightctx=0; --logadd=false; --lr=0.10000000000000001; --lr_decay=9223372036854775807; --lr_decay_step=9223372036854775807; --lrcosine=false; --lrcrit=0; --max_devices_per_node=8; --maxdecoderoutputlen=200; --maxgradnorm=1; --maxisz=9223372036854775807; --maxload=-1; --maxrate=10; --maxsil=50; --maxtsz=9223372036854775807; --maxword=-1; --melfloor=1; --memstepsize=10485760; --mfcc=false; --mfcccoeffs=13; --mfsc=true; --minisz=0; --minrate=3; --minsil=0; --mintsz=0; --momentum=0; --netoptim=sgd; --noresample=false; --nthread=4; --nthread_decoder=1; --nthread_decoder_am_forward=1; --numattnhead=8; --onorm=target; --optimepsilon=1e-08; --optimrho=0.90000000000000002; --outputbinsize=5; --pctteacherforcing=100; --pcttraineval=100; --pow=false; --pretrainWindow=0; --replabel=1; --reportiters=0; --rightWindowSize=50; --rndv_filepath=; --rundir=/data/user01/Self/facebook/wav2letter/train_runs; --runname=librispeech_clean_trainlogs; --samplerate=16000; --sampletarget=0; --samplingstrategy=rand; --saug_fmaskf=27; --saug_fmaskn=2; --saug_start_update=-1; --saug_tmaskn=2; --saug_tmaskp=1; --saug_tmaskt=100; --sclite=; --seed=0; --show=false; --showletters=false; --silscore=0; --smearing=none; --smoothingtemperature=1; --softwoffset=10; --softwrate=5; --softwstd=5; --sqnorm=true; --stepsize=9223372036854775807; --surround=|; --tag=; --target=tkn; --test=; --tokens=/home/user01/Suki/Research/Work/Self/facebook/data/am/tokens.txt; --tokensdir=; --train=lists/train-clean-100.lst; --trainWithWindow=false; --transdiag=0; --unkscore=-inf; --use_memcache=false; --uselexicon=true; --usewordpiece=false; --valid=lists/dev-clean.lst; --warmup=1; --weightdecay=0; --wordscore=0; --wordseparator=|; --world_rank=0; --world_size=1; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=;
I0705 15:39:30.309458 8436 Train.cpp:152] Experiment path: /data/user01/Self/facebook/wav2letter/train_runs/librispeech_clean_trainlogs
I0705 15:39:30.309460 8436 Train.cpp:153] Experiment runidx: 1
I0705 15:39:30.309729 8436 Train.cpp:199] Number of classes (network): 30
I0705 15:39:30.341761 8436 Train.cpp:206] Number of words: 34795
I0705 15:39:30.350679 8436 Train.cpp:220] Loading architecture file from /data/user01/Self/facebook/wav2letter/tutorials/1-librispeech_clean/network.arch
I0705 15:39:30.885213 8436 Train.cpp:252] [Network] Sequential [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> output]
(0): View (-1 1 40 0)
(1): Conv2D (40->256, 8x1, 2,1, SAME,SAME, 1, 1) (with bias)
(2): ReLU
(3): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(4): ReLU
(5): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(6): ReLU
(7): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(8): ReLU
(9): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(10): ReLU
(11): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(12): ReLU
(13): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(14): ReLU
(15): Conv2D (256->256, 8x1, 1,1, SAME,SAME, 1, 1) (with bias)
(16): ReLU
(17): Reorder (2,0,3,1)
(18): Linear (256->512) (with bias)
(19): ReLU
(20): Linear (512->30) (with bias)
I0705 15:39:30.885252 8436 Train.cpp:253] [Network Params: 3900958]
I0705 15:39:30.885257 8436 Train.cpp:254] [Criterion] ConnectionistTemporalClassificationCriterion
I0705 15:39:30.885267 8436 Train.cpp:262] [Network Optimizer] SGD
I0705 15:39:30.885269 8436 Train.cpp:263] [Criterion Optimizer] SGD
I0705 15:39:31.180352 8436 W2lListFilesDataset.cpp:141] 28539 files found.
I0705 15:39:31.180716 8436 Utils.cpp:102] Filtered 0/28539 samples
I0705 15:39:31.182535 8436 W2lListFilesDataset.cpp:62] Total batches (i.e. iters): 7135
I0705 15:39:31.215492 8436 W2lListFilesDataset.cpp:141] 2703 files found.
I0705 15:39:31.215519 8436 Utils.cpp:102] Filtered 0/2703 samples
I0705 15:39:31.215649 8436 W2lListFilesDataset.cpp:62] Total batches (i.e. iters): 676
I0705 15:39:31.216138 8436 Train.cpp:564] Shuffling trainset
I0705 15:39:31.216651 8436 Train.cpp:571] Epoch 1 started!
I0705 15:39:45.632812 8436 Train.cpp:345] epoch: 1 | nupdates: 26 | lr: 0.100000 | lrcriterion: 0.000000 | runtime: 00:00:05 | bch(ms): 204.63 | smp(ms): 5.18 | fwd(ms): 127.09 | crit-fwd(ms): 36.39 | bwd(ms): 25.18 | optim(ms): 20.58 | loss: 56.91273 | train-TER: 99.95 | train-WER: 100.00 | lists/dev-clean.lst-loss: 40.77376 | lists/dev-clean.lst-TER: 100.00 | lists/dev-clean.lst-WER: 100.00 | avg-isz: 1301 | avg-tsz: 217 | max-tsz: 299 | hrs: 0.38 | thrpt(sec/sec): 254.44
Memory Manager Stats
MemoryManager type: CachingMemoryManager
Number of allocated blocks:20
Size of free block pool (small):18
Size of free block pool (large):12
Total native mallocs:19
Total native frees:0
I0705 15:39:45.666921 8436 Train.cpp:747] Finished trainingAlso I tried to run SOTA 2019 librispeech train_am_tds_s2s.cfg
I prepared the data as using this command python3 prepare.py --dst with dst as some data directory.
And my train.cfg is as follows:
# Replace `[...]`, `[MODEL_DST]`, `[DATA_DST]`, with appropriate paths
--runname=am_tds_s2s_librispeech
--rundir=/data/user01/Self/facebook/wav2letter/data/sota/run/
--archdir=/data/user01/Self/facebook/wav2letter/recipes/models/sota/2019/
--arch=am_arch/am_tds_s2s.arch
--tokensdir=/data/user01/Self/facebook/wav2letter/data/sota/models/am
--tokens=librispeech-train-all-unigram-10000.tokens
--lexicon=/data/user01/Self/facebook/wav2letter/data/sota/models/am/librispeech-train+dev-unigram-10000-nbest10.lexicon
--train=/data/user01/Self/facebook/wav2letter/data/sota/data/lists/train-clean-100.lst,/data/user01/Self/facebook/wav2letter/data/sota/data/lists/train-clean-360.lst,/data/user01/Self/facebook/wav2letter/data/sota/data/lists/train-other-500.lst
--valid=dev-clean:/data/user01/Self/facebook/wav2letter/data/sota/data/lists/dev-clean.lst,dev-other:/data/user01/Self/facebook/wav2letter/data/sota/data/lists/dev-other.lst
--batchsize=2
--lr=0.06
--lrcrit=0.06
--momentum=0.5
--maxgradnorm=15
--mfsc=true
--nthread=6
--criterion=seq2seq
--maxdecoderoutputlen=120
--labelsmooth=0.05
--dataorder=output_spiral
--inputbinsize=25
--attnWindow=softPretrain
--softwstd=4
--trainWithWindow=true
--pretrainWindow=3
--attention=keyvalue
--encoderdim=512
--memstepsize=8338608
--eostoken=true
--pcttraineval=1
--pctteacherforcing=99
--listdata=true
--usewordpiece=true
--wordseparator=_
--target=ltr
--filterbanks=80
--stepsize=150
--gamma=0.5
--sampletarget=0.01
--enable_distributed=true
--iter=600
--framesizems=30
--framestridems=10
--decoderdropout=0.1
--decoderattnround=2
--decoderrnnlayer=3
--seed=2
After 2 epochs, the training is getting stopped and seeing the following the prints
Memory Manager Stats
MemoryManager type: CachingMemoryManager
Number of allocated blocks:455
Size of free block pool (small):111
Size of free block pool (large):138
Total native mallocs:242
Total native frees:0
I0705 17:22:56.667855 17991 Train.cpp:747] Finished training
Training command
./build/Train train --flagsfile recipes/models/sota/2019/librispeech/train_am_tds_s2s.cfg --logtostderr=1 --minloglevel=0
Complete log of SOTA training is attached here
sota_tds_s2s.log
tlikhomanenko
rajeevbaalwan
mironnn
iggygeek
Hi,
I'm following instructions as given here: https://github.com/facebookresearch/wav2letter/tree/master/tutorials/1-librispeech_clean
After processing the data: I tried running the following command:
build/Train train --flagsfile tutorials/1-librispeech_clean/train.cfgMy train.cfg is as follows:
I'm getting the following output and the training is not even getting started:
I tried with
lr=0.0001, But still the same issue.Also tried to run training with command params using the given command:
build/Train train --tokensdir=/data/Self/placeholder/facebook/data/am --tokens=tokens.txt --runname=librispeech -rundir=/data/Self/placeholder/facebook/data/run --archdir=tutorials/1-librispeech_clean/ --arch=network.arch --lexicon=/data/Self/placeholder/facebook/data/am/lexicon.txt --train=/data/Self/placeholder/facebook/data/lists/train-clean-100.lst --valid=/data/Self/placeholder/facebook/data/lists/dev-clean.lst --criterion=ctc --lr=0.0001 --maxgradnorm=1.0 --surround=| --sqnorm=true --mfsc=true --filterbanks=40 --batchsize=4 --iter=25I'm getting the following error:
Ran
ldd build/Trainshows the following:Am I missing something, or doing anything wrong ? Please suggest how to resolve the issues?
Thank you