megharangaswamy
commented
5 years ago As suggested in one of the thread, I kept only audio files with size less than 400KB for training. I edded up with below error
megha@megha-ERAZER-X4709-D-C519:/media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/1-librispeech_clean$ **/media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/build/Train train --flagsfile /media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/1-librispeech_clean/train.cfg**
I0425 11:30:18.199692 9725 Train.cpp:139] Gflags after parsing
--flagfile=; --fromenv=; --tryfromenv=; --undefok=; --tab_completion_columns=80; --tab_completion_word=; --help=false; --helpfull=false; --helpmatch=; --helpon=; --helppackage=false; --helpshort=false; --helpxml=false; --version=false; --am=; --arch=network.arch; --archdir=/media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/1-librispeech_clean/; --attention=content; --attnWindow=no; --batchsize=1; --beamscore=25; --beamsize=2500; --channels=1; --criterion=ctc; --datadir=/media/home/megha/5_wav2letter/WAV_2_LETTER/; --dataorder=input; --devwin=0; --emission_dir=; --enable_distributed=false; --encoderdim=0; --eostoken=false; --everstoredb=false; --fftcachesize=1; --filterbanks=40; --flagsfile=/media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/1-librispeech_clean/train.cfg; --forceendsil=false; --gamma=0.20000000000000001; --garbage=false; --input=flac; --inputbinsize=100; --inputfeeding=false; --iter=100; --itersave=false; --labelsmooth=0; --leftWindowSize=50; --lexicon=; --linlr=-1; --linlrcrit=-1; --linseg=0; --lm=; --lmtype=kenlm; --lmweight=1; --localnrmlleftctx=0; --localnrmlrightctx=0; --logadd=false; --lr=0.050000000000000003; --lrcrit=0.0060000000000000001; --maxdecoderoutputlen=200; --maxgradnorm=1; --maxisz=9223372036854775807; --maxload=-1; --maxrate=10; --maxsil=50; --maxtsz=9223372036854775807; --maxword=-1; --melfloor=1; --memstepsize=10485760; --mfcc=false; --mfcccoeffs=13; --mfsc=true; --minisz=0; --minrate=3; --minsil=0; --mintsz=0; --momentum=0.80000000000000004; --noresample=false; --nthread=1; --nthread_decoder=1; --onorm=target; --outputbinsize=5; --pctteacherforcing=100; --pcttraineval=100; --pow=false; --replabel=2; --reportiters=0; --rightWindowSize=50; --rndv_filepath=; --rundir=/media/home/megha/5_wav2letter/WAV_2_LETTER/; --runname=deutsche_Combined_clean_trainlogs; --samplerate=16000; --samplingstrategy=rand; --sclite=; --seed=0; --show=false; --showletters=false; --silweight=0; --skipoov=false; --smearing=none; --softwoffset=10; --softwrate=5; --softwstd=5; --sqnorm=true; --stepsize=1; --surround=|; --tag=; --target=tkn; --targettype=video; --test=; --tokens=wav2letter/tutorials/output/data/tokens.txt; --tokensdir=/media/home/megha/5_wav2letter/WAV_2_LETTER/; --train=wav2letter/tutorials/output/data/train-clean-100; --trainWithWindow=false; --transdiag=0; --unkweight=-inf; --valid=wav2letter/tutorials/output/data/dev-clean; --weightdecay=0; --wordscore=1; --world_rank=0; --world_size=1; --alsologtoemail=; --alsologtostderr=false; --colorlogtostderr=false; --drop_log_memory=true; --log_backtrace_at=; --log_dir=; --log_link=; --log_prefix=true; --logbuflevel=0; --logbufsecs=30; --logemaillevel=999; --logfile_mode=436; --logmailer=/bin/mail; --logtostderr=true; --max_log_size=1800; --minloglevel=0; --stderrthreshold=2; --stop_logging_if_full_disk=false; --symbolize_stacktrace=true; --v=0; --vmodule=;
I0425 11:30:18.199710 9725 Train.cpp:141] Experiment path: /media/home/megha/5_wav2letter/WAV_2_LETTER/deutsche_Combined_clean_trainlogs
I0425 11:30:18.199714 9725 Train.cpp:142] Experiment runidx: 1
I0425 11:30:18.200085 9725 Train.cpp:160] Number of classes (network) = 35
I0425 11:30:18.200101 9725 Train.cpp:171] Loading architecture file from /media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/1-librispeech_clean/network.arch
I0425 11:30:18.877243 9725 Train.cpp:191] [Network] Sequential [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> output]
(0): View (-1 1 40 0)
(1): Conv2D (40->256, 8x1, 2,1, SAME,SAME) (with bias)
(2): ReLU
(3): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(4): ReLU
(5): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(6): ReLU
(7): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(8): ReLU
(9): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(10): ReLU
(11): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(12): ReLU
(13): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(14): ReLU
(15): Conv2D (256->256, 8x1, 1,1, SAME,SAME) (with bias)
(16): ReLU
(17): Reorder (2,0,3,1)
(18): Linear (256->512) (with bias)
(19): ReLU
(20): Linear (512->35) (with bias)
I0425 11:30:18.877285 9725 Train.cpp:192] [Network Params: 3903523]
I0425 11:30:18.877290 9725 Train.cpp:193] [Criterion] ConnectionistTemporalClassificationCriterion
I0425 11:30:18.877542 9725 NumberedFilesLoader.cpp:29] Adding dataset /media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/output/data/train-clean-100 ...
I0425 11:30:18.877698 9725 NumberedFilesLoader.cpp:68] 364 files found.
I0425 11:30:18.881237 9725 Utils.cpp:102] Filtered 0/364 samples
I0425 11:30:18.881323 9725 W2lNumberedFilesDataset.cpp:57] Total batches (i.e. iters): 364
I0425 11:30:18.881399 9725 NumberedFilesLoader.cpp:29] Adding dataset /media/home/megha/5_wav2letter/WAV_2_LETTER/wav2letter/tutorials/output/data/dev-clean ...
I0425 11:30:18.881530 9725 NumberedFilesLoader.cpp:68] 5181 files found.
I0425 11:30:18.925501 9725 Utils.cpp:102] Filtered 0/5181 samples
I0425 11:30:18.926044 9725 W2lNumberedFilesDataset.cpp:57] Total batches (i.e. iters): 5181
I0425 11:30:19.020403 9725 Train.cpp:441] Shuffling trainset
I0425 11:30:19.020476 9725 Train.cpp:448] Epoch 1 started!
terminate called after throwing an instance of 'std::invalid_argument'
what(): CUDNN_STATUS_BAD_PARAM
*** Aborted at 1556184624 (unix time) try "date -d @1556184624" if you are using GNU date ***
PC: @ 0x7f004681e428 gsignal
*** SIGABRT (@0x3e8000025fd) received by PID 9725 (TID 0x7f00c6178800) from PID 9725; stack trace: ***
@ 0x7f004681e4b0 (unknown)
@ 0x7f004681e428 gsignal
@ 0x7f004682002a abort
@ 0x7f004738384d __gnu_cxx::__verbose_terminate_handler()
@ 0x7f00473816b6 (unknown)
@ 0x7f0047381701 std::terminate()
@ 0x7f0047381919 __cxa_throw
@ 0x60a17d fl::TensorDescriptor::TensorDescriptor()
@ 0x608927 fl::conv2d()
@ 0x5e9836 fl::Conv2D::forward()
@ 0x5f7b9f fl::UnaryModule::forward()
@ 0x5e8832 fl::Sequential::forward()
@ 0x45e228 _ZZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEERNS0_19FirstOrderOptimizerES9_biE4_clES2_S5_S7_S9_S9_biENKUllddE0_clEldd
@ 0x45ff06 _ZZ4mainENKUlSt10shared_ptrIN2fl6ModuleEES_IN3w2l17SequenceCriterionEES_INS3_10W2lDatasetEERNS0_19FirstOrderOptimizerES9_biE4_clES2_S5_S7_S9_S9_bi.constprop.8756
@ 0x418f72 main
@ 0x7f0046809830 __libc_start_main
@ 0x45bb19 _start
Aborted
jacobkahn
vineelpratap
Dear all, Kindly help me with this.
I am trying to train my model. I am getting stuck with a few errors. Can u please elaborate how you solved this problem. Below is my log on console. I get different errors in a random fashion. There are few suggestions on them how others encountered such problems, but nowhere the exact solution is mentioned on how to solve them. some times I get 1) what(): Error: compute_ctc_loss, stat = cuda memcpy or memset failed (detailed logs as below) 2) Error: compute_ctc_loss, stat = unknown error (detailed logs as below)
My train config file contains settings as below:
@jacobkahn kindly point out the mistake I have done. My GPU info is as below
I am happy to provide any information needed, Kindly help me with this issue. I really want to start training soon :(
Thanks, Megha