Closed rezame closed 6 years ago
What is the dimension of the input layer in the file that defines the architecture of the network? For a quick and dirty solution, delete the entire experiment folder and start the training again?
my dimension of the input layer is 39. delete the entire experiment folder and It runs successfully thanks
I want to run LSTM-CTC by MFCC feature. I set input_feat_dim to 13 and --add-deltas false but it Error:
EPOCH 1 RUNNING ... steps/train_ctc_parallel.sh: line 132: 8383 Aborted (core dumped) $train_tool --report-step=$report_step --num-sequence=$num_sequence --frame-limit=$frame_num_limit --learn-rate=$learn_rate --momentum=$momentum --verbose=$verbose "$feats_tr" "$labels_tr" $dir/nnet/nnet.iter$[iter-1] $dir/nnet/nnet.iter${iter} 1>&$dir/log/tr.iter$iter.log Removing features tmpdir /tmp/tmp.Xm0jMPjW0k @ dsplab-All-Series
and I set input_feat_dim to 39 and --add-deltas true but the same error. what wrong? how can fix it?
log is ;
train-ctc-parallel --report-step=1000 --num-sequence=20 --frame-limit=25000 --learn-rate=0.00004 --momentum=0.9 --verbose=1 'ark,s,cs:copy-feats scp:exp/train_char_l2_c150_mfcc/train_local.scp ark:- |' 'ark:gunzip -c exp/train_char_l2_c150_mfcc/labels.tr.gz|' exp/train_char_l2_c150_mfcc/nnet/nnet.iter0 exp/train_char_l2_c150_mfcc/nnet/nnet.iter1 LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:262) Selecting from 1 GPUs LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:277) cudaSetDevice(0): GeForce GTX 750 Ti free:1744M, used:254M, total:1999M, free/total:0.872722 LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:310) Selected device: 0 (automatically) LOG (train-ctc-parallel:FinalizeActiveGpu():cuda-device.cc:194) The active GPU is [0]: GeForce GTX 750 Ti free:1715M, used:283M, total:1999M, free/total:0.858215 version 5.0 LOG (train-ctc-parallel:PrintMemoryUsage():cuda-device.cc:334) Memory used: 0 bytes. LOG (train-ctc-parallel:DisableCaching():cuda-device.cc:731) Disabling caching of GPU memory. LOG (train-ctc-parallel:SetUpdateAlgorithm():net.cc:483) Selecting SGD with momentum as optimization algorithm. LOG (train-ctc-parallel:SetTrainMode():net.cc:408) Setting TrainMode for layer 0 LOG (train-ctc-parallel:SetTrainMode():net.cc:408) Setting TrainMode for layer 1 copy-feats scp:exp/train_char_l2_c150_mfcc/train_local.scp ark:- LOG (train-ctc-parallel:main():train-ctc-parallel.cc:133) TRAINING STARTED KALDI_ASSERT: at train-ctc-parallel:CopyFromVec:vector.cc:199, failed: Dim() == v.Dim() Stack trace is: eesen::KaldiGetStackTrace[abi:cxx11]() eesen::KaldiAssertFailure_(char const*, char const*, int, char const*) eesen::VectorBase<float>::CopyFromVec(eesen::VectorBase<float> const&) train-ctc-parallel(main+0x13f8) [0x434bbc] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff5f49aa830] train-ctc-parallel(_start+0x29) [0x432129]
and my feat to dim is:
feat-to-dim scp:exp/train_char_l2_c150_mfcc/train.scp - 13
best regards