fmetze
commented
5 years ago What is the dimension of the input layer in the file that defines the architecture of the network? For a quick and dirty solution, delete the entire experiment folder and start the training again?
rezame
I want to run LSTM-CTC by MFCC feature. I set input_feat_dim to 13 and --add-deltas false but it Error:
EPOCH 1 RUNNING ... steps/train_ctc_parallel.sh: line 132: 8383 Aborted (core dumped) $train_tool --report-step=$report_step --num-sequence=$num_sequence --frame-limit=$frame_num_limit --learn-rate=$learn_rate --momentum=$momentum --verbose=$verbose "$feats_tr" "$labels_tr" $dir/nnet/nnet.iter$[iter-1] $dir/nnet/nnet.iter${iter} 1>&$dir/log/tr.iter$iter.log Removing features tmpdir /tmp/tmp.Xm0jMPjW0k @ dsplab-All-Seriesand I set input_feat_dim to 39 and --add-deltas true but the same error. what wrong? how can fix it?
log is ;
train-ctc-parallel --report-step=1000 --num-sequence=20 --frame-limit=25000 --learn-rate=0.00004 --momentum=0.9 --verbose=1 'ark,s,cs:copy-feats scp:exp/train_char_l2_c150_mfcc/train_local.scp ark:- |' 'ark:gunzip -c exp/train_char_l2_c150_mfcc/labels.tr.gz|' exp/train_char_l2_c150_mfcc/nnet/nnet.iter0 exp/train_char_l2_c150_mfcc/nnet/nnet.iter1 LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:262) Selecting from 1 GPUs LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:277) cudaSetDevice(0): GeForce GTX 750 Ti free:1744M, used:254M, total:1999M, free/total:0.872722 LOG (train-ctc-parallel:SelectGpuIdAuto():cuda-device.cc:310) Selected device: 0 (automatically) LOG (train-ctc-parallel:FinalizeActiveGpu():cuda-device.cc:194) The active GPU is [0]: GeForce GTX 750 Ti free:1715M, used:283M, total:1999M, free/total:0.858215 version 5.0 LOG (train-ctc-parallel:PrintMemoryUsage():cuda-device.cc:334) Memory used: 0 bytes. LOG (train-ctc-parallel:DisableCaching():cuda-device.cc:731) Disabling caching of GPU memory. LOG (train-ctc-parallel:SetUpdateAlgorithm():net.cc:483) Selecting SGD with momentum as optimization algorithm. LOG (train-ctc-parallel:SetTrainMode():net.cc:408) Setting TrainMode for layer 0 LOG (train-ctc-parallel:SetTrainMode():net.cc:408) Setting TrainMode for layer 1 copy-feats scp:exp/train_char_l2_c150_mfcc/train_local.scp ark:- LOG (train-ctc-parallel:main():train-ctc-parallel.cc:133) TRAINING STARTED KALDI_ASSERT: at train-ctc-parallel:CopyFromVec:vector.cc:199, failed: Dim() == v.Dim() Stack trace is: eesen::KaldiGetStackTrace[abi:cxx11]() eesen::KaldiAssertFailure_(char const*, char const*, int, char const*) eesen::VectorBase<float>::CopyFromVec(eesen::VectorBase<float> const&) train-ctc-parallel(main+0x13f8) [0x434bbc] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ff5f49aa830] train-ctc-parallel(_start+0x29) [0x432129]and my feat to dim is:
feat-to-dim scp:exp/train_char_l2_c150_mfcc/train.scp - 13best regards