search

mravanelli / pytorch-kaldi

pytorch-kaldi is a project for developing state-of-the-art DNN/RNN hybrid speech recognition systems. The DNN part is managed by pytorch, while feature extraction, label computation, and decoding are performed with the kaldi toolkit.
2.37k stars 446 forks source link

Error in the decoding #218

Closed mnabihali closed 4 years ago

mnabihali commented 4 years ago

I was run a TIMIT mfcc exp. The WER has not appeared I go to the log file this error is found

kaldi_decoding_scripts//decode_dnn.sh: line 87: 15110 Segmentation fault (core dumped) latgen-faster-mapped$thread_string --min-active=$min_active --max-active=$max_active --max-mem=$max_mem --beam=$beam --lattice-beam=$latbeam --acoustic-scale=$acwt --allow-partial=true --word-symbol-table=$graphdir/words.txt $alidir/final.mdl $graphdir/HCLG.fst "$finalfeats" "ark:|gzip -c > $dir/lat.$JOB.gz" &>$dir/log/decode.$JOB.log run.pl: job failed, log is in /hardmnt/moissan0/home/mnabih/piccadilly0/home/mnabih/pytorch-kaldi/exp/TIMIT_MLP_basic/decode_TIMIT_test_out_dnn1/scoring/log/best_path_basic.1.1.log

TParcollet commented 4 years ago

Hi,

Could you please give us the log on /hardmnt/moissan0/home/mnabih/piccadilly0/home/mnabih/Documents/pytorch-kaldi/exp/TIMIT_MLP_2/decode_TIMIT_test_out_dnn1/scoring/log/best_path.1.1.log ?

Thanks

mnabihali commented 4 years ago

This is the problem run.pl: job failed, log is in /hardmnt/moissan0/home/mnabih/piccadilly0/home/mnabih/Documents/pytorch-kaldi/exp/TIMIT_MLP_2/decode_TIMIT_test_out_dnn1/scoring/log/best_path.1.1.log The log latgen-faster-mapped --min-active=200 --max-active=7000 --max-mem=50000000 --beam=13.0 --lattice-beam=8.0 --acoustic-scale=0.2 --allow-partial=true --word-symbol-table=/home/mnabih/kaldi/egs/timit/s5/exp/tri3/graph/words.txt /home/mnabih/kaldi/egs/timit/s5/exp/tri3_ali_dev/final.mdl /home/mnabih/kaldi/egs/timit/s5/exp/tri3/graph/HCLG.fst 'ark,s,cs: cat /hardmnt/moissan0/home/mnabih/piccadilly0/home/mnabih/Documents/pytorch-kaldi/exp/TIMIT_MLP_2/exp_files/forward_TIMIT_test_ep23_ck0_out_dnn1_to_decode.ark |' 'ark:|gzip -c > /hardmnt/moissan0/home/mnabih/piccadilly0/home/mnabih/Documents/pytorch-kaldi/exp/TIMIT_MLP_2/decode_TIMIT_test_out_dnn1/lat.1.gz' ERROR (latgen-faster-mapped[5.5.600~2-f1969]:DecodableMatrixScaledMapped():decoder/decodable-matrix.h:42) DecodableMatrixScaledMapped: mismatch, matrix has 1944 rows but transition-model has 1920 pdf-ids.

[ Stack-Trace: ] /hardmnt/moissan0/home/mnabih/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0x8b7) [0x7f08d9f0dd5d] latgen-faster-mapped(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x11) [0x421d01] latgen-faster-mapped(kaldi::DecodableMatrixScaledMapped::DecodableMatrixScaledMapped(kaldi::TransitionModel const&, kaldi::Matrix const&, float)+0xbd) [0x429b71] latgen-faster-mapped(main+0x8fd) [0x41f1aa] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f08d2274545] latgen-faster-mapped() [0x41e7e9]

WARNING (latgen-faster-mapped[5.5.600~2-f1969]:Close():kaldi-io.cc:515) Pipe cat /hardmnt/moissan0/home/mnabih/piccadilly0/home/mnabih/Documents/pytorch-kaldi/exp/TIMIT_MLP_2/exp_files/forward_TIMIT_test_ep23_ck0_out_dnn1_to_decode.ark | had nonzero return status 13 kaldi::KaldiFatalError

TParcollet commented 4 years ago

"DecodableMatrixScaledMapped: mismatch, matrix has 1944 rows but transition-model has 1920 pdf-ids." -> I believe there is something wrong in either your training or configuration file.

mnabihali commented 4 years ago

[cfg_proto] cfg_proto = proto/global.proto cfg_proto_chunk = proto/global_chunk.proto

[exp] cmd = run_nn_script = run_nn out_folder = exp/TIMIT_MLP_3 seed = 1234 use_cuda = True multi_gpu = False save_gpumem = False n_epochs_tr = 5

[dataset1] data_name = TIMIT_tr fea = fea_name=mfcc fea_lst=/home/mnabih/kaldi/egs/timit/s5/data/train/feats.scp fea_opts=apply-cmvn --utt2spk=ark:/home/mnabih/kaldi/egs/timit/s5/data/train/utt2spk ark:/home/mnabih/kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- | cw_left=5 cw_right=5

lab = lab_name=lab_cd lab_folder=/home/mnabih/kaldi/egs/timit/s5/exp/tri3_ali lab_opts=ali-to-pdf lab_count_file=auto lab_data_folder=/home/mnabih/kaldi/egs/timit/s5/data/train/ lab_graph=/home/mnabih/kaldi/egs/timit/s5/exp/tri3/graph

n_chunks = 5

[dataset2] data_name = TIMIT_dev fea = fea_name=mfcc fea_lst=/home/mnabih/kaldi/egs/timit/s5/data/dev/feats.scp fea_opts=apply-cmvn --utt2spk=ark:/home/mnabih/kaldi/egs/timit/s5/data/dev/utt2spk ark:/home/mnabih/kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- | cw_left=5 cw_right=5

lab = lab_name=lab_cd lab_folder=/home/mnabih/kaldi/egs/timit/s5/exp/tri3_ali_dev lab_opts=ali-to-pdf lab_count_file=auto lab_data_folder=/home/mnabih/kaldi/egs/timit/s5/data/dev/ lab_graph=/home/mnabih/kaldi/egs/timit/s5/exp/tri3/graph

n_chunks = 1

[dataset3] data_name = TIMIT_test fea = fea_name=mfcc fea_lst=/home/mnabih/kaldi/egs/timit/s5/data/test/feats.scp fea_opts=apply-cmvn --utt2spk=ark:/home/mnabih/kaldi/egs/timit/s5/data/test/utt2spk ark:/home/mnabih/kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- | cw_left=5 cw_right=5

lab = lab_name=lab_cd lab_folder=/home/mnabih/kaldi/egs/timit/s5/exp/tri3_ali_test lab_opts=ali-to-pdf lab_count_file=auto lab_data_folder=/home/mnabih/kaldi/egs/timit/s5/data/test/ lab_graph=/home/mnabih/kaldi/egs/timit/s5/exp/tri3/graph

n_chunks = 1

[data_use] train_with = TIMIT_tr valid_with = TIMIT_dev forward_with = TIMIT_test

[batches] batch_size_train = 128 max_seq_length_train = 1000 increase_seq_length_train = False start_seq_len_train = 100 multply_factor_seq_len_train = 2 batch_size_valid = 128 max_seq_length_valid = 1000

[architecture1] arch_name = MLP_layers1 arch_proto = proto/MLP.proto arch_library = neural_networks arch_class = MLP arch_pretrain_file = none arch_freeze = False arch_seq_model = False dnn_lay = 1024,1024,1024,1024,N_out_lab_cd dnn_drop = 0.15,0.15,0.15,0.15,0.0 dnn_use_laynorm_inp = False dnn_use_batchnorm_inp = False dnn_use_batchnorm = True,True,True,True,False dnn_use_laynorm = False,False,False,False,False dnn_act = relu,relu,relu,relu,softmax arch_lr = 0.08 arch_halving_factor = 0.5 arch_improvement_threshold = 0.001 arch_opt = sgd opt_momentum = 0.0 opt_weight_decay = 0.0 opt_dampening = 0.0 opt_nesterov = False

[model] model_proto = proto/model.proto model = out_dnn1=compute(MLP_layers1,mfcc) loss_final=cost_nll(out_dnn1,lab_cd) err_final=cost_err(out_dnn1,lab_cd)

[forward] forward_out = out_dnn1 normalize_posteriors = True normalize_with_counts_from = lab_cd save_out_file = True require_decoding = True

[decoding] decoding_script_folder = kaldi_decoding_scripts/ decoding_script = decode_dnn.sh decoding_proto = proto/decoding.proto min_active = 200 max_active = 7000 max_mem = 50000000 beam = 13.0 latbeam = 8.0 acwt = 0.2 max_arcs = -1 skip_scoring = false scoring_script = local/score.sh scoring_opts = "--min-lmwt 1 --max-lmwt 10" norm_vars = False

mnabihali commented 4 years ago

This is my cfg file,

TParcollet commented 4 years ago

@mravanelli any ideas? Could you try to build your graph again ?

mnabihali commented 4 years ago

Do you mean run the kaldi timit recipe from scratch

TParcollet commented 4 years ago

You can just mkgraph to rebuild the decoding graph, but maybe running the recipe again could help

mnabihali commented 4 years ago

This happens when I run mkgraph Note: the --mono, --left-biphone and --quinphone options are now deprecated and will be ignored.

On Wed, Mar 25, 2020 at 7:14 PM Parcollet Titouan notifications@github.com wrote:

You can just mkgraph to rebuild the decoding graph, but maybe running the recipe again could help

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mravanelli/pytorch-kaldi/issues/218#issuecomment-604002326, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANWBGBF5BPUQXNJKKHZZ2CTRJJCZHANCNFSM4LTB6VGQ .

-- Mohamed Nabih Ali *Assistant *Lecturer Faculty of Computers and IT Egyptian E-Learning University Ain Shams Center Mail: mohmed.nabih@gmail.com mohmed.nabih@gmail.com Mobile: +201285659213

Work: 02-33318417

TParcollet commented 4 years ago

Have you been able to solve your issue ?

mnabihali commented 4 years ago

Yes, thanks a lot

On Fri, May 29, 2020 at 2:44 PM Parcollet Titouan notifications@github.com wrote:

Have you been able to solve your issue ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mravanelli/pytorch-kaldi/issues/218#issuecomment-635952294, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANWBGBGD5T244BHPAY3GCK3RT6U3JANCNFSM4LTB6VGQ .

-- Mohamed Nabih Ali *Assistant *Lecturer Faculty of Computers and IT Egyptian E-Learning University Ain Shams Center Mail: mohmed.nabih@gmail.com mohmed.nabih@gmail.com Mobile: +201285659213

Work: 02-33318417