librispeech rnnlm training

[Sang-Hoon-Pakr]

when i follow the example script from librispeech for rnnlm i met the issue how can i fix the this issue? ...

compute_prob.0.log

rnnlm-get-egs --bos-symbol=200005 --eos-symbol=200006 --brk-symbol=200007 --vocab-size=200008 exp/rnnlm_lstm_1a/text/dev.txt ark:- | rnnlm-compute-prob exp/rnnlm_lstm_1a/0.raw "rnnlm-get-word-embedding exp/rnnlm_lstm_1a/word_feats.txt exp/rnnlm_lstm_1a/feat_embedding.0.mat -|" ark:-

Started at Tue Aug 22 18:00:42 KST 2023

# rnnlm-compute-prob exp/rnnlm_lstm_1a/0.raw 'rnnlm-get-word-embedding exp/rnnlm_lstm_1a/word_feats.txt exp/rnnlm_lstm_1a/feat_embedding.0.mat -|' ark:- LOG (rnnlm-compute-prob[5.5.1074~1-71f3]:SelectGpuId():cu-device.cc:168) Manually selected to compute on CPU. rnnlm-get-egs --bos-symbol=200005 --eos-symbol=200006 --brk-symbol=200007 --vocab-size=200008 exp/rnnlm_lstm_1a/text/dev.txt ark:- rnnlm-get-word-embedding exp/rnnlm_lstm_1a/word_feats.txt exp/rnnlm_lstm_1a/feat_embedding.0.mat - ERROR (rnnlm-get-word-embedding[5.5.1074~1-71f3]:ReadSparseWordFeatures():rnnlm-utils.cc:41) The word-indexes are expected to be in order 0, 1, 2, ...

[ Stack-Trace: ] .....kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0xb42) [0x7fcd12f7d6e2] ......kaldi/src/lib/libkaldi-rnnlm.so(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x7fcd13684765] ....kaldi/src/lib/libkaldi-rnnlm.so(kaldi::rnnlm::ReadSparseWordFeatures(std::istream&, int, kaldi::SparseMatrix*)+0x507) [0x7fcd13696dd5] rnnlm-get-word-embedding(main+0x1ce) [0x56461f807558] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fcd12606b97] rnnlm-get-word-embedding(_start+0x2a) [0x56461f8072aa]

kaldi::KaldiFatalError ERROR (rnnlm-compute-prob[5.5.1074~1-71f3]:Read():kaldi-matrix.cc:1620) Failed to read matrix from stream. : Expected "[", got EOF File position at start is -1, currently -1

[ Stack-Trace: ] /home3/work/kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0xb42) [0x7f8369d296e2] rnnlm-compute-prob(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x55eaead305d5] ...kaldi/src/lib/libkaldi-matrix.so(kaldi::Matrix::Read(std::istream&, bool, bool)+0x1f60) [0x7f8368ccf5a0] ...kaldi/src/lib/libkaldi-cudamatrix.so(kaldi::CuMatrix::Read(std::istream&, bool)+0x48) [0x7f836a1d7096] rnnlm-compute-prob(main+0x590) [0x55eaead2f75a] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f8369193b97] rnnlm-compute-prob(_start+0x2a) [0x55eaead2f0ea]

WARNING (rnnlm-compute-prob[5.5.1074~1-71f3]:Close():kaldi-io.cc:515) Pipe rnnlm-get-word-embedding exp/rnnlm_lstm_1a/word_feats.txt exp/rnnlm_lstm_1a/feat_embedding.0.mat -| had nonzero return status 65280 kaldi::KaldiFatalError

Accounting: time=2 threads=1

Ended (code 255) at Tue Aug 22 18:00:44 KST 2023, elapsed time 2 seconds

---

train.0.1.log

rnnlm-train --rnnlm.max-param-change=0.5 --rnnlm.l2_regularize_factor=1 --embedding.max-param-change=0.5 --embedding.learning-rate=0.0001 --embedding.l2_regularize=0.005 --read-sparse-word-features=exp/rnnlm_lstm_1a/word_feats.txt --use-gpu=wait --rnnlm.backstitch-training-scale=0.0 --rnnlm.backstitch-training-interval=1 --embedding.backstitch-training-scale=0.0 --embedding.backstitch-training-interval=1 "--read-rnnlm=nnet3-copy --learning-rate=0.001 exp/rnnlm_lstm_1a/0.raw -|" --write-rnnlm=exp/rnnlm_lstm_1a/1.raw --read-embedding=exp/rnnlm_lstm_1a/feat_embedding.0.mat --write-embedding=exp/rnnlm_lstm_1a/feat_embedding.1.mat "ark,bg:cat exp/rnnlm_lstm_1a/text/1.txt | rnnlm-get-egs --chunk-length=32 --srand=0 --vocab-size=200008 --bos-symbol=200005 --eos-symbol=200006 --brk-symbol=200007 --num-samples=512 --sample-group-size=2 --num-threads=10 exp/rnnlm_lstm_1a/sampling.lm - ark:- |"

Started at Tue Aug 22 18:00:42 KST 2023

# rnnlm-train --rnnlm.max-param-change=0.5 --rnnlm.l2_regularize_factor=1 --embedding.max-param-change=0.5 --embedding.learning-rate=0.0001 --embedding.l2_regularize=0.005 --read-sparse-word-features=exp/rnnlm_lstm_1a/word_feats.txt --use-gpu=wait --rnnlm.backstitch-training-scale=0.0 --rnnlm.backstitch-training-interval=1 --embedding.backstitch-training-scale=0.0 --embedding.backstitch-training-interval=1 '--read-rnnlm=nnet3-copy --learning-rate=0.001 exp/rnnlm_lstm_1a/0.raw -|' --write-rnnlm=exp/rnnlm_lstm_1a/1.raw --read-embedding=exp/rnnlm_lstm_1a/feat_embedding.0.mat --write-embedding=exp/rnnlm_lstm_1a/feat_embedding.1.mat 'ark,bg:cat exp/rnnlm_lstm_1a/text/1.txt | rnnlm-get-egs --chunk-length=32 --srand=0 --vocab-size=200008 --bos-symbol=200005 --eos-symbol=200006 --brk-symbol=200007 --num-samples=512 --sample-group-size=2 --num-threads=10 exp/rnnlm_lstm_1a/sampling.lm - ark:- |' WARNING (rnnlm-train[5.5.1074~1-71f3]:SelectGpuId():cu-device.cc:229) Waited 0 seconds before creating CUDA context LOG (rnnlm-train[5.5.1074~1-71f3]:SelectGpuId():cu-device.cc:238) CUDA setup operating under Compute Exclusive Mode. LOG (rnnlm-train[5.5.1074~1-71f3]:FinalizeActiveGpu():cu-device.cc:338) The active GPU is [0]: NVIDIA Tesla ~~ free:31926M, used:584M, total:32510M, free/total:0.982021 version 7.0 nnet3-copy --learning-rate=0.001 exp/rnnlm_lstm_1a/0.raw - LOG (nnet3-copy[5.5.1074~1-71f3]:main():nnet3-copy.cc:123) Copied raw neural net from exp/rnnlm_lstm_1a/0.raw to - ERROR (rnnlm-train[5.5.1074~1-71f3]:ReadSparseWordFeatures():rnnlm-utils.cc:41) The word-indexes are expected to be in order 0, 1, 2, ...

[ Stack-Trace: ] ...kaldi/src/lib/libkaldi-base.so(kaldi::MessageLogger::LogMessage() const+0xb42) [0x7fa7844736e2] rnnlm-train(kaldi::MessageLogger::LogAndThrow::operator=(kaldi::MessageLogger const&)+0x21) [0x56502784eeff] ......kaldi/src/lib/libkaldi-rnnlm.so(kaldi::rnnlm::ReadSparseWordFeatures(std::istream&, int, kaldi::SparseMatrix*)+0x507) [0x7fa7866f6dd5] rnnlm-train(main+0xba8) [0x56502784de02] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fa7838ddb97] rnnlm-train(_start+0x2a) [0x56502784d17a]

kaldi::KaldiFatalError

Accounting: time=4 threads=1

Ended (code 255) at Tue Aug 22 18:00:46 KST 2023, elapsed time 4 seconds