run.sh not working with CGN

[miguelvr]

I was trying to use this recipe to train an ASR model with the CGN corpus, but I'm getting an empty segments error in the cgn_data_prep.sh when I run the run.sh script.

I changed the CGN root to my own. Am I missing something?

I have the version 2.0.3 of the corpus, if that makes any difference.

[laurensw75]

Without any additional info, I cannot say what went wrong. If the segments file is indeed empty, then this suggest process_flist.pl did not run correctly. Is the flist file occupied?

Perhaps the directory tree of CGN is somehow different from the one I used? Or maybe some other command failed?

[miguelvr]

The file lists are populated but something is clearly going wrong... This is the feedback I'm getting

/code/kaldi-asr/egs/kaldi-training/s5/utils/prepare_lang.sh /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp <unk> data/local/lang_tmp_nosp data/lang_nosp
Checking /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/silence_phones.txt ...
--> reading /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/silence_phones.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/silence_phones.txt is OK

Checking /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/optional_silence.txt ...
--> reading /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/optional_silence.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/optional_silence.txt is OK

Checking /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/nonsilence_phones.txt ...
--> reading /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/nonsilence_phones.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/nonsilence_phones.txt is OK

Checking disjoint: silence_phones.txt, nonsilence_phones.txt
--> disjoint property is OK.

Checking /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/lexicon.txt
--> reading /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/lexicon.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/lexicon.txt is OK

Checking /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/extra_questions.txt ...
--> /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/extra_questions.txt is empty (this is OK)
--> SUCCESS [validating dictionary directory /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp]

**Creating /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/lexiconp.txt from /code/kaldi-asr/egs/kaldi-training/s5/data/local/dict_nosp/lexicon.txt
fstaddselfloops data/lang_nosp/phones/wdisambig_phones.int data/lang_nosp/phones/wdisambig_words.int 
prepare_lang.sh: validating output directory
utils/validate_lang.pl data/lang_nosp
Checking data/lang_nosp/phones.txt ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> data/lang_nosp/phones.txt is OK

Checking words.txt: #0 ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> data/lang_nosp/words.txt is OK

Checking disjoint: silence.txt, nonsilence.txt, disambig.txt ...
--> silence.txt and nonsilence.txt are disjoint
--> silence.txt and disambig.txt are disjoint
--> disambig.txt and nonsilence.txt are disjoint
--> disjoint property is OK

Checking sumation: silence.txt, nonsilence.txt, disambig.txt ...
--> summation property is OK

Checking data/lang_nosp/phones/context_indep.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 5 entry/entries in data/lang_nosp/phones/context_indep.txt
--> data/lang_nosp/phones/context_indep.int corresponds to data/lang_nosp/phones/context_indep.txt
--> data/lang_nosp/phones/context_indep.csl corresponds to data/lang_nosp/phones/context_indep.txt
--> data/lang_nosp/phones/context_indep.{txt, int, csl} are OK

Checking data/lang_nosp/phones/nonsilence.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 4 entry/entries in data/lang_nosp/phones/nonsilence.txt
--> data/lang_nosp/phones/nonsilence.int corresponds to data/lang_nosp/phones/nonsilence.txt
--> data/lang_nosp/phones/nonsilence.csl corresponds to data/lang_nosp/phones/nonsilence.txt
--> data/lang_nosp/phones/nonsilence.{txt, int, csl} are OK

Checking data/lang_nosp/phones/silence.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 5 entry/entries in data/lang_nosp/phones/silence.txt
--> data/lang_nosp/phones/silence.int corresponds to data/lang_nosp/phones/silence.txt
--> data/lang_nosp/phones/silence.csl corresponds to data/lang_nosp/phones/silence.txt
--> data/lang_nosp/phones/silence.{txt, int, csl} are OK

Checking data/lang_nosp/phones/optional_silence.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 1 entry/entries in data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.int corresponds to data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.csl corresponds to data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.{txt, int, csl} are OK

Checking data/lang_nosp/phones/disambig.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 5 entry/entries in data/lang_nosp/phones/disambig.txt
--> data/lang_nosp/phones/disambig.int corresponds to data/lang_nosp/phones/disambig.txt
--> data/lang_nosp/phones/disambig.csl corresponds to data/lang_nosp/phones/disambig.txt
--> data/lang_nosp/phones/disambig.{txt, int, csl} are OK

Checking data/lang_nosp/phones/roots.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 2 entry/entries in data/lang_nosp/phones/roots.txt
--> data/lang_nosp/phones/roots.int corresponds to data/lang_nosp/phones/roots.txt
--> data/lang_nosp/phones/roots.{txt, int} are OK

Checking data/lang_nosp/phones/sets.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 2 entry/entries in data/lang_nosp/phones/sets.txt
--> data/lang_nosp/phones/sets.int corresponds to data/lang_nosp/phones/sets.txt
--> data/lang_nosp/phones/sets.{txt, int} are OK

Checking data/lang_nosp/phones/extra_questions.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 9 entry/entries in data/lang_nosp/phones/extra_questions.txt
--> data/lang_nosp/phones/extra_questions.int corresponds to data/lang_nosp/phones/extra_questions.txt
--> data/lang_nosp/phones/extra_questions.{txt, int} are OK

Checking data/lang_nosp/phones/word_boundary.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 9 entry/entries in data/lang_nosp/phones/word_boundary.txt
--> data/lang_nosp/phones/word_boundary.int corresponds to data/lang_nosp/phones/word_boundary.txt
--> data/lang_nosp/phones/word_boundary.{txt, int} are OK

Checking optional_silence.txt ...
--> reading data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.txt is OK

Checking disambiguation symbols: #0 and #1
--> data/lang_nosp/phones/disambig.txt has "#0" and "#1"
--> data/lang_nosp/phones/disambig.txt is OK

Checking topo ...

Checking word_boundary.txt: silence.txt, nonsilence.txt, disambig.txt ...
--> data/lang_nosp/phones/word_boundary.txt doesn't include disambiguation symbols
--> data/lang_nosp/phones/word_boundary.txt is the union of nonsilence.txt and silence.txt
--> data/lang_nosp/phones/word_boundary.txt is OK

Checking word-level disambiguation symbols...
--> data/lang_nosp/phones/wdisambig.txt exists (newer prepare_lang.sh)
Checking word_boundary.int and disambig.int
--> generating a 50 word sequence
--> resulting phone sequence from L.fst corresponds to the word sequence
--> L.fst is OK
--> generating a 87 word sequence
--> resulting phone sequence from L_disambig.fst corresponds to the word sequence
--> L_disambig.fst is OK

Checking data/lang_nosp/oov.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 1 entry/entries in data/lang_nosp/oov.txt
--> data/lang_nosp/oov.int corresponds to data/lang_nosp/oov.txt
--> data/lang_nosp/oov.{txt, int} are OK

--> data/lang_nosp/L.fst is olabel sorted
--> data/lang_nosp/L_disambig.fst is olabel sorted
--> SUCCESS [validating lang directory data/lang_nosp]
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_t/utt2spk is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_t/spk2utt is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_t/text is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_t/segments is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_t/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 196879 utterances.
fix_data_dir.sh: old files are kept in data/train_t/.backup
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_s/utt2spk is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_s/spk2utt is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_s/text is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_s/segments is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/train_s/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 506074 utterances.
fix_data_dir.sh: old files are kept in data/train_s/.backup
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_t/utt2spk is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_t/spk2utt is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_t/text is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_t/segments is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 3996 utterances.
fix_data_dir.sh: old files are kept in data/dev_t/.backup
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_s/utt2spk is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_s/spk2utt is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_s/text is not in sorted order or not unique, sorting it
/code/kaldi-asr/egs/kaldi-training/s5/utils/fix_data_dir.sh: file data/dev_s/segments is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 409 utterances.
fix_data_dir.sh: old files are kept in data/dev_s/.backup
Data preparation succeeded
local/cgn_train_lms.sh --dict-suffix _nosp
Not installing the kaldi_lm toolkit since it is already there.
Getting training data with OOV words replaced with <unk> (train_nounk.gz)
Not creating raw N-gram counts ngrams.gz and heldout_ngrams.gz since they already exist in data/local/local_lm/3gram-mincount
(remove them if you want them regenerated)
Iteration 1/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=0.825000 phi=2.000000
/code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/train_lm.sh: line 333: compute_perplexity: command not found
/code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/train_lm.sh: line 333: compute_perplexity: command not found
/code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/train_lm.sh: line 333: compute_perplexity: command not found
interpolate_ngrams: 3 words in wordslist
interpolate_ngrams: 3 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.100000 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.485000 phi=2.000000
interpolate_ngrams: 3 words in wordslist

real    0m0.011s
user    0m0.000s
sys 0m0.000s

real    0m0.009s
user    0m0.000s
sys 0m0.000s

real    0m0.012s
user    0m0.000s
sys 0m0.000s
Usage: optimize_alpha.pl alpha1 perplexity@alpha1 alpha2 perplexity@alpha2 alpha3 perplexity@alph3 at /code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/optimize_alpha.pl line 23.
Expecting files data/local/local_lm/3gram-mincount//ngrams_disc and data/local/local_lm/3gram-mincount//../word_map to exist
E.g. see egs/wsj/s3/local/wsj_train_lm.sh for examples.
Not creating raw N-gram counts ngrams.gz and heldout_ngrams.gz since they already exist in data/local/local_lm/4gram-mincount
(remove them if you want them regenerated)
Iteration 1/8 of optimizing discounting parameters
/code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/train_lm.sh: line 333: compute_perplexity: command not found
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=0.675000 phi=2.000000
interpolate_ngrams: 3 words in wordslist

real    0m0.004s
user    0m0.000s
sys 0m0.000s
/code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/train_lm.sh: line 333: compute_perplexity: command not found
interpolate_ngrams: 3 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=1.215000 phi=2.000000
interpolate_ngrams: 3 words in wordslist

real    0m0.005s
user    0m0.000s
sys 0m0.000s
/code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/train_lm.sh: line 333: compute_perplexity: command not found

real    0m0.004s
user    0m0.000s
sys 0m0.000s
Usage: optimize_alpha.pl alpha1 perplexity@alpha1 alpha2 perplexity@alpha2 alpha3 perplexity@alph3 at /code/kaldi-asr/egs/kaldi-training/s5/../../../tools/kaldi_lm/optimize_alpha.pl line 23.
Expecting files data/local/local_lm/4gram-mincount/ngrams_disc and data/local/local_lm/4gram-mincount/../word_map to exist
E.g. see egs/wsj/s3/local/wsj_train_lm.sh for examples.
local/cgn_format_local_lms.sh --lang-suffix _nosp
gzip: data/local/local_lm/3gram-mincount/lm_pr1.5.gz: No such file or directory
arpa2fst --disambig-symbol=#0 --read-symbol-table=data/lang_nosp/words.txt - data/lang_nosp_test_tgpr/G.fst 
ERROR (arpa2fst[5.4.133~1401-10883]:Read():arpa-file-parser.cc:127) line 1 []: \data\ section missing or empty.

[ Stack-Trace: ]
arpa2fst() [0x595e2a]
kaldi::MessageLogger::HandleMessage(kaldi::LogMessageEnvelope const&, char const*)
kaldi::MessageLogger::~MessageLogger()
kaldi::ArpaFileParser::Read(std::istream&)
main
__libc_start_main
_start

steps/make_mfcc.sh --cmd run.pl --nj 30 data/train_s
utils/validate_data_dir.sh: Mal-formed spk2gender file

[miguelvr]

Looks like the script it is not detecting any words. I don't have any experience with perl, so I can't really tell if it is the process_flist.pl that is failing.

[laurensw75]

According to the log you show, the LM generation is not working properly. It is also not generating certain files because they already exist (an earlier failed run perhaps?), so I would at least recommend cleaning things up after a failed run.

One of the errors shown is 'compute_perplexity: command not found', which is part of kaldi_lm. In your $KALDI_ROOT/tools/kaldi_lm directory, this program is supposed to exist. If not, your kaldi_lm installation may be problematic.

[miguelvr]

I fixed the kaldi_lm problem. For some reason it was not installing everything (only some functionalities). Now it works fine, but the script still fails at the same stage.

I checked the data/local/dict_nosp/lexicon.txt and the file only contains the 3 special symbols. Is this normal behaviour?

[laurensw75]

Clearly, that is not correct.

The line: cat $cgn/data/lexicon/xml/cgnlex.lex | recode -d h..u8 | perl -CSD $local/format_lexicon.pl $lang | sort >$dictdir/lexicon.txt (line 65 in cgn_data_prep.sh)

Should create a lexicon.txt file. Do you have recode installed? If so, is it possible that you don't have the /data/lexicon/xml/cgnlex.lex file? Either problem can be overcome if you uncomment the following few lines (as instructed in the code). As CGN is not in UTF-8 by default, I would expect you to need recode anyway though.

[miguelvr]

I only have /data/lexicon/xml/cgnlex_2.0.lex is this equivalent? Maybe our corpus versions do indeed differ

[laurensw75]

Pretty sure you can use that file. My copy of CGN may be a bit older, and I don't know what changes were made since then.

[miguelvr]

I've decided to give this another shot, and now the script is exiting at this line: $utils/fix_data_dir.sh data/$x || exit 1; for the train_t folder

But I'm looking at the files and they look okay-ish, but I have no clue what anomalies to look for.

Would you be kind enough to look at the generated files?

Best regards, Miguel

EDIT: There are a bunch of UNKNOWN speakers, is this normal?

[laurensw75]

Usually fix_data_dir.sh gives some reason for not finishing correctly.

In your spk2utt file the presence of the UNKNOWN speakers is normal, but the sorting is different from my files. The sorting in your utt2spk file looks fine, so it seems a bit odd, as I would have expected to see the same order in spk2utt.

I cannot say for sure if the sorting is the cause of your issues, but if fix_data_dir.sh fails there should be some message there.

[miguelvr]

It was being caused by the issue solved by this commit

I'm going to pull the most recent version of kaldi and try again. This is a pain as there a no kaldi versions, they recommend using the bleeding edge version from the master

[miguelvr]

Managed to get it training, but had to make a small change in process_flist.pl because there were speakers that were not being tagged by genre.

However I'm getting a few warnings... not sure if normal

a few of these:

(...)
WARNING (gmm-init-model[5.4.198~1402-be7c1]:InitAmGmm():gmm-init-model.cc:55) Tree has pdf-id 80 with no stats; corresponding phone list: 322 323 324 325 
** The warnings above about 'no stats' generally mean you have phones **
** (or groups of phones) in your phone set that had no corresponding data. **
** You should probably figure out whether something went wrong, **
** or whether your data just doesn't happen to have examples of those **
** phones. **

these:

analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 15.7937974944% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 18.8334360238% of the time at utterance end.  This may not be optimal.

and some warnings in logs that I haven't checked yet:

98 warnings in exp/train_t/tri1/log/init_model.log
2 warnings in exp/train_t/tri1/log/analyze_alignments.log
1306 warnings in exp/train_t/tri1/log/update.*.log
3 warnings in exp/train_t/tri1/log/build_tree.log
748 warnings in exp/train_t/tri1/log/acc.*.*.log
1135 warnings in exp/train_t/tri1/log/align.*.*.log
34 warnings in exp/train_t/tri1/log/questions.log

is this expected?

Best regards, Miguel

[laurensw75]

The first one is not fatal, but may mean that some of the words in the lexicon cannot be produced due to missing evidence in the training material. Most likely there is a minor mismatch somewhere, but I doubt it will impact performance very much. The other two seem par for the course :-)

[miguelvr]

My training process seems to be stuck at stage 6 for 3-4h now:

    steps/cleanup/clean_and_segment_data.sh --nj $nj --cmd "$train_cmd" --segmentation-opts "--min-segment-length 0.3 --min-new-segment-length 0.6" \
      data/train_${x} data/lang_${x} exp/train_${x}/tri3 exp/train_${x}/tri3_cleaned_work data/train_${x}_cleaned

Is this behaviour normal?

You have the code commented saying that the user is free to skip that step if a NN is not going to be trained. By default this doesn't happen, which makes me wonder, if the next steps are really necessary, i.e. the extra SAT pass.

[miguelvr]

it took about 7h, but in the end, it worked.

I have a few following questions:

• Could you point me to the easiest way to test the model?
• And how to export it to use it with your Kaldi_NL repo for example

Best regards, Miguel

[laurensw75]

I was not available to reply (in time) to your earlier message, but 7 hours sounds about right. Not sure how much it brings exactly, but you should only do it once (at most) anyway.

The recipe includes decode examples, so in your exp/train_?/ directories there should be some results on the dev set. If you want to use them with my Kaldi_NL repo, I would simply install that and then make a copy of the standard model directories (AM/LM/lexicon) to a name that identifies your efforts and replace the relevant files with the ones you just trained. Just make sure that you use the right 'types' of model (i.e. tri4 or chain, etc).

[miguelvr]

Just ran cat train_cleaned/tri4/decode_dev_s_tgpr/wer_* | grep WER and got

%WER 15.61 [ 848 / 5434, 171 ins, 107 del, 570 sub ]
%WER 15.20 [ 826 / 5434, 162 ins, 110 del, 554 sub ]
%WER 15.07 [ 819 / 5434, 152 ins, 114 del, 553 sub ]
%WER 15.24 [ 828 / 5434, 149 ins, 124 del, 555 sub ]
%WER 15.24 [ 828 / 5434, 142 ins, 128 del, 558 sub ]
%WER 15.35 [ 834 / 5434, 128 ins, 138 del, 568 sub ]
%WER 15.79 [ 858 / 5434, 130 ins, 140 del, 588 sub ]
%WER 16.18 [ 879 / 5434, 128 ins, 148 del, 603 sub ]
%WER 20.79 [ 1130 / 5434, 307 ins, 73 del, 750 sub ]
%WER 19.10 [ 1038 / 5434, 270 ins, 76 del, 692 sub ]
%WER 18.26 [ 992 / 5434, 250 ins, 79 del, 663 sub ]
%WER 16.97 [ 922 / 5434, 226 ins, 81 del, 615 sub ]
%WER 16.38 [ 890 / 5434, 212 ins, 85 del, 593 sub ]
%WER 15.44 [ 839 / 5434, 177 ins, 93 del, 569 sub ]

Which is an average WER of 16.47%. I don't have a baseline for this corpora, but this seems okay, right?

Best regards, Miguel

PS: If you have some of the WER values of your models, it would be useful to share them in the README.

PPS: For the telephony dev set I get 49.18% WER which seems rather high

[laurensw75]

Those numbers look ok. The standard recipe uses only data from CGN, whose phonetic lexicon is not great. If you use a different (better) phonetic lexicon, such as the one that comes with my Kaldi_NL models, you can improve a bit.

As a reference, I've used the Kaldi_NL lexicon (look in the local/cgn_data_prep.sh script for hints on how to do this). That results in: %WER 12.13 [ 659 / 5434, 106 ins, 92 del, 461 sub ]

If you choose to also train nnet models, for the chain_cleaned lstm models you can then end up with: %WER 4.45 [ 242 / 5434, 18 ins, 43 del, 181 sub ]

I've also experimented with RNN language models. As an additional rescoring pass on the chain_cleaned lstm results that gives: %WER 3.40 [ 185 / 5434, 11 ins, 29 del, 145 sub ]

PS: for telephony dev set I get: %WER 45.78 [ 11138 / 24328, 1841 ins, 2574 del, 6723 sub ] [PARTIAL] (tri4) %WER 24.75 [ 6021 / 24328, 1287 ins, 1692 del, 3042 sub ] [PARTIAL] (lstm) %WER 22.87 [ 5563 / 24328, 1168 ins, 1708 del, 2687 sub ] [PARTIAL] (lstm+rnn)

[miguelvr]

One more question:

What's is the procedure regarding numerical values vs theirs written down counterparts e.g. "1 vs een" or "2 vs twee", etc.

I'm also running experiments with other frameworks, namely some end-to-end character-based ones, and I'm not sure how to process the transcriptions for these cases.

Best regards, Miguel

[laurensw75]

I've just made a script that converts the spelled out numbers into their numeric value. For most cases, it will be best to recognize numbers as words and then convert them to whatever representation you need. I have not experimented with end-to-end systems, so cannot comment on that.

[Aariciah]

Hi, I'm also having some trouble with run.sh. I cleaned things up after some failed runs, as recommended above but I don't understand what happened with this failed run. Thank you kindly in advance.

/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/prepare_lang.sh /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp <unk> data/local/lang_tmp_nosp data/lang_nosp
Checking /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/silence_phones.txt ...
--> reading /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/silence_phones.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/silence_phones.txt is OK

Checking /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/optional_silence.txt ...
--> reading /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/optional_silence.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/optional_silence.txt is OK

Checking /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/nonsilence_phones.txt ...
--> reading /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/nonsilence_phones.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/nonsilence_phones.txt is OK

Checking disjoint: silence_phones.txt, nonsilence_phones.txt
--> disjoint property is OK.

Checking /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/lexicon.txt
--> reading /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/lexicon.txt
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/lexicon.txt is OK

Checking /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/extra_questions.txt ...
--> /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/extra_questions.txt is empty (this is OK)
--> SUCCESS [validating dictionary directory /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp]

**Creating /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/lexiconp.txt from /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/data/local/dict_nosp/lexicon.txt
fstaddselfloops data/lang_nosp/phones/wdisambig_phones.int data/lang_nosp/phones/wdisambig_words.int 
prepare_lang.sh: validating output directory
utils/validate_lang.pl data/lang_nosp
Checking existence of separator file
separator file data/lang_nosp/subword_separator.txt is empty or does not exist, deal in word case.
Checking data/lang_nosp/phones.txt ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> data/lang_nosp/phones.txt is OK

Checking words.txt: #0 ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> data/lang_nosp/words.txt is OK

Checking disjoint: silence.txt, nonsilence.txt, disambig.txt ...
--> silence.txt and nonsilence.txt are disjoint
--> silence.txt and disambig.txt are disjoint
--> disambig.txt and nonsilence.txt are disjoint
--> disjoint property is OK

Checking sumation: silence.txt, nonsilence.txt, disambig.txt ...
--> found no unexplainable phones in phones.txt

Checking data/lang_nosp/phones/context_indep.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 5 entry/entries in data/lang_nosp/phones/context_indep.txt
--> data/lang_nosp/phones/context_indep.int corresponds to data/lang_nosp/phones/context_indep.txt
--> data/lang_nosp/phones/context_indep.csl corresponds to data/lang_nosp/phones/context_indep.txt
--> data/lang_nosp/phones/context_indep.{txt, int, csl} are OK

Checking data/lang_nosp/phones/nonsilence.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 320 entry/entries in data/lang_nosp/phones/nonsilence.txt
--> data/lang_nosp/phones/nonsilence.int corresponds to data/lang_nosp/phones/nonsilence.txt
--> data/lang_nosp/phones/nonsilence.csl corresponds to data/lang_nosp/phones/nonsilence.txt
--> data/lang_nosp/phones/nonsilence.{txt, int, csl} are OK

Checking data/lang_nosp/phones/silence.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 5 entry/entries in data/lang_nosp/phones/silence.txt
--> data/lang_nosp/phones/silence.int corresponds to data/lang_nosp/phones/silence.txt
--> data/lang_nosp/phones/silence.csl corresponds to data/lang_nosp/phones/silence.txt
--> data/lang_nosp/phones/silence.{txt, int, csl} are OK

Checking data/lang_nosp/phones/optional_silence.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 1 entry/entries in data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.int corresponds to data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.csl corresponds to data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.{txt, int, csl} are OK

Checking data/lang_nosp/phones/disambig.{txt, int, csl} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 11 entry/entries in data/lang_nosp/phones/disambig.txt
--> data/lang_nosp/phones/disambig.int corresponds to data/lang_nosp/phones/disambig.txt
--> data/lang_nosp/phones/disambig.csl corresponds to data/lang_nosp/phones/disambig.txt
--> data/lang_nosp/phones/disambig.{txt, int, csl} are OK

Checking data/lang_nosp/phones/roots.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 81 entry/entries in data/lang_nosp/phones/roots.txt
--> data/lang_nosp/phones/roots.int corresponds to data/lang_nosp/phones/roots.txt
--> data/lang_nosp/phones/roots.{txt, int} are OK

Checking data/lang_nosp/phones/sets.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 81 entry/entries in data/lang_nosp/phones/sets.txt
--> data/lang_nosp/phones/sets.int corresponds to data/lang_nosp/phones/sets.txt
--> data/lang_nosp/phones/sets.{txt, int} are OK

Checking data/lang_nosp/phones/extra_questions.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 9 entry/entries in data/lang_nosp/phones/extra_questions.txt
--> data/lang_nosp/phones/extra_questions.int corresponds to data/lang_nosp/phones/extra_questions.txt
--> data/lang_nosp/phones/extra_questions.{txt, int} are OK

Checking data/lang_nosp/phones/word_boundary.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 325 entry/entries in data/lang_nosp/phones/word_boundary.txt
--> data/lang_nosp/phones/word_boundary.int corresponds to data/lang_nosp/phones/word_boundary.txt
--> data/lang_nosp/phones/word_boundary.{txt, int} are OK

Checking optional_silence.txt ...
--> reading data/lang_nosp/phones/optional_silence.txt
--> data/lang_nosp/phones/optional_silence.txt is OK

Checking disambiguation symbols: #0 and #1
--> data/lang_nosp/phones/disambig.txt has "#0" and "#1"
--> data/lang_nosp/phones/disambig.txt is OK

Checking topo ...

Checking word_boundary.txt: silence.txt, nonsilence.txt, disambig.txt ...
--> data/lang_nosp/phones/word_boundary.txt doesn't include disambiguation symbols
--> data/lang_nosp/phones/word_boundary.txt is the union of nonsilence.txt and silence.txt
--> data/lang_nosp/phones/word_boundary.txt is OK

Checking word-level disambiguation symbols...
--> data/lang_nosp/phones/wdisambig.txt exists (newer prepare_lang.sh)
Checking word_boundary.int and disambig.int
--> generating a 20 word/subword sequence
--> resulting phone sequence from L.fst corresponds to the word sequence
--> L.fst is OK
--> generating a 49 word/subword sequence
--> resulting phone sequence from L_disambig.fst corresponds to the word sequence
--> L_disambig.fst is OK

Checking data/lang_nosp/oov.{txt, int} ...
--> text seems to be UTF-8 or ASCII, checking whitespaces
--> text contains only allowed whitespaces
--> 1 entry/entries in data/lang_nosp/oov.txt
--> data/lang_nosp/oov.int corresponds to data/lang_nosp/oov.txt
--> data/lang_nosp/oov.{txt, int} are OK

--> data/lang_nosp/L.fst is olabel sorted
--> data/lang_nosp/L_disambig.fst is olabel sorted
--> SUCCESS [validating lang directory data/lang_nosp]
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_t/utt2spk is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_t/spk2utt is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_t/text is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_t/segments is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_t/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 196879 utterances.
fix_data_dir.sh: old files are kept in data/train_t/.backup
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_s/utt2spk is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_s/spk2utt is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_s/text is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_s/segments is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/train_s/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 506074 utterances.
fix_data_dir.sh: old files are kept in data/train_s/.backup
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_t/utt2spk is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_t/spk2utt is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_t/text is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_t/segments is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 3996 utterances.
fix_data_dir.sh: old files are kept in data/dev_t/.backup
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_s/utt2spk is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_s/spk2utt is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_s/text is not in sorted order or not unique, sorting it
/home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/fix_data_dir.sh: file data/dev_s/segments is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 409 utterances.
fix_data_dir.sh: old files are kept in data/dev_s/.backup
Data preparation succeeded
local/cgn_train_lms.sh --dict-suffix _nosp
Not installing the kaldi_lm toolkit since it is already there.
Getting training data with OOV words replaced with <unk> (train_nounk.gz)
Getting raw N-gram counts
discount_ngrams: for n-gram order 1, D=0.000000, tau=0.000000 phi=1.000000
discount_ngrams: for n-gram order 2, D=0.000000, tau=0.000000 phi=1.000000
discount_ngrams: for n-gram order 3, D=1.000000, tau=0.000000 phi=1.000000
Iteration 1/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=0.825000 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.100000 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.485000 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 159.522903
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 165.480886

real    0m7.454s
user    0m9.344s
sys 0m0.536s
Perplexity over 115036.000000 words is 159.128529
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 165.122108
Perplexity over 115036.000000 words is 159.878877
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 165.807496

real    0m7.476s
user    0m9.362s
sys 0m0.527s

real    0m7.516s
user    0m9.562s
sys 0m0.514s
optimize_alpha.pl: alpha=1.31273108080119 is too positive, limiting it to 0.7
Projected perplexity change from setting alpha=0.7 is 159.522903->158.8554762, reduction of 0.667426800000158
Alpha value on iter 1 is 0.7
Iteration 2/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.402500 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.870000 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.524500 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 159.077362
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 165.113473
Perplexity over 115036.000000 words is 158.906842
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.936610

real    0m7.595s
user    0m9.434s
sys 0m0.565s

real    0m7.591s
user    0m9.467s
sys 0m0.543s
Perplexity over 115036.000000 words is 158.756764
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.782568

real    0m7.649s
user    0m9.577s
sys 0m0.517s
Projected perplexity change from setting alpha=0.682878172589042 is 158.906842->158.709987245877, reduction of 0.196854754122597
Alpha value on iter 2 is 0.682878172589042
Iteration 3/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.750000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=2.350000
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 158.671736
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.695308
Perplexity over 115036.000000 words is 158.674197
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.699548
Perplexity over 115036.000000 words is 158.714994
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.738626

real    0m7.543s
user    0m9.442s
sys 0m0.522s

real    0m7.558s
user    0m9.473s
sys 0m0.518s

real    0m7.521s
user    0m9.440s
sys 0m0.522s
Projected perplexity change from setting alpha=-0.102868420714761 is 158.671736->158.66938261301, reduction of 0.00235338699039289
Alpha value on iter 3 is -0.102868420714761
Iteration 4/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=1.080000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 162.538130
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 168.825655

real    0m5.329s
user    0m6.637s
sys 0m0.345s
Perplexity over 115036.000000 words is 158.669074
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.693197

real    0m7.691s
user    0m9.624s
sys 0m0.592s
Perplexity over 115036.000000 words is 158.653242
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.641991

real    0m7.744s
user    0m9.708s
sys 0m0.546s
Projected perplexity change from setting alpha=-0.126728523021393 is 158.669074->158.374876244238, reduction of 0.294197755762468
Alpha value on iter 4 is -0.126728523021393
Iteration 5/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=1.147500 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=2.065500 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 158.506542
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.505799

real    0m7.575s
user    0m9.478s
sys 0m0.550s
Perplexity over 115036.000000 words is 158.421698
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.422926

real    0m7.594s
user    0m9.471s
sys 0m0.547s
Perplexity over 115036.000000 words is 158.593769
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.588728

real    0m7.781s
user    0m9.810s
sys 0m0.531s
optimize_alpha.pl: alpha=0.857871078344494 is too positive, limiting it to 0.7
Projected perplexity change from setting alpha=0.7 is 158.506542->158.3803401, reduction of 0.126201900000041
Alpha value on iter 5 is 0.7
Iteration 6/6 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=2.601000 phi=1.750000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=2.601000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=2.601000 phi=2.350000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 158.354767
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.349239

real    0m7.518s
user    0m9.360s
sys 0m0.554s
Perplexity over 115036.000000 words is 158.392099
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.397430
Perplexity over 115036.000000 words is 158.420657
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.435436

real    0m7.585s
user    0m9.483s
sys 0m0.530s

real    0m7.599s
user    0m9.451s
sys 0m0.575s
optimize_alpha.pl: alpha=4.40254038952905 is too positive, limiting it to 0.7
Projected perplexity change from setting alpha=0.7 is 158.392099->158.320525733333, reduction of 0.0715732666666327
Alpha value on iter 6 is 0.7
Final config is:
D=0.6 tau=1.53 phi=2
D=0.698617181582886 tau=2.601 phi=2.7
D=0 tau=3.14698218274151 phi=1.89713157928524
Discounting N-grams.
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.698617, tau=2.601000 phi=2.700000
discount_ngrams: for n-gram order 3, D=0.000000, tau=3.146982 phi=1.897132
Computing final perplexity
Building ARPA LM (perplexity computation is in background)
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 158.367475
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 164.357110
158.367475
Done training LM of type 3gram-mincount
Pruning N-grams
Removed 1063563 parameters, total divergence 494359.437025
Average divergence per parameter is 0.464814, versus threshold 1.500000
Computing pruned perplexity
interpolate_ngrams: 133557 words in wordslist
After pruning, number of N-grams is 474712
Building ARPA LM (perplexity computation is in background)
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 167.556966
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 174.078534
167.556966
ARPA output is in data/local/local_lm/3gram-mincount//lm_pr1.5.gz
Done pruning LM with threshold 1.5
Getting raw N-gram counts
discount_ngrams: for n-gram order 1, D=0.000000, tau=0.000000 phi=1.000000
discount_ngrams: for n-gram order 2, D=0.000000, tau=0.000000 phi=1.000000
discount_ngrams: for n-gram order 3, D=1.000000, tau=0.000000 phi=1.000000
discount_ngrams: for n-gram order 4, D=1.000000, tau=0.000000 phi=1.000000
Iteration 1/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=0.675000 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=0.900000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=0.900000 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.215000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=1.215000 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 157.505601
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 163.290314

real    0m9.752s
user    0m12.233s
sys 0m0.685s
Perplexity over 115036.000000 words is 156.566585
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.410693

real    0m9.760s
user    0m12.166s
sys 0m0.730s
Perplexity over 115036.000000 words is 157.059713
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.871141

real    0m9.887s
user    0m12.368s
sys 0m0.686s
optimize_alpha.pl: alpha=1.30330854088574 is too positive, limiting it to 0.7
Projected perplexity change from setting alpha=0.7 is 157.059713->156.226424733333, reduction of 0.833288266666585
Alpha value on iter 1 is 0.7
Iteration 2/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=1.147500 phi=2.000000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=1.530000 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.065500 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 156.380350
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.262134

real    0m9.731s
user    0m12.196s
sys 0m0.654s
Perplexity over 115036.000000 words is 156.258032
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.135801

real    0m9.777s
user    0m12.186s
sys 0m0.722s
Perplexity over 115036.000000 words is 156.149828
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.024502

real    0m9.829s
user    0m12.416s
sys 0m0.705s
Projected perplexity change from setting alpha=0.689920401260892 is 156.258032->156.115141567241, reduction of 0.14289043275906
Alpha value on iter 2 is 0.689920401260892
Iteration 3/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.750000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=2.000000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=2.350000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 156.018155
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.889029

real    0m9.964s
user    0m12.345s
sys 0m0.748s
Perplexity over 115036.000000 words is 156.072369
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.945069

real    0m10.096s
user    0m12.560s
sys 0m0.699s
Perplexity over 115036.000000 words is 156.149035
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.024396

real    0m10.115s
user    0m12.589s
sys 0m0.711s
optimize_alpha.pl: alpha=-29.8351774535627 is too negative, limiting it to -0.5
Projected perplexity change from setting alpha=-0.5 is 156.072369->155.964397190476, reduction of 0.107971809523832
Alpha value on iter 3 is -0.5
Iteration 4/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.147500 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.065500 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 156.144478
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 162.020621
Perplexity over 115036.000000 words is 155.825596
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.690066
Perplexity over 115036.000000 words is 155.974526
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.844339

real    0m9.883s
user    0m12.270s
sys 0m0.746s

real    0m9.858s
user    0m12.363s
sys 0m0.723s

real    0m9.882s
user    0m12.299s
sys 0m0.708s
Projected perplexity change from setting alpha=0.67699544284017 is 155.974526->155.780278308854, reduction of 0.194247691146018
Alpha value on iter 4 is 0.67699544284017
Iteration 5/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.750000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=2.000000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=2.350000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 155.798216
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.661575

real    0m9.761s
user    0m12.172s
sys 0m0.726s
Perplexity over 115036.000000 words is 155.705262
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.567419

real    0m9.849s
user    0m12.363s
sys 0m0.695s
Perplexity over 115036.000000 words is 155.731891
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.593887

real    0m9.920s
user    0m12.398s
sys 0m0.711s
optimize_alpha.pl: alpha=-0.51007182107332 is too negative, limiting it to -0.5
Projected perplexity change from setting alpha=-0.5 is 155.731891->155.695921333333, reduction of 0.0359696666666025
Alpha value on iter 5 is -0.5
Iteration 6/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.800000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=1.080000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 159.509748
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 165.631957

real    0m7.421s
user    0m9.186s
sys 0m0.473s
Perplexity over 115036.000000 words is 155.707118
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.537072

real    0m9.665s
user    0m12.159s
sys 0m0.685s
Perplexity over 115036.000000 words is 155.717955
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.582697

real    0m9.773s
user    0m12.218s
sys 0m0.663s
Projected perplexity change from setting alpha=-0.126205188383732 is 155.717955->155.431511777556, reduction of 0.286443222443495
Alpha value on iter 6 is -0.126205188383732
Iteration 7/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=1.147500 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=2.065500 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 155.647336
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.483250

real    0m9.812s
user    0m12.320s
sys 0m0.719s
Perplexity over 115036.000000 words is 155.560051
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.400276

real    0m9.820s
user    0m12.269s
sys 0m0.699s
Perplexity over 115036.000000 words is 155.480147
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.322731

real    0m9.836s
user    0m12.339s
sys 0m0.708s
optimize_alpha.pl: alpha=0.741762028608297 is too positive, limiting it to 0.7
Projected perplexity change from setting alpha=0.7 is 155.560051->155.449587166667, reduction of 0.110463833333284
Alpha value on iter 7 is 0.7
Iteration 8/8 of optimizing discounting parameters
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=2.601000 phi=1.750000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=2.601000 phi=2.000000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=2.601000 phi=2.350000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 155.451820
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.298601

real    0m9.810s
user    0m12.313s
sys 0m0.699s
Perplexity over 115036.000000 words is 155.478636
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.334794
Perplexity over 115036.000000 words is 155.414176
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.250228

real    0m9.867s
user    0m12.403s
sys 0m0.740s

real    0m9.906s
user    0m12.433s
sys 0m0.713s
optimize_alpha.pl: objective function is not convex; returning alpha=0.7
Projected perplexity change from setting alpha=0.7 is 155.451820->155.376413466667, reduction of 0.0754065333333358
Alpha value on iter 8 is 0.7
Final config is:
D=0.6 tau=1.53 phi=2.0
D=0.699035849293014 tau=2.601 phi=2.7
D=0.0 tau=2.56580302754546 phi=1.5
D=0.0 tau=2.58557821392916 phi=1.5
Discounting N-grams.
discount_ngrams: for n-gram order 1, D=0.600000, tau=1.530000 phi=2.000000
discount_ngrams: for n-gram order 2, D=0.699036, tau=2.601000 phi=2.700000
discount_ngrams: for n-gram order 3, D=0.000000, tau=2.565803 phi=1.500000
discount_ngrams: for n-gram order 4, D=0.000000, tau=2.585578 phi=1.500000
Computing final perplexity
Building ARPA LM (perplexity computation is in background)
interpolate_ngrams: 133557 words in wordslist
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 155.427936
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 161.259152
155.427936
Done training LM of type 4gram-mincount
Pruning N-grams
Removed 1320191 parameters, total divergence 580420.241259
Average divergence per parameter is 0.439649, versus threshold 1.500000
Computing pruned perplexity
interpolate_ngrams: 133557 words in wordslist
After pruning, number of N-grams is 517418
Building ARPA LM (perplexity computation is in background)
interpolate_ngrams: 133557 words in wordslist
Perplexity over 115036.000000 words is 165.347818
Perplexity over 112774.000000 words (excluding 2262.000000 OOVs) is 171.745769
165.347818
ARPA output is in data/local/local_lm/4gram-mincount/lm_pr1.5.gz
Done pruning LM with threshold 1.5
local/cgn_format_local_lms.sh --lang-suffix _nosp
arpa2fst --disambig-symbol=#0 --read-symbol-table=data/lang_nosp/words.txt - data/lang_nosp_test_tgpr/G.fst 
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:94) Reading \data\ section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \1-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \2-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \3-grams: section.
LOG (arpa2fst[5.5]:RemoveRedundantStates():lm/arpa-lm-compiler.cc:359) Reduced num-states from 364067 to 75783
fstisstochastic data/lang_nosp_test_tgpr/G.fst 
1.0609e-05 -0.337552
arpa2fst --disambig-symbol=#0 --read-symbol-table=data/lang_nosp/words.txt - data/lang_nosp_test_tg/G.fst 
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:94) Reading \data\ section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \1-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \2-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \3-grams: section.
LOG (arpa2fst[5.5]:RemoveRedundantStates():lm/arpa-lm-compiler.cc:359) Reduced num-states from 1081217 to 214817
fstisstochastic data/lang_nosp_test_tg/G.fst 
9.22247e-06 -0.539687
arpa-to-const-arpa --bos-symbol=133559 --eos-symbol=133560 --unk-symbol=140 - data/lang_nosp_test_tgconst/G.carpa 
LOG (arpa-to-const-arpa[5.5]:BuildConstArpaLm():lm/const-arpa-lm.cc:1078) Reading -
utils/map_arpa_lm.pl: Processing "\data\"
utils/map_arpa_lm.pl: Processing "\1-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:94) Reading \data\ section.
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \1-grams: section.
utils/map_arpa_lm.pl: Processing "\2-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \2-grams: section.
utils/map_arpa_lm.pl: Processing "\3-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \3-grams: section.
arpa2fst --disambig-symbol=#0 --read-symbol-table=data/lang_nosp/words.txt - data/lang_nosp_test_fg/G.fst 
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:94) Reading \data\ section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \1-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \2-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \3-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \4-grams: section.
LOG (arpa2fst[5.5]:RemoveRedundantStates():lm/arpa-lm-compiler.cc:359) Reduced num-states from 1486197 to 361272
fstisstochastic data/lang_nosp_test_fg/G.fst 
9.09031e-06 -0.612351
arpa-to-const-arpa --bos-symbol=133559 --eos-symbol=133560 --unk-symbol=140 - data/lang_nosp_test_fgconst/G.carpa 
LOG (arpa-to-const-arpa[5.5]:BuildConstArpaLm():lm/const-arpa-lm.cc:1078) Reading -
utils/map_arpa_lm.pl: Processing "\data\"
utils/map_arpa_lm.pl: Processing "\1-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:94) Reading \data\ section.
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \1-grams: section.
utils/map_arpa_lm.pl: Processing "\2-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \2-grams: section.
utils/map_arpa_lm.pl: Processing "\3-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \3-grams: section.
utils/map_arpa_lm.pl: Processing "\4-grams:\"
LOG (arpa-to-const-arpa[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \4-grams: section.
arpa2fst --disambig-symbol=#0 --read-symbol-table=data/lang_nosp/words.txt - data/lang_nosp_test_fgpr/G.fst 
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:94) Reading \data\ section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \1-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \2-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \3-grams: section.
LOG (arpa2fst[5.5]:Read():lm/arpa-file-parser.cc:149) Reading \4-grams: section.
LOG (arpa2fst[5.5]:RemoveRedundantStates():lm/arpa-lm-compiler.cc:359) Reduced num-states from 498792 to 101342
fstisstochastic data/lang_nosp_test_fgpr/G.fst 
1.02835e-05 -0.690422
steps/make_mfcc.sh --cmd run.pl --nj 30 data/train_s
utils/validate_data_dir.sh: Successfully validated data-directory data/train_s
steps/make_mfcc.sh [info]: segments file exists: using that.
steps/make_mfcc.sh: It seems not all of the feature files were successfully procesed (495917 != 506074); consider using utils/fix_data_dir.sh data/train_s
steps/make_mfcc.sh: Succeeded creating MFCC features for train_s
steps/compute_cmvn_stats.sh data/train_s
steps/compute_cmvn_stats.sh: warning: it seems not all of the speakers got cmvn stats (2351 != 2353);
Succeeded creating CMVN stats for train_s
steps/make_mfcc.sh --cmd run.pl --nj 30 data/dev_s
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_s
steps/make_mfcc.sh [info]: segments file exists: using that.
steps/make_mfcc.sh: Succeeded creating MFCC features for dev_s
steps/compute_cmvn_stats.sh data/dev_s
Succeeded creating CMVN stats for dev_s
steps/make_mfcc.sh --cmd run.pl --nj 30 --mfcc-config conf/mfcc_t.conf data/train_t
utils/validate_data_dir.sh: Successfully validated data-directory data/train_t
steps/make_mfcc.sh [info]: segments file exists: using that.
steps/make_mfcc.sh: It seems not all of the feature files were successfully procesed (192719 != 196879); consider using utils/fix_data_dir.sh data/train_t
steps/make_mfcc.sh: Succeeded creating MFCC features for train_t
steps/compute_cmvn_stats.sh data/train_t
Succeeded creating CMVN stats for train_t
steps/make_mfcc.sh --cmd run.pl --nj 30 --mfcc-config conf/mfcc_t.conf data/dev_t
utils/validate_data_dir.sh: Successfully validated data-directory data/dev_t
steps/make_mfcc.sh [info]: segments file exists: using that.
steps/make_mfcc.sh: It seems not all of the feature files were successfully procesed (3884 != 3996); consider using utils/fix_data_dir.sh data/dev_t
steps/make_mfcc.sh: Succeeded creating MFCC features for dev_t
steps/compute_cmvn_stats.sh data/dev_t
Succeeded creating CMVN stats for dev_t
utils/subset_data_dir.sh: reducing #utt from 506074 to 5000
utils/subset_data_dir.sh: reducing #utt from 196879 to 5000
utils/fix_data_dir.sh: filtered /tmp/kaldi.5DJl/speakers from 2353 to 2351 lines based on filter data/train_s/cmvn.scp.
utils/fix_data_dir.sh: filtered data/train_s/spk2utt from 2353 to 2351 lines based on filter /tmp/kaldi.5DJl/speakers.
utils/fix_data_dir.sh: filtered data/train_s/spk2gender from 2353 to 2351 lines based on filter /tmp/kaldi.5DJl/speakers.
fix_data_dir.sh: kept 495917 utterances out of 506071
fix_data_dir.sh: old files are kept in data/train_s/.backup
fix_data_dir.sh: kept 192719 utterances out of 196879
fix_data_dir.sh: old files are kept in data/train_t/.backup
fix_data_dir.sh: kept all 409 utterances.
fix_data_dir.sh: old files are kept in data/dev_s/.backup
fix_data_dir.sh: kept 3884 utterances out of 3996
fix_data_dir.sh: old files are kept in data/dev_t/.backup
steps/train_mono.sh --nj 30 --cmd run.pl data/train_s_5k data/lang_nosp exp/train_s/mono0a
** split_data.sh: warning, #lines is (utt2spk,feats.scp) is (5000,4888); you can 
**  use utils/fix_data_dir.sh data/train_s_5k to fix this.
filter_scps.pl: warning: some input lines were output to multiple files [OK if splitting per utt] 
steps/train_mono.sh: Initializing monophone system.
steps/train_mono.sh: Compiling training graphs
steps/train_mono.sh: Aligning data equally (pass 0)
steps/train_mono.sh: Pass 1
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 2
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 3
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 4
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 5
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 6
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 7
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 8
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 9
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 10
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 11
steps/train_mono.sh: Pass 12
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 13
steps/train_mono.sh: Pass 14
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 15
steps/train_mono.sh: Pass 16
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 17
steps/train_mono.sh: Pass 18
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 19
steps/train_mono.sh: Pass 20
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 21
steps/train_mono.sh: Pass 22
steps/train_mono.sh: Pass 23
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 24
steps/train_mono.sh: Pass 25
steps/train_mono.sh: Pass 26
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 27
steps/train_mono.sh: Pass 28
steps/train_mono.sh: Pass 29
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 30
steps/train_mono.sh: Pass 31
steps/train_mono.sh: Pass 32
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 33
steps/train_mono.sh: Pass 34
steps/train_mono.sh: Pass 35
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 36
steps/train_mono.sh: Pass 37
steps/train_mono.sh: Pass 38
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 39
steps/diagnostic/analyze_alignments.sh --cmd run.pl data/lang_nosp exp/train_s/mono0a
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 18.5093679226% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 16.6459369818% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_alignments.sh: see stats in exp/train_s/mono0a/log/analyze_alignments.log
2 warnings in exp/train_s/mono0a/log/analyze_alignments.log
4525 warnings in exp/train_s/mono0a/log/update.*.log
17536 warnings in exp/train_s/mono0a/log/align.*.*.log
1529 warnings in exp/train_s/mono0a/log/acc.*.*.log
exp/train_s/mono0a: nj=30 align prob=-99.60 over 3.29h [retry=8.9%, fail=2.9%] states=245 gauss=997
steps/train_mono.sh: Done training monophone system in exp/train_s/mono0a
steps/train_mono.sh --nj 30 --cmd run.pl data/train_t_5k data/lang_nosp exp/train_t/mono0a
** split_data.sh: warning, #lines is (utt2spk,feats.scp) is (5000,4891); you can 
**  use utils/fix_data_dir.sh data/train_t_5k to fix this.
filter_scps.pl: warning: some input lines were output to multiple files [OK if splitting per utt] 
steps/train_mono.sh: Initializing monophone system.
steps/train_mono.sh: Compiling training graphs
steps/train_mono.sh: Aligning data equally (pass 0)
steps/train_mono.sh: Pass 1
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 2
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 3
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 4
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 5
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 6
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 7
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 8
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 9
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 10
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 11
steps/train_mono.sh: Pass 12
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 13
steps/train_mono.sh: Pass 14
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 15
steps/train_mono.sh: Pass 16
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 17
steps/train_mono.sh: Pass 18
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 19
steps/train_mono.sh: Pass 20
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 21
steps/train_mono.sh: Pass 22
steps/train_mono.sh: Pass 23
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 24
steps/train_mono.sh: Pass 25
steps/train_mono.sh: Pass 26
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 27
steps/train_mono.sh: Pass 28
steps/train_mono.sh: Pass 29
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 30
steps/train_mono.sh: Pass 31
steps/train_mono.sh: Pass 32
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 33
steps/train_mono.sh: Pass 34
steps/train_mono.sh: Pass 35
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 36
steps/train_mono.sh: Pass 37
steps/train_mono.sh: Pass 38
steps/train_mono.sh: Aligning data
steps/train_mono.sh: Pass 39
steps/diagnostic/analyze_alignments.sh --cmd run.pl data/lang_nosp exp/train_t/mono0a
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 16.7796262066% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 22.1459762944% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_alignments.sh: see stats in exp/train_t/mono0a/log/analyze_alignments.log
2 warnings in exp/train_t/mono0a/log/analyze_alignments.log
871 warnings in exp/train_t/mono0a/log/acc.*.*.log
4793 warnings in exp/train_t/mono0a/log/update.*.log
12494 warnings in exp/train_t/mono0a/log/align.*.*.log
exp/train_t/mono0a: nj=30 align prob=-90.03 over 2.17h [retry=6.3%, fail=2.6%] states=245 gauss=996
steps/train_mono.sh: Done training monophone system in exp/train_t/mono0a
tree-info exp/train_s/mono0a/tree 
tree-info exp/train_s/mono0a/tree 
fstdeterminizestar --use-log=true 
fstminimizeencoded 
fsttablecompose data/lang_nosp_test_tgpr/L_disambig.fst data/lang_nosp_test_tgpr/G.fst 
fstpushspecial 
fstisstochastic data/lang_nosp_test_tgpr/tmp/LG.fst 
-0.0216729 -0.0225837
[info]: LG not stochastic.
fstcomposecontext --context-size=1 --central-position=0 --read-disambig-syms=data/lang_nosp_test_tgpr/phones/disambig.int --write-disambig-syms=data/lang_nosp_test_tgpr/tmp/disambig_ilabels_1_0.int data/lang_nosp_test_tgpr/tmp/ilabels_1_0.27791 data/lang_nosp_test_tgpr/tmp/LG.fst 
fstisstochastic data/lang_nosp_test_tgpr/tmp/CLG_1_0.fst 
-0.0216729 -0.0225837
[info]: CLG not stochastic.
make-h-transducer --disambig-syms-out=exp/train_s/mono0a/graph_nosp_tgpr/disambig_tid.int --transition-scale=1.0 data/lang_nosp_test_tgpr/tmp/ilabels_1_0 exp/train_s/mono0a/tree exp/train_s/mono0a/final.mdl 
fsttablecompose exp/train_s/mono0a/graph_nosp_tgpr/Ha.fst data/lang_nosp_test_tgpr/tmp/CLG_1_0.fst 
fstdeterminizestar --use-log=true 
fstminimizeencoded 
fstrmsymbols exp/train_s/mono0a/graph_nosp_tgpr/disambig_tid.int 
fstrmepslocal 
fstisstochastic exp/train_s/mono0a/graph_nosp_tgpr/HCLGa.fst 
0.000386927 -0.0442039
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/train_s/mono0a/final.mdl exp/train_s/mono0a/graph_nosp_tgpr/HCLGa.fst 
steps/decode.sh --nj 4 --cmd run.pl exp/train_s/mono0a/graph_nosp_tgpr data/dev_s exp/train_s/mono0a/decode_nosp_tgpr
decode.sh: feature type is delta
steps/diagnostic/analyze_lats.sh --cmd run.pl exp/train_s/mono0a/graph_nosp_tgpr exp/train_s/mono0a/decode_nosp_tgpr
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 11.9804400978% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 1.95599022005% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_lats.sh: see stats in exp/train_s/mono0a/decode_nosp_tgpr/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(5,71,357) and mean=138.8
steps/diagnostic/analyze_lats.sh: see stats in exp/train_s/mono0a/decode_nosp_tgpr/log/analyze_lattice_depth_stats.log
tree-info exp/train_t/mono0a/tree 
tree-info exp/train_t/mono0a/tree 
make-h-transducer --disambig-syms-out=exp/train_t/mono0a/graph_nosp_tgpr/disambig_tid.int --transition-scale=1.0 data/lang_nosp_test_tgpr/tmp/ilabels_1_0 exp/train_t/mono0a/tree exp/train_t/mono0a/final.mdl 
fstdeterminizestar --use-log=true 
fstrmsymbols exp/train_t/mono0a/graph_nosp_tgpr/disambig_tid.int 
fstrmepslocal 
fsttablecompose exp/train_t/mono0a/graph_nosp_tgpr/Ha.fst data/lang_nosp_test_tgpr/tmp/CLG_1_0.fst 
fstminimizeencoded 
fstisstochastic exp/train_t/mono0a/graph_nosp_tgpr/HCLGa.fst 
0.000386927 -0.0442039
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/train_t/mono0a/final.mdl exp/train_t/mono0a/graph_nosp_tgpr/HCLGa.fst 
steps/decode.sh --nj 10 --cmd run.pl exp/train_t/mono0a/graph_nosp_tgpr data/dev_t exp/train_t/mono0a/decode_nosp_tgpr
filter_scps.pl: warning: some input lines were output to multiple files [OK if splitting per utt] 
decode.sh: feature type is delta
steps/diagnostic/analyze_lats.sh --cmd run.pl exp/train_t/mono0a/graph_nosp_tgpr exp/train_t/mono0a/decode_nosp_tgpr
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 18.8722966014% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 23.7315010571% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_lats.sh: see stats in exp/train_t/mono0a/decode_nosp_tgpr/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(17,150,593) and mean=249.2
steps/diagnostic/analyze_lats.sh: see stats in exp/train_t/mono0a/decode_nosp_tgpr/log/analyze_lattice_depth_stats.log
steps/align_si.sh --nj 30 --cmd run.pl data/train_s_5k data/lang_nosp exp/train_s/mono0a exp/train_s/mono0a_ali
steps/align_si.sh: feature type is delta
steps/align_si.sh: aligning data in data/train_s_5k using model from exp/train_s/mono0a, putting alignments in exp/train_s/mono0a_ali
steps/diagnostic/analyze_alignments.sh --cmd run.pl data/lang_nosp exp/train_s/mono0a_ali
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 18.2417129916% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 16.5664524155% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_alignments.sh: see stats in exp/train_s/mono0a_ali/log/analyze_alignments.log
steps/align_si.sh: done aligning data.
steps/train_deltas.sh --cmd run.pl 2000 10000 data/train_s_5k data/lang_nosp exp/train_s/mono0a_ali exp/train_s/tri1
** split_data.sh: warning, #lines is (utt2spk,feats.scp) is (5000,4888); you can 
**  use utils/fix_data_dir.sh data/train_s_5k to fix this.
steps/train_deltas.sh: accumulating tree stats
steps/train_deltas.sh: getting questions for tree-building, via clustering
steps/train_deltas.sh: building the tree
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 3 with no stats; corresponding phone list: 14 15 16 17 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 8 with no stats; corresponding phone list: 34 35 36 37 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 12 with no stats; corresponding phone list: 50 51 52 53 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 16 with no stats; corresponding phone list: 66 67 68 69 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 18 with no stats; corresponding phone list: 74 75 76 77 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 19 with no stats; corresponding phone list: 78 79 80 81 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 22 with no stats; corresponding phone list: 90 91 92 93 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 25 with no stats; corresponding phone list: 102 103 104 105 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 29 with no stats; corresponding phone list: 118 119 120 121 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 30 with no stats; corresponding phone list: 122 123 124 125 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 31 with no stats; corresponding phone list: 126 127 128 129 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 35 with no stats; corresponding phone list: 142 143 144 145 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 36 with no stats; corresponding phone list: 146 147 148 149 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 39 with no stats; corresponding phone list: 158 159 160 161 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 41 with no stats; corresponding phone list: 166 167 168 169 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 44 with no stats; corresponding phone list: 178 179 180 181 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 46 with no stats; corresponding phone list: 186 187 188 189 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 48 with no stats; corresponding phone list: 194 195 196 197 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 50 with no stats; corresponding phone list: 202 203 204 205 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 54 with no stats; corresponding phone list: 218 219 220 221 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 58 with no stats; corresponding phone list: 234 235 236 237 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 60 with no stats; corresponding phone list: 242 243 244 245 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 62 with no stats; corresponding phone list: 250 251 252 253 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 64 with no stats; corresponding phone list: 258 259 260 261 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 66 with no stats; corresponding phone list: 266 267 268 269 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 68 with no stats; corresponding phone list: 274 275 276 277 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 69 with no stats; corresponding phone list: 278 279 280 281 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 71 with no stats; corresponding phone list: 286 287 288 289 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 73 with no stats; corresponding phone list: 294 295 296 297 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 75 with no stats; corresponding phone list: 302 303 304 305 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 77 with no stats; corresponding phone list: 310 311 312 313 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 80 with no stats; corresponding phone list: 322 323 324 325 
** The warnings above about 'no stats' generally mean you have phones **
** (or groups of phones) in your phone set that had no corresponding data. **
** You should probably figure out whether something went wrong, **
** or whether your data just doesn't happen to have examples of those **
** phones. **
steps/train_deltas.sh: converting alignments from exp/train_s/mono0a_ali to use current tree
steps/train_deltas.sh: compiling graphs of transcripts
steps/train_deltas.sh: training pass 1
steps/train_deltas.sh: training pass 2
steps/train_deltas.sh: training pass 3
steps/train_deltas.sh: training pass 4
steps/train_deltas.sh: training pass 5
steps/train_deltas.sh: training pass 6
steps/train_deltas.sh: training pass 7
steps/train_deltas.sh: training pass 8
steps/train_deltas.sh: training pass 9
steps/train_deltas.sh: training pass 10
steps/train_deltas.sh: aligning data
steps/train_deltas.sh: training pass 11
steps/train_deltas.sh: training pass 12
steps/train_deltas.sh: training pass 13
steps/train_deltas.sh: training pass 14
steps/train_deltas.sh: training pass 15
steps/train_deltas.sh: training pass 16
steps/train_deltas.sh: training pass 17
steps/train_deltas.sh: training pass 18
steps/train_deltas.sh: training pass 19
steps/train_deltas.sh: training pass 20
steps/train_deltas.sh: aligning data
steps/train_deltas.sh: training pass 21
steps/train_deltas.sh: training pass 22
steps/train_deltas.sh: training pass 23
steps/train_deltas.sh: training pass 24
steps/train_deltas.sh: training pass 25
steps/train_deltas.sh: training pass 26
steps/train_deltas.sh: training pass 27
steps/train_deltas.sh: training pass 28
steps/train_deltas.sh: training pass 29
steps/train_deltas.sh: training pass 30
steps/train_deltas.sh: aligning data
steps/train_deltas.sh: training pass 31
steps/train_deltas.sh: training pass 32
steps/train_deltas.sh: training pass 33
steps/train_deltas.sh: training pass 34
steps/diagnostic/analyze_alignments.sh --cmd run.pl data/lang_nosp exp/train_s/tri1
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 16.2273476112% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 14.0780730897% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_alignments.sh: see stats in exp/train_s/tri1/log/analyze_alignments.log
1262 warnings in exp/train_s/tri1/log/update.*.log
2 warnings in exp/train_s/tri1/log/analyze_alignments.log
1693 warnings in exp/train_s/tri1/log/align.*.*.log
2 warnings in exp/train_s/tri1/log/build_tree.log
107 warnings in exp/train_s/tri1/log/init_model.log
32 warnings in exp/train_s/tri1/log/questions.log
1099 warnings in exp/train_s/tri1/log/acc.*.*.log
exp/train_s/tri1: nj=30 align prob=-97.50 over 3.29h [retry=6.0%, fail=2.9%] states=1416 gauss=10021 tree-impr=3.06
steps/train_deltas.sh: Done training system with delta+delta-delta features in exp/train_s/tri1
steps/align_si.sh --nj 30 --cmd run.pl data/train_t_5k data/lang_nosp exp/train_t/mono0a exp/train_t/mono0a_ali
steps/align_si.sh: feature type is delta
steps/align_si.sh: aligning data in data/train_t_5k using model from exp/train_t/mono0a, putting alignments in exp/train_t/mono0a_ali
steps/diagnostic/analyze_alignments.sh --cmd run.pl data/lang_nosp exp/train_t/mono0a_ali
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 16.7625308135% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 22.2337770383% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_alignments.sh: see stats in exp/train_t/mono0a_ali/log/analyze_alignments.log
steps/align_si.sh: done aligning data.
steps/train_deltas.sh --cmd run.pl 2000 10000 data/train_t_5k data/lang_nosp exp/train_t/mono0a_ali exp/train_t/tri1
** split_data.sh: warning, #lines is (utt2spk,feats.scp) is (5000,4891); you can 
**  use utils/fix_data_dir.sh data/train_t_5k to fix this.
steps/train_deltas.sh: accumulating tree stats
steps/train_deltas.sh: getting questions for tree-building, via clustering
steps/train_deltas.sh: building the tree
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 1 with no stats; corresponding phone list: 6 7 8 9 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 3 with no stats; corresponding phone list: 14 15 16 17 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 5 with no stats; corresponding phone list: 22 23 24 25 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 8 with no stats; corresponding phone list: 34 35 36 37 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 12 with no stats; corresponding phone list: 50 51 52 53 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 16 with no stats; corresponding phone list: 66 67 68 69 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 18 with no stats; corresponding phone list: 74 75 76 77 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 19 with no stats; corresponding phone list: 78 79 80 81 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 20 with no stats; corresponding phone list: 82 83 84 85 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 22 with no stats; corresponding phone list: 90 91 92 93 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 25 with no stats; corresponding phone list: 102 103 104 105 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 29 with no stats; corresponding phone list: 118 119 120 121 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 30 with no stats; corresponding phone list: 122 123 124 125 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 31 with no stats; corresponding phone list: 126 127 128 129 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 35 with no stats; corresponding phone list: 142 143 144 145 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 36 with no stats; corresponding phone list: 146 147 148 149 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 39 with no stats; corresponding phone list: 158 159 160 161 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 41 with no stats; corresponding phone list: 166 167 168 169 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 44 with no stats; corresponding phone list: 178 179 180 181 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 46 with no stats; corresponding phone list: 186 187 188 189 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 48 with no stats; corresponding phone list: 194 195 196 197 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 50 with no stats; corresponding phone list: 202 203 204 205 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 52 with no stats; corresponding phone list: 210 211 212 213 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 54 with no stats; corresponding phone list: 218 219 220 221 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 60 with no stats; corresponding phone list: 242 243 244 245 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 62 with no stats; corresponding phone list: 250 251 252 253 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 64 with no stats; corresponding phone list: 258 259 260 261 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 66 with no stats; corresponding phone list: 266 267 268 269 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 69 with no stats; corresponding phone list: 278 279 280 281 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 71 with no stats; corresponding phone list: 286 287 288 289 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 73 with no stats; corresponding phone list: 294 295 296 297 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 75 with no stats; corresponding phone list: 302 303 304 305 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 77 with no stats; corresponding phone list: 310 311 312 313 
WARNING (gmm-init-model[5.5]:InitAmGmm():gmmbin/gmm-init-model.cc:55) Tree has pdf-id 80 with no stats; corresponding phone list: 322 323 324 325 
** The warnings above about 'no stats' generally mean you have phones **
** (or groups of phones) in your phone set that had no corresponding data. **
** You should probably figure out whether something went wrong, **
** or whether your data just doesn't happen to have examples of those **
** phones. **
steps/train_deltas.sh: converting alignments from exp/train_t/mono0a_ali to use current tree
steps/train_deltas.sh: compiling graphs of transcripts
steps/train_deltas.sh: training pass 1
steps/train_deltas.sh: training pass 2
steps/train_deltas.sh: training pass 3
steps/train_deltas.sh: training pass 4
steps/train_deltas.sh: training pass 5
steps/train_deltas.sh: training pass 6
steps/train_deltas.sh: training pass 7
steps/train_deltas.sh: training pass 8
steps/train_deltas.sh: training pass 9
steps/train_deltas.sh: training pass 10
steps/train_deltas.sh: aligning data
steps/train_deltas.sh: training pass 11
steps/train_deltas.sh: training pass 12
steps/train_deltas.sh: training pass 13
steps/train_deltas.sh: training pass 14
steps/train_deltas.sh: training pass 15
steps/train_deltas.sh: training pass 16
steps/train_deltas.sh: training pass 17
steps/train_deltas.sh: training pass 18
steps/train_deltas.sh: training pass 19
steps/train_deltas.sh: training pass 20
steps/train_deltas.sh: aligning data
steps/train_deltas.sh: training pass 21
steps/train_deltas.sh: training pass 22
steps/train_deltas.sh: training pass 23
steps/train_deltas.sh: training pass 24
steps/train_deltas.sh: training pass 25
steps/train_deltas.sh: training pass 26
steps/train_deltas.sh: training pass 27
steps/train_deltas.sh: training pass 28
steps/train_deltas.sh: training pass 29
steps/train_deltas.sh: training pass 30
steps/train_deltas.sh: aligning data
steps/train_deltas.sh: training pass 31
steps/train_deltas.sh: training pass 32
steps/train_deltas.sh: training pass 33
steps/train_deltas.sh: training pass 34
steps/diagnostic/analyze_alignments.sh --cmd run.pl data/lang_nosp exp/train_t/tri1
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 16.228430567% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 20.0249532127% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_alignments.sh: see stats in exp/train_t/tri1/log/analyze_alignments.log
2 warnings in exp/train_t/tri1/log/analyze_alignments.log
98 warnings in exp/train_t/tri1/log/init_model.log
782 warnings in exp/train_t/tri1/log/acc.*.*.log
1160 warnings in exp/train_t/tri1/log/align.*.*.log
34 warnings in exp/train_t/tri1/log/questions.log
1353 warnings in exp/train_t/tri1/log/update.*.log
3 warnings in exp/train_t/tri1/log/build_tree.log
exp/train_t/tri1: nj=30 align prob=-87.55 over 2.17h [retry=3.5%, fail=2.6%] states=857 gauss=10018 tree-impr=2.74
steps/train_deltas.sh: Done training system with delta+delta-delta features in exp/train_t/tri1
tree-info exp/train_s/tri1/tree 
tree-info exp/train_s/tri1/tree 
fstcomposecontext --context-size=3 --central-position=1 --read-disambig-syms=data/lang_nosp_test_tgpr/phones/disambig.int --write-disambig-syms=data/lang_nosp_test_tgpr/tmp/disambig_ilabels_3_1.int data/lang_nosp_test_tgpr/tmp/ilabels_3_1.10034 data/lang_nosp_test_tgpr/tmp/LG.fst 
fstisstochastic data/lang_nosp_test_tgpr/tmp/CLG_3_1.fst 
0 -0.0225837
[info]: CLG not stochastic.
make-h-transducer --disambig-syms-out=exp/train_s/tri1/graph_nosp_tgpr/disambig_tid.int --transition-scale=1.0 data/lang_nosp_test_tgpr/tmp/ilabels_3_1 exp/train_s/tri1/tree exp/train_s/tri1/final.mdl 
fstrmepslocal 
fstrmsymbols exp/train_s/tri1/graph_nosp_tgpr/disambig_tid.int 
fstdeterminizestar --use-log=true 
fsttablecompose exp/train_s/tri1/graph_nosp_tgpr/Ha.fst data/lang_nosp_test_tgpr/tmp/CLG_3_1.fst 
fstminimizeencoded 
fstisstochastic exp/train_s/tri1/graph_nosp_tgpr/HCLGa.fst 
0.000487832 -0.0565551
HCLGa is not stochastic
add-self-loops --self-loop-scale=0.1 --reorder=true exp/train_s/tri1/final.mdl exp/train_s/tri1/graph_nosp_tgpr/HCLGa.fst 
steps/decode.sh --nj 4 --cmd run.pl exp/train_s/tri1/graph_nosp_tgpr data/dev_s exp/train_s/tri1/decode_nosp_tgpr
decode.sh: feature type is delta
steps/diagnostic/analyze_lats.sh --cmd run.pl exp/train_s/tri1/graph_nosp_tgpr exp/train_s/tri1/decode_nosp_tgpr
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 8.0684596577% of the time at utterance begin.  This may not be optimal.
analyze_phone_length_stats.py: WARNING: optional-silence SIL is seen only 2.20048899756% of the time at utterance end.  This may not be optimal.
steps/diagnostic/analyze_lats.sh: see stats in exp/train_s/tri1/decode_nosp_tgpr/log/analyze_alignments.log
Overall, lattice depth (10,50,90-percentile)=(1,10,70) and mean=26.1
steps/diagnostic/analyze_lats.sh: see stats in exp/train_s/tri1/decode_nosp_tgpr/log/analyze_lattice_depth_stats.log

[laurensw75]

Can you explain exactly where the run failed? I see some warnings. The missing stats may be a problem, but it's been a while since I've ran this particular configuration so I'm not sure if this would be fatal.

[Aariciah]

Thanks for the quick reply. I honestly don't understand much of what is happening so the above is the whole log...

[laurensw75]

The log shows the output of the stages up to (and including) 2. You should now have a few 'wer(some number)' files in you exp/train(s or t)/tri1/decode_nosp_tgpr(_tg) directories. If these exist and have error rates well below 100% (they will be quite high since it's only the first stages of training), then you should be good to go for stage 3 and onwards. Unless there is an error (not warning) somewhere, but that doesn't show up in this logfile.

[Aariciah]

the files and folders in exp/train_s/tri1/decode_nosp_tgpr are lat.1.gz lat.2.gz lat.3.gz lat.4.gz log num_jobs scoring

There is no decode folder in train_t

Does that help?

[laurensw75]

Ok, so the decoding didn't work on the 'S' data. You should be able to find a log under the exp/train_s/tri1/decode_nosp_tgpr/log folder in decode.*.log. That will contain the error message.

[Aariciah]

I got 4 such files and all state "done X utterances, failed for 0" and exit code 0. I'm very confused.

decode.4.log as example:

# gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/train_s/tri1/graph_nosp_t$
# Started at Fri Apr 22 15:09:22 CEST 2022
#
gmm-latgen-faster --max-active=7000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.083333 --allow-partial=true --word-symbol-table=exp/train_s/tri1/graph_nosp_tgp$
apply-cmvn --utt2spk=ark:data/dev_s/split4/4/utt2spk scp:data/dev_s/split4/4/cmvn.scp scp:data/dev_s/split4/4/feats.scp ark:-
add-deltas ark:- ark:-
N02007-fn002646.1 nederland viert vandaag bevrijdingsdag
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.1 is -8.92132 over 185 f$
N02007-fn002646.2 het officiële startsein wordt aan het eind van de ochtend gegevens nederland
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.2 is -9.00432 over 318 f$
N02007-fn002646.3 in stadhuis van meer dan wat de voorzitter van de eerste kamer korthals altes een lezing
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.3 is -8.87039 over 415 f$
LOG (apply-cmvn[5.5]:main():featbin/apply-cmvn.cc:162) Applied cepstral mean normalization to 8 utterances, errors on 0
N02007-fn002646.4 hopelijk ook voor 't wordt
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.4 is -9.21374 over 130 f$
N02007-fn002646.5 daarna zijn op verschillende plaatsen de klanten studietijd er behalve in elkaar zit gebieden
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.5 is -9.06671 over 417 f$
N02007-fn002646.6 vanavond wordt een nationale viering van bevrijdingsdag afgesloten met een concert op de amstel vlak voor theatercafé
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.6 is -9.02349 over 545 f$
N02007-fn002646.7 gisteravond zijn op verschillende plaatsen de dode herdacht die sinds de tweede wereldoorlog zijn gevallen
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.7 is -8.98328 over 410 f$
N02007-fn002646.8 het thema van de dodenherdenking en van bevrijdingsdag is vrijheid is een leven zonder angst
LOG (gmm-latgen-faster[5.5]:DecodeUtteranceLatticeFaster():decoder/decoder-wrappers.cc:375) Log-like per frame for utterance N02007-fn002646.8 is -8.8384 over 441 fr$
LOG (gmm-latgen-faster[5.5]:main():gmmbin/gmm-latgen-faster.cc:176) Time taken 10.3678s: real-time factor assuming 100 frames/sec is 0.362385
LOG (gmm-latgen-faster[5.5]:main():gmmbin/gmm-latgen-faster.cc:179) Done 8 utterances, failed for 0
LOG (gmm-latgen-faster[5.5]:main():gmmbin/gmm-latgen-faster.cc:181) Overall log-likelihood per frame is -8.9732 over 2861 frames.
# Accounting: time=10 threads=1
# Ended (code 0) at Fri Apr 22 15:09:32 CEST 2022, elapsed time 10 seconds

[Aariciah]

I tried running it again and a bunch of lines were added to the fail log:

some looked like this: run.pl: Error opening log file exp/train_s/tri1/log/acc_tree.*.log at /home/s4951638/kaldi/egs/kaldi_egs_CGN/s5/utils/run.pl line 275.

others like this: bash: line 1: exp/train_s/tri1/log/acc_tree.*.log: Cannot send after transport endpoint shutdown

so something seems to go wrong here (run.pl lines 274 and 275)

system("mkdir -p `dirname $logfile` 2>/dev/null");
open(F, ">$logfile") || die "run.pl: Error opening log file $logfile";

[laurensw75]

Have you successfully ran other recipes?

[Aariciah]

I haven't run other recipes yet. Learning how to work with Kaldi as I go through this one.

[laurensw75]

I think you're running into some non-Kaldi/recipe issues here, so maybe you can debug them first. The Librispeech (and mini) recipes can be run with just publicly available data, and are well established and debugged. You may want to try those first to make sure Kaldi and your general setup are working well.