Closed lifeiteng closed 1 year ago
part of kaldi JOBs failed
Jun 1 23:10:36 kernel: [ 3284.095243] compile-train-g[10881]: segfault at 5567b3ba2000 ip 00007fce2a270ffc sp 00007fff078fc630 error 4 in libkaldi-decoder.so[7fce2a248000+13f000]
What version of kaldi is installed in the conda environment? We updated it to 5.5.1068 a few weeks ago. If you're running that, you could try installing the 5.5.1016 and see if that fixes it? (or upgrading it to the newest version if you're using 5.5.1016 still)
@mmcauliffe kaldi version is 5.5.1074
commit id 71f38e62cad01c3078555bfe78d0f3a527422d75
Oh, did you compile it from source? In that case, the error's from changes since 1068 or in how it was compiled, so I would focus on fixing that or install the conda version.
Oh, did you compile it from source? In that case, the error's from changes since 1068 or in how it was compiled, so I would focus on fixing that or install the conda version.
trying to use docker version but new errors
Job 8 encountered an error:
Traceback (most recent call last):
File "/env/lib/python3.10/site-packages/montreal_forced_aligner/abc.py", line 89, in run
yield from self._run()
File "/env/lib/python3.10/site-packages/montreal_forced_aligner/corpus/features.py", line 705, in _run
self.check_call(copy_proc)
File "/env/lib/python3.10/site-packages/montreal_forced_aligner/abc.py", line 116, in check_call
raise KaldiProcessingError([self.log_path])
montreal_forced_aligner.exceptions.KaldiProcessingError: KaldiProcessingError:
There were 1 job(s) with errors when running Kaldi binaries.
See the log files below for more information.
/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/log/generate_final_features.8.log
$ cat /home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/log/generate_final_features.8.log
/env/bin/apply-cmvn --utt2spk=ark:/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/utt2spk.8.scp scp:/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/cmvn.8.scp scp:/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/feats.8.scp ark:-
/env/bin/copy-feats --compress=true ark:- ark,scp:/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/final_features.8.ark,/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/final_features.8.scp
ERROR (apply-cmvn[5.5.1016]:HasKey():util/kaldi-table-inl.h:2639) Attempting to read key 101-68298, which is not present in utt2spk map or similar map being read from ark:/home/feiteng/Documents/MFA/mls-english_train1_splits00/mls-english_train1_splits00/split8/utt2spk.8.scp
kaldi::KaldiFatalErrorLOG (copy-feats[5.5.1016]:main():featbin/copy-feats.cc:143) Copied 0 feature matrices.
split8/utt2spk.8.scp
missing data.
Oh, did you compile it from source? In that case, the error's from changes since 1068 or in how it was compiled, so I would focus on fixing that or install the conda version.
version 1068, still error
There is a problem with the machine.
pg_log_global.txt