Closed zhangjiulong closed 8 years ago
Please check if you have log files exp/make_fbank/train/*.log, and if you see error messages in them.
Hi @yajiemiao there is no error log in the dir, but after I runed qhost -q I found all.q status is like this
all.q BIP 0/0/48
is this means that there is no resourece to run the jobe submited ? thx
The jobs are in error state, not assigned to any node yet.
Check qstat -j 28 to see the detailed error.
在 2016/8/1 10:58, john 写道:
Hi @yajiemiao https://github.com/yajiemiao there is no error log in the dir, but after I runed qhost -q I found all.q status is like this ··· all.q BIP 0/0/48 ··· is this means that there is no resourece to run the jobe submited ? thx
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/78#issuecomment-236478944, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKpxPOxtBU6-DFNnqwIcJqciNN471K-ks5qbWC-gaJpZM4JX-yW.
Hi @naxingyu I found something useful, the error log is lik this:
job-array tasks: 1-20:1
error reason 1: 07/29/2016 15:58:05 [1003:17512]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 2: 07/29/2016 15:58:05 [1003:17515]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 3: 07/29/2016 15:58:05 [1003:17528]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 4: 07/29/2016 15:58:05 [1003:17530]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 5: 07/29/2016 15:58:05 [1003:17519]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 7: 07/29/2016 15:58:05 [1003:17526]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 9: 07/29/2016 15:58:05 [1003:17521]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 11: 07/29/2016 15:58:05 [1003:17532]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 13: 07/29/2016 15:58:05 [1003:17531]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 15: 07/29/2016 15:58:05 [1003:17522]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 17: 07/29/2016 15:58:05 [1003:17529]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
error reason 19: 07/29/2016 15:58:05 [1003:17527]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit
there is a quote before the path, but i don't know how this quote comes. My cmd is like this:
export train_cmd="queue.pl -cwd -q all.q -l arch=*64"
export decode_cmd="queue.pl -cwd -q all.q -l arch=*64,mem_free=2G,ram_free=2G"
export mkgraph_cmd="queue.pl -cwd -q all.q -l arch=*64,ram_free=4G,mem_free=4G"
export big_memory_cmd="queue.pl -cwd -q all.q -l arch=*64,ram_free=8G,mem_free=8G"
export cuda_cmd="queue.pl -cwd -q all.q -l gpu=1"
When I changed the cmd to run.pl the program runs ok thx
Check your summitted shell in "q" dir.
在 2016/8/1 13:45, john 写道:
Hi @naxingyu https://github.com/naxingyu I found something useful, the error log is lik this:
|job-array tasks: 1-20:1 error reason 1: 07/29/2016 15:58:05 [1003:17512]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 2: 07/29/2016 15:58:05 [1003:17515]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 3: 07/29/2016 15:58:05 [1003:17528]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 4: 07/29/2016 15:58:05 [1003:17530]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 5: 07/29/2016 15:58:05 [1003:17519]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 7: 07/29/2016 15:58:05 [1003:17526]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 9: 07/29/2016 15:58:05 [1003:17521]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 11: 07/29/2016 15:58:05 [1003:17532]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 13: 07/29/2016 15:58:05 [1003:17531]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 15: 07/29/2016 15:58:05 [1003:17522]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 17: 07/29/2016 15:58:05 [1003:17529]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit error reason 19: 07/29/2016 15:58:05 [1003:17527]: error: can't open output file "/home/kaldi/git/eesen/asr_egs/timit |
there is a quote before the path, but i don't know how this quote comes. thx
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/78#issuecomment-236494297, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKpxI7zmtHtSe70-w85kfG_cnl46Db4ks5qbYfqgaJpZM4JX-yW.
Hi @naxingyu suummited shell is like this, please help me check the shell, thx
#!/bin/bash
cd /home/kaldi/git/eesen/asr_egs/timit/vc1
. ./path.sh
( echo '#' Running on `hostname`
echo '#' Started at `date`
echo -n '# '; cat <<EOF
extract-segments scp,p:data/train/wav.scp exp/make_fbank/train/segments.${SGE_TASK_ID} ark:- | compute-fbank-feats --verbose=2 --config=conf/fbank.conf ark:- ark:- | copy-feats --compress=true ark:- ark,scp:/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_train.\
${SGE_TASK_ID}.ark,/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_train.${SGE_TASK_ID}.scp
EOF
) >exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log
time1=`date +"%s"`
( extract-segments scp,p:data/train/wav.scp exp/make_fbank/train/segments.${SGE_TASK_ID} ark:- | compute-fbank-feats --verbose=2 --config=conf/fbank.conf ark:- ark:- | copy-feats --compress=true ark:- ark,scp:/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_tra\
in.${SGE_TASK_ID}.ark,/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_train.${SGE_TASK_ID}.scp ) 2>>exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log >>exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log
ret=$?
time2=`date +"%s"`
echo '#' Accounting: time=$(($time2-$time1)) threads=1 >>exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log
echo '#' Finished at `date` with status $ret >>exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log
[ $ret -eq 137 ] && exit 100;
touch exp/make_fbank/train/q/done.30991.$SGE_TASK_ID
exit $[$ret ? 1 : 0]
## submitted with:
# qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64* -o exp/make_fbank/train/q/make_fbank_train.log -q all.q -l arch=*64 -t 1:20 /home/kaldi/git/eesen/asr_egs/timit/vc1/exp/make_fbank/train/q/make_fbank_train.sh >>exp/make_fbank/train/q/make_fbank_train.log 2>&1
Seems a NFS problem. Is the master on node1 or node2? The master and execute nodes should have shared file system.
在 2016/8/1 14:46, john 写道:
Hi @naxingyu https://github.com/naxingyu suummited shell is like this, please help me check the shell, thx
|#!/bin/bash cd /home/kaldi/git/eesen/asr_egs/timit/vc1 . ./path.sh ( echo '#' Running on
hostname
echo '#' Started atdate
echo -n '# '; cat <<EOF extract-segments scp,p:data/train/wav.scp exp/make_fbank/train/segments.${SGE_TASK_ID} ark:- | compute-fbank-feats --verbose=2 --config=conf/fbank.conf ark:- ark:- | copy-feats --compress=true ark:- ark,scp:/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_train.\ ${SGE_TASK_ID}.ark,/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_train.${SGE_TASK_ID}.scp EOF ) >exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log time1=date +"%s"
( extract-segments scp,p:data/train/wav.scp exp/make_fbank/train/segments.${SGE_TASK_ID} ark:- | compute-fbank-feats --verbose=2 --config=conf/fbank.conf ark:- ark:- | copy-feats --compress=true ark:- ark,scp:/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_tra\ in.${SGE_TASK_ID}.ark,/home/kaldi/git/eesen/asr_egs/timit/vc1/fbank/raw_fbank_train.${SGE_TASK_ID}.scp ) 2>>exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.logexp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log ret=$? time2=
date +"%s"
echo '#' Accounting: time=$(($time2-$time1)) threads=1 >>exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log echo '#' Finished atdate
with status $ret exp/make_fbank/train/make_fbank_train.$SGE_TASK_ID.log [ $ret -eq 137 ] && exit 100; touch exp/make_fbank/train/q/done.30991.$SGE_TASK_ID exit $[$ret ? 1 : 0] ## submitted with: # qsub -v PATH -cwd -S /bin/bash -j y -l arch=64 -o exp/make_fbank/train/q/make_fbank_train.log -q all.q -l arch=*64 -t 1:20 /home/kaldi/git/eesen/asr_egs/timit/vc1/exp/make_fbank/train/q/make_fbank_train.sh exp/make_fbank/train/q/make_fbank_train.log 2>&1 |— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/78#issuecomment-236501902, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKpxMVRV0W1AGnhtBjE_f7NvFMOt7Fxks5qbZY8gaJpZM4JX-yW.
node1 is master and execute node and node2 is execute node nfs is anoter node, called node3, all my data is on node3 I think if it is the nfs problem the run.pl should runs error, I'm not sure wether it it right or not, but I runed run.pl ok.
all my data on node3 is mounted to node1 and node2, essen is built on node1 and node2 separately
NFS is not just for accessing the raw data. It's set up for sharing the experiment files among execute nodes. Different nodes should have the same write permission to a shared exp dir. And that's not true in your cluster. Have a look at http://kaldi-asr.org/doc/queue.html. run.pl is for a local setup.
在 2016/8/1 15:07, john 写道:
node1 is master and execute node and node2 is execute node nfs is anoter node, called node3, all my data is on node3 I think if it is the nfs problem the run.pl should runs error, I'm not sure wether it it right or not, but I runed run.pl ok.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/78#issuecomment-236504987, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKpxKD7Ws5qSzMwyGGv-3KxQfzMnAWsks5qbZsagaJpZM4JX-yW.
Hi @naxingyu do you means exp dir, fbank dir and data dir created by eesen should be shared with the same permission, that is the three dirs should be on the nfs?
Thanks @naxingyu ,As you said It is nfs' problem.
Hi, I am facing the same problem while setting up the cluster. Could you please guide me on how you solved the problem. This was the error which I received
error reason 2: 04/26/2018 11:19:28 [1002:22648]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 4: 04/26/2018 11:19:28 [1002:22651]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 6: 04/26/2018 11:19:28 [1002:22652]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 8: 04/26/2018 11:19:28 [1002:22653]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 9: 04/26/2018 11:19:43 [1002:22674]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 10: 04/26/2018 11:19:43 [1002:22677]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 11: 04/26/2018 11:19:43 [1002:22678]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 12: 04/26/2018 11:19:43 [1002:22679]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 14: 04/26/2018 11:19:58 [1002:22701]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 16: 04/26/2018 11:19:58 [1002:22702]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 18: 04/26/2018 11:19:58 [1002:22703]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 20: 04/26/2018 11:19:58 [1002:22704]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5
Thanks.
Regards, Raghav
The log says the executing node can not access the folder. It looks like a permission issue.
Best, Xingyu Na
On Thu, Apr 26, 2018 at 5:41 PM, raghav-menon notifications@github.com wrote:
Hi, I am facing the same problem while setting up the cluster. Could you please guide me on how you solved the problem. This was the error which I received
error reason 2: 04/26/2018 11:19:28 [1002:22648]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 4: 04/26/2018 11:19:28 [1002:22651]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 6: 04/26/2018 11:19:28 [1002:22652]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 8: 04/26/2018 11:19:28 [1002:22653]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 9: 04/26/2018 11:19:43 [1002:22674]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 10: 04/26/2018 11:19:43 [1002:22677]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 11: 04/26/2018 11:19:43 [1002:22678]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 12: 04/26/2018 11:19:43 [1002:22679]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 14: 04/26/2018 11:19:58 [1002:22701]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 16: 04/26/2018 11:19:58 [1002:22702]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 18: 04/26/2018 11:19:58 [1002:22703]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5 error reason 20: 04/26/2018 11:19:58 [1002:22704]: error: can't chdir to /home/astik/kaldi-trunk/kaldi/egs/swahili/s5
Thanks.
Regards, Raghav
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/srvk/eesen/issues/78#issuecomment-384578760, or mute the thread https://github.com/notifications/unsubscribe-auth/ADKpxLd_ENKAe4d5mbZQqRarGkPgJnzVks5tsZY6gaJpZM4JX-yW .
Hi, I am facing the same problem while setting up the cluster. Using "qstat -j 29" checking the error as following:
script_file: /home/gpu/software/kaldi-20181120/egs/thchs30/s5/exp/make_mfcc/train/q/make_mfcc_train.sh job-array tasks: 1-8:1 error reason 4: 01/07/2019 15:19:42 [0:14111]: error: can't open output file "/home/gpu/software/kaldi-20181120/egs/ error reason 5: 01/07/2019 15:19:42 [0:14112]: error: can't open output file "/home/gpu/software/kaldi-20181120/egs/ error reason 6: 01/07/2019 15:19:42 [0:14113]: error: can't open output file "/home/gpu/software/kaldi-20181120/egs/ error reason 7: 01/07/2019 15:19:42 [0:14114]: error: can't open output file "/home/gpu/software/kaldi-20181120/egs/ error reason 8: 01/07/2019 15:19:42 [0:14115]: error: can't open output file "/home/gpu/software/kaldi-20181120/egs/ scheduling info: Job is in error state
I had checked the output file. It exists. It looks like a permission issue like @naxingyu said. But I don't know how to figure out. Could anyone please guide me on how to solve the problem. Thx.
Hi I get a gridengine cluster which has two nodes(node1,node2), each node has 3 gpus. Also I have another node(node3) as nfs and all the wav data is on node3 then I mount the data dir to node1 and node2(node 1 and node2 only can read the mount dir, can not write)
On node1 and node2 I build eesen separately, and modify the cmd.sh to queue format as follows:
But when I runned ./run_ctc_phn.sh on node1 the screen stops on the make fbank step as follows:
But no gpu or cpu is running and I runned qstat shows that some error happened like this
But no error log is found Please give me some suggestion, thanks very much.