I've already started a localhost parasol-hub according to a manuscript from http://genomewiki.ucsc.edu/index.php/Parasol_job_control_system.
Then I started the doBlastzChainNet.pl, and it ran for about 30 minutes before crushes. It produced a log file named do.log.
I check the log file, found two parts may be indicating an error, and they are as bellow:
do.log line8~line13
profile db not found in sqlProfileToMyCnf() -- failed for file /GPUFS/sysu_mhwang_1/.hgsql.cnf-m2VqGT failed with errno 2
profile db not found in sqlProfileToMyCnf() -- failed for file /GPUFS/sysu_mhwang_1/.hgsql.cnf-C7Hy3D failed with errno 2
HgStepManager: executing from step 'partition' through step 'syntenicNet'.
HgStepManager: executing step 'partition' Wed Dec 29 09:02:38 2021.
sort: write failed: 'standard output': Broken pipe
sort: write error
do.log line536~
gensub2 ptri.lst pcla.lst gsub jobList
para make jobList
Checking input files
14535 jobs written to /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/run.blastz/batch
14535 jobs in batch
0 jobs (including everybody's) in Parasol queue or running.
Checking finished jobs
updated job database on disk
Pushed Jobs: 14535
total sick machines: 1 failures: 5
Checking job status 0 minutes after launch
14535 jobs in batch
0 jobs (including everybody's) in Parasol queue or running.
Sick Batch: consecutive crashes (36) >= sick batch threshold (25)
Checking finished jobs
updated job database on disk
total sick machines: 1 failures: 36
Sick batch! will sleep 10 minutes, clear sick nodes and retry
Told hub to clear sick nodes
...
...
...
Checking job status 33 minutes after launch
14535 jobs in batch
0 jobs (including everybody's) in Parasol queue or running.
Sick Batch: consecutive crashes (36) >= sick batch threshold (25)
Checking finished jobs
updated job database on disk
Batch failed after 4 tries on /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/scripts/blastz-run-ucsc -outFormat psl /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/ptri.2bit:NC_059295.1:0-32110000 /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/pcla.2bit:LG03:0-10000000 ../DEF ../psl/ptri.2bit:NC_059295.1:0-32110000/ptri.2bit:NC_059295.1:0-32110000_pcla.2bit:LG03:0-10000000.psl
Command failed:
ssh -x -o 'StrictHostKeyChecking = no' -o 'BatchMode = yes' localhost nice /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/run.blastz/doClusterRun.csh
real 33m9.299s
user 0m0.082s
sys 0m0.069s
OS: Linux
Browser Putty
Version Ubuntu 18.04.5
Additional context
I also checked my parasol log file, and in the file 192.168.206.146.2021-12-29T07\:56.log.node:
Thank you for making all these bioinfo scripts, they are very useful! I'm constructing some karyotype chromosomes now using some de-novo genome assembly, and I'm trying to adopt the doBlastzChainNet.pl into my pipeline, following a manuscript from http://genomewiki.ucsc.edu/index.php/DoBlastzChainNet.pl#The_new_streamline_pairLastz_script.
I've already started a localhost parasol-hub according to a manuscript from http://genomewiki.ucsc.edu/index.php/Parasol_job_control_system. Then I started the doBlastzChainNet.pl, and it ran for about 30 minutes before crushes. It produced a log file named do.log.
I check the log file, found two parts may be indicating an error, and they are as bellow:
do.log line8~line13 profile db not found in sqlProfileToMyCnf() -- failed for file /GPUFS/sysu_mhwang_1/.hgsql.cnf-m2VqGT failed with errno 2 profile db not found in sqlProfileToMyCnf() -- failed for file /GPUFS/sysu_mhwang_1/.hgsql.cnf-C7Hy3D failed with errno 2 HgStepManager: executing from step 'partition' through step 'syntenicNet'. HgStepManager: executing step 'partition' Wed Dec 29 09:02:38 2021. sort: write failed: 'standard output': Broken pipe sort: write error
do.log line536~ gensub2 ptri.lst pcla.lst gsub jobList para make jobList Checking input files 14535 jobs written to /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/run.blastz/batch 14535 jobs in batch 0 jobs (including everybody's) in Parasol queue or running. Checking finished jobs updated job database on disk Pushed Jobs: 14535 total sick machines: 1 failures: 5
Checking job status 0 minutes after launch 14535 jobs in batch 0 jobs (including everybody's) in Parasol queue or running. Sick Batch: consecutive crashes (36) >= sick batch threshold (25) Checking finished jobs updated job database on disk total sick machines: 1 failures: 36 Sick batch! will sleep 10 minutes, clear sick nodes and retry Told hub to clear sick nodes ... ... ...
Checking job status 33 minutes after launch 14535 jobs in batch 0 jobs (including everybody's) in Parasol queue or running. Sick Batch: consecutive crashes (36) >= sick batch threshold (25) Checking finished jobs updated job database on disk Batch failed after 4 tries on /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/scripts/blastz-run-ucsc -outFormat psl /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/ptri.2bit:NC_059295.1:0-32110000 /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/pcla.2bit:LG03:0-10000000 ../DEF ../psl/ptri.2bit:NC_059295.1:0-32110000/ptri.2bit:NC_059295.1:0-32110000_pcla.2bit:LG03:0-10000000.psl Command failed: ssh -x -o 'StrictHostKeyChecking = no' -o 'BatchMode = yes' localhost nice /GPUFS/sysu_mhwang_1/sysu_mhwang_1/zwei_liuziwei/02_parasol/data/genomes/run.blastz/doClusterRun.csh
real 33m9.299s user 0m0.082s sys 0m0.069s
OS: Linux
Additional context I also checked my parasol log file, and in the file 192.168.206.146.2021-12-29T07\:56.log.node:
2021/12/29 07:56:39: info: starting paraNode on zwei-mysql5-29153733-5c7c7d66c4-gdqnz 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-104-1 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-105-1 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-103-1 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-102-1 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-101-1 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-107-1 2021/12/29 08:26:10: warn: duplicate packet filtered out: 192.168.206.146-9694-108-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-136-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-138-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-139-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-137-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-140-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-143-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-142-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-141-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-144-1 2021/12/29 09:03:10: warn: duplicate packet filtered out: 192.168.206.146-9694-145-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-175-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-176-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-178-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-177-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-179-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-180-1 2021/12/29 09:13:40: warn: duplicate packet filtered out: 192.168.206.146-9694-181-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-211-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-205-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-207-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-206-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-212-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-209-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-208-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-210-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-213-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-214-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-215-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-216-1 2021/12/29 09:24:10: warn: duplicate packet filtered out: 192.168.206.146-9694-217-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-246-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-248-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-249-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-250-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-251-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-252-1 2021/12/29 09:34:55: warn: duplicate packet filtered out: 192.168.206.146-9694-253-1