Closed Mcfoofa closed 10 years ago
Mcfoofa
commented
10 years ago Here is the rest of the cactus.log that preceded the ALERT shown above:
2013-11-21 20:26:40.344724: Beginning Progressive Cactus Alignment
Got message from job at time: 1385101026.67 : Blocking on ktserver
benedictpaten
commented
10 years ago Hi,
Looks like the blossum tool went awry. Could you rerun, but edit the config file (should be in the foo/cactus/progressive/cactus_progressive_workflow_config.file, where foo is the path to the cactus submodule directory), to replace the line: matchingAlgorithm="blossom5" which matchingAlgorithm="greedy".
This will test if it is truly an issue with blossum, or a more general issue.
Thanks!
Benedict
On Fri, Nov 22, 2013 at 7:24 AM, Hussain notifications@github.com wrote:
Here is the rest of the cactus.log that preceded the ALERT shown above:
2013-11-21 20:26:40.344724: Beginning Progressive Cactus Alignment
Got message from job at time: 1385101026.67 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/work/jobTree/jobs/gTD2/tmp_ruCMcy0z5a/tmp_cgJhQMAdPy_kill.txt Got message from job at time: 1385101043.0 : Input-sample: ./work/sequenceData/dmel-all-chromosome-r5.53.fasta Total-sequences: 15 Total-length: 168736537 Proportion-repeat-masked: 0.131561 ProportionNs: 0.037744 N50: 142743 Median-sequence-length: 3288761 Got message from job at time: 1385101043.0 : Input-sample: ./work/sequenceData/dsim-all-chromosome-r1.4.fasta Total-sequences: 10005 Total-length: 137828247 Proportion-repeat-masked: 0.099978 ProportionNs: 0.076833 N50: 165255 Median-sequence-length: 1223 Got message from job at time: 1385101043.0 : Starting caf phase target with index 0 at 1385101032.26 seconds (recursing = 1) Got message from job at time: 1385101043.0 : Adding an oversize flower for target class and stats flower name: 0 total bases: 306564784 total-ends: 20040 total-caps: 20040 max-end-degree: 1 max-adjacency-length: 29004657 total-blocks: 0 total-groups: 1 total-edges: 10020 total-free-ends: 20040 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1385109215.83 : Pinch graph component with 10921 nodes and 11949 edges is being split up by breaking 964 edges to reduce size to less than 714 max, but found 0 pointless edges Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527232, header Uextra with length 29004656 and 2234908 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527227, header 3R with length 27905053 and 22146229 total bases aligned and 58635 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527212, header 3L with length 24543557 and 18315740 total bases aligned and 188857 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527202, header 2L with length 23011544 and 17348961 total bases aligned and 230802 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527207, header X with length 22422827 and 14155982 total bases aligned and 330194 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527222, header 2R with length 21146708 and 15262809 total bases aligned and 318738 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527217, header 4 with length 1351857 and 552287 total bases aligned and 36014 bases aligned to other chromosome threads Got message from job at time: 1385110476.96 : Starting bar phase target with index 0 at 1385109216.1 seconds (recursing = 1) Got message from job at time: 1385110477.04 : Adding an oversize flower for target class and stats flower name: 2133298848490051438 total bases: 3454268 total-ends: 796 total-caps: 2162 max-end-degree: 30 max-adjacency-length: 123624 total-blocks: 0 total-groups: 1 total-edges: 798 total-free-ends: 62 total-attached-ends: 734 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889605 total bases: 3434216 total-ends: 840 total-caps: 2298 max-end-degree: 29 max-adjacency-length: 141481 total-blocks: 0 total-groups: 1 total-edges: 849 total-free-ends: 33 total-attached-ends: 807 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889681 total bases: 3335596 total-ends: 811 total-caps: 2216 max-end-degree: 22 max-adjacency-length: 70229 total-blocks: 0 total-groups: 1 total-edges: 818 total-free-ends: 30 total-attached-ends: 781 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889685 total bases: 3566142 total-ends: 902 total-caps: 2316 max-end-degree: 25 max-adjacency-length: 79970 total-blocks: 0 total-groups: 1 total-edges: 907 total-free-ends: 36 total-attached-ends: 866 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889689 total bases: 3670542 total-ends: 863 total-caps: 2306 max-end-degree: 22 max-adjacency-length: 54690 total-blocks: 0 total-groups: 1 total-edges: 869 total-free-ends: 35 total-attached-ends: 828 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889821 total bases: 2888874 total-ends: 824 total-caps: 2166 max-end-degree: 41 max-adjacency-length: 71147 total-blocks: 0 total-groups: 1 total-edges: 842 total-free-ends: 74 total-attached-ends: 750 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889825 total bases: 3629179 total-ends: 816 total-caps: 2194 max-end-degree: 23 max-adjacency-length: 95616 total-blocks: 0 total-groups: 1 total-edges: 820 total-free-ends: 31 total-attached-ends: 785 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889913 total bases: 3525679 total-ends: 911 total-caps: 2400 max-end-degree: 50 max-adjacency-length: 37444 total-blocks: 0 total-groups: 1 total-edges: 922 total-free-ends: 40 total-attached-ends: 871 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889917 total bases: 3347355 total-ends: 876 total-caps: 2368 max-end-degree: 29 max-adjacency-length: 39264 total-blocks: 0 total-groups: 1 total-edges: 886 total-free-ends: 31 total-attached-ends: 845 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889925 total bases: 3263116 total-ends: 802 total-caps: 2120 max-end-degree: 24 max-adjacency-length: 116911 total-blocks: 0 total-groups: 1 total-edges: 811 total-free-ends: 45 total-attached-ends: 757 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889929 total bases: 3061174 total-ends: 791 total-caps: 2038 max-end-degree: 19 max-adjacency-length: 106442 total-blocks: 0 total-groups: 1 total-edges: 794 total-free-ends: 117 total-attached-ends: 674 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889933 total bases: 3447168 total-ends: 828 total-caps: 2142 max-end-degree: 20 max-adjacency-length: 87702 total-blocks: 0 total-groups: 1 total-edges: 833 total-free-ends: 30 total-attached-ends: 798 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889937 total bases: 3649393 total-ends: 858 total-caps: 2206 max-end-degree: 23 max-adjacency-length: 77729 total-blocks: 0 total-groups: 1 total-edges: 862 total-free-ends: 29 total-attached-ends: 829 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889941 total bases: 3226879 total-ends: 807 total-caps: 2090 max-end-degree: 15 max-adjacency-length: 42295 total-blocks: 0 total-groups: 1 total-edges: 810 total-free-ends: 58 total-attached-ends: 749 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821890009 total bases: 3480968 total-ends: 825 total-caps: 2254 max-end-degree: 50 max-adjacency-length: 98450 total-blocks: 0 total-groups: 1 total-edges: 828 total-free-ends: 33 total-attached-ends: 792 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110584.69 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111585.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111616.42 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111616.51 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111616.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111641.68 : Breaking bar job into 5 separate jobs Got message from job at time: 1385111641.77 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111645.92 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.01 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.39 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.49 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.67 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.83 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111655.85 : Breaking bar job into 7 separate jobs Got message from job at time: 1385120612.16 : Starting avg phase target with index 0 at 1385120611.95 seconds (recursing = 0) Got message from job at time: 1385120612.16 : Starting reference phase target with index 0 at 1385120611.97 seconds (recursing = 1) Got message from job at time: 1385120622.59 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/gTD1/tmp_yuJIaNmPsH/tmp_oZP91YPYsD_kill.txt The job seems to have left a log file, indicating failure: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job Reporting file: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/log.txt log.txt: ---JOBTREE SLAVE OUTPUT LOG--- log.txt: sh: 1: Syntax error: Bad fd number log.txt: Something went wrong with the command: blossom5 -e /tmp/stTmp17374_0 -w /tmp/stTmp17374_1 >& /dev/null log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py", line 271, in main log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth) log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/scriptTree/stack.py", line 153, in execute log.txt: self.target.run() log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/pipeline/cactus_workflow.py", line 676, in run log.txt: wiggle=self.getOptionalPhaseAttrib("wiggle", float)) log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/shared/common.py", line 346, in runCactusReference log.txt: popenPush(command, stdinString=flowerNames) log.txt: File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 210, in popenPush log.txt: raise RuntimeError("Command: %s with stdin string '%s' exited with non-zero status %i" % (command, stdinString, sts)) log.txt: RuntimeError: Command: cactus_reference --cactusDisk ' log.txt: log.txt: log.txt: ' --logLevel CRITICAL --matchingAlgorithm blossom5 --referenceEventString Anc0 --permutations 10 --useSimulatedAnnealing --theta 0.0001 --maxWalkForCalculatingZ 100000 --ignoreUnalignedGaps --wiggle 0.9999 with stdin string '1 0' exited with non-zero status 1 log.txt: Exiting the slave because of a failed job on host hahnlab-pc01 log.txt: Due to failure we are reducing the remaining retry count of job /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job to 0 log.txt: We have set the default memory of the failed job to 8589934592bytes Job: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job is completely failed
Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/9#issuecomment-29080349 .
benedictpaten
commented
10 years ago PS, you'll need to edit the file, then wipe out your earlier run and start a new one.
On Fri, Nov 22, 2013 at 2:06 PM, Benedict Paten benedict@soe.ucsc.eduwrote:
Hi,
Looks like the blossum tool went awry. Could you rerun, but edit the config file (should be in the foo/cactus/progressive/cactus_progressive_workflow_config.file, where foo is the path to the cactus submodule directory), to replace the line: matchingAlgorithm="blossom5" which matchingAlgorithm="greedy".
This will test if it is truly an issue with blossum, or a more general issue.
Thanks!
Benedict
On Fri, Nov 22, 2013 at 7:24 AM, Hussain notifications@github.com wrote:
Here is the rest of the cactus.log that preceded the ALERT shown above:
2013-11-21 20:26:40.344724: Beginning Progressive Cactus Alignment
Got message from job at time: 1385101026.67 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/work/jobTree/jobs/gTD2/tmp_ruCMcy0z5a/tmp_cgJhQMAdPy_kill.txt Got message from job at time: 1385101043.0 : Input-sample: ./work/sequenceData/dmel-all-chromosome-r5.53.fasta Total-sequences: 15 Total-length: 168736537 Proportion-repeat-masked: 0.131561 ProportionNs: 0.037744 N50: 142743 Median-sequence-length: 3288761 Got message from job at time: 1385101043.0 : Input-sample: ./work/sequenceData/dsim-all-chromosome-r1.4.fasta Total-sequences: 10005 Total-length: 137828247 Proportion-repeat-masked: 0.099978 ProportionNs: 0.076833 N50: 165255 Median-sequence-length: 1223 Got message from job at time: 1385101043.0 : Starting caf phase target with index 0 at 1385101032.26 seconds (recursing = 1) Got message from job at time: 1385101043.0 : Adding an oversize flower for target class and stats flower name: 0 total bases: 306564784 total-ends: 20040 total-caps: 20040 max-end-degree: 1 max-adjacency-length: 29004657 total-blocks: 0 total-groups: 1 total-edges: 10020 total-free-ends: 20040 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1385109215.83 : Pinch graph component with 10921 nodes and 11949 edges is being split up by breaking 964 edges to reduce size to less than 714 max, but found 0 pointless edges Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527232, header Uextra with length 29004656 and 2234908 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527227, header 3R with length 27905053 and 22146229 total bases aligned and 58635 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527212, header 3L with length 24543557 and 18315740 total bases aligned and 188857 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527202, header 2L with length 23011544 and 17348961 total bases aligned and 230802 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527207, header X with length 22422827 and 14155982 total bases aligned and 330194 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527222, header 2R with length 21146708 and 15262809 total bases aligned and 318738 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527217, header 4 with length 1351857 and 552287 total bases aligned and 36014 bases aligned to other chromosome threads Got message from job at time: 1385110476.96 : Starting bar phase target with index 0 at 1385109216.1 seconds (recursing = 1) Got message from job at time: 1385110477.04 : Adding an oversize flower for target class and stats flower name: 2133298848490051438 total bases: 3454268 total-ends: 796 total-caps: 2162 max-end-degree: 30 max-adjacency-length: 123624 total-blocks: 0 total-groups: 1 total-edges: 798 total-free-ends: 62 total-attached-ends: 734 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889605 total bases: 3434216 total-ends: 840 total-caps: 2298 max-end-degree: 29 max-adjacency-length: 141481 total-blocks: 0 total-groups: 1 total-edges: 849 total-free-ends: 33 total-attached-ends: 807 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889681 total bases: 3335596 total-ends: 811 total-caps: 2216 max-end-degree: 22 max-adjacency-length: 70229 total-blocks: 0 total-groups: 1 total-edges: 818 total-free-ends: 30 total-attached-ends: 781 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889685 total bases: 3566142 total-ends: 902 total-caps: 2316 max-end-degree: 25 max-adjacency-length: 79970 total-blocks: 0 total-groups: 1 total-edges: 907 total-free-ends: 36 total-attached-ends: 866 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889689 total bases: 3670542 total-ends: 863 total-caps: 2306 max-end-degree: 22 max-adjacency-length: 54690 total-blocks: 0 total-groups: 1 total-edges: 869 total-free-ends: 35 total-attached-ends: 828 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889821 total bases: 2888874 total-ends: 824 total-caps: 2166 max-end-degree: 41 max-adjacency-length: 71147 total-blocks: 0 total-groups: 1 total-edges: 842 total-free-ends: 74 total-attached-ends: 750 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889825 total bases: 3629179 total-ends: 816 total-caps: 2194 max-end-degree: 23 max-adjacency-length: 95616 total-blocks: 0 total-groups: 1 total-edges: 820 total-free-ends: 31 total-attached-ends: 785 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889913 total bases: 3525679 total-ends: 911 total-caps: 2400 max-end-degree: 50 max-adjacency-length: 37444 total-blocks: 0 total-groups: 1 total-edges: 922 total-free-ends: 40 total-attached-ends: 871 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889917 total bases: 3347355 total-ends: 876 total-caps: 2368 max-end-degree: 29 max-adjacency-length: 39264 total-blocks: 0 total-groups: 1 total-edges: 886 total-free-ends: 31 total-attached-ends: 845 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889925 total bases: 3263116 total-ends: 802 total-caps: 2120 max-end-degree: 24 max-adjacency-length: 116911 total-blocks: 0 total-groups: 1 total-edges: 811 total-free-ends: 45 total-attached-ends: 757 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889929 total bases: 3061174 total-ends: 791 total-caps: 2038 max-end-degree: 19 max-adjacency-length: 106442 total-blocks: 0 total-groups: 1 total-edges: 794 total-free-ends: 117 total-attached-ends: 674 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889933 total bases: 3447168 total-ends: 828 total-caps: 2142 max-end-degree: 20 max-adjacency-length: 87702 total-blocks: 0 total-groups: 1 total-edges: 833 total-free-ends: 30 total-attached-ends: 798 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889937 total bases: 3649393 total-ends: 858 total-caps: 2206 max-end-degree: 23 max-adjacency-length: 77729 total-blocks: 0 total-groups: 1 total-edges: 862 total-free-ends: 29 total-attached-ends: 829 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889941 total bases: 3226879 total-ends: 807 total-caps: 2090 max-end-degree: 15 max-adjacency-length: 42295 total-blocks: 0 total-groups: 1 total-edges: 810 total-free-ends: 58 total-attached-ends: 749 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821890009 total bases: 3480968 total-ends: 825 total-caps: 2254 max-end-degree: 50 max-adjacency-length: 98450 total-blocks: 0 total-groups: 1 total-edges: 828 total-free-ends: 33 total-attached-ends: 792 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110584.69 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111585.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111616.42 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111616.51 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111616.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111641.68 : Breaking bar job into 5 separate jobs Got message from job at time: 1385111641.77 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111645.92 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.01 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.39 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.49 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.67 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.83 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111655.85 : Breaking bar job into 7 separate jobs Got message from job at time: 1385120612.16 : Starting avg phase target with index 0 at 1385120611.95 seconds (recursing = 0) Got message from job at time: 1385120612.16 : Starting reference phase target with index 0 at 1385120611.97 seconds (recursing = 1) Got message from job at time: 1385120622.59 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/gTD1/tmp_yuJIaNmPsH/tmp_oZP91YPYsD_kill.txt The job seems to have left a log file, indicating failure: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job Reporting file: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/log.txt log.txt: ---JOBTREE SLAVE OUTPUT LOG--- log.txt: sh: 1: Syntax error: Bad fd number log.txt: Something went wrong with the command: blossom5 -e /tmp/stTmp17374_0 -w /tmp/stTmp17374_1 >& /dev/null log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py", line 271, in main log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth) log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/scriptTree/stack.py", line 153, in execute log.txt: self.target.run() log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/pipeline/cactus_workflow.py", line 676, in run log.txt: wiggle=self.getOptionalPhaseAttrib("wiggle", float)) log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/shared/common.py", line 346, in runCactusReference log.txt: popenPush(command, stdinString=flowerNames) log.txt: File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 210, in popenPush log.txt: raise RuntimeError("Command: %s with stdin string '%s' exited with non-zero status %i" % (command, stdinString, sts)) log.txt: RuntimeError: Command: cactus_reference --cactusDisk ' log.txt: log.txt: log.txt: ' --logLevel CRITICAL --matchingAlgorithm blossom5 --referenceEventString Anc0 --permutations 10 --useSimulatedAnnealing --theta 0.0001 --maxWalkForCalculatingZ 100000 --ignoreUnalignedGaps --wiggle 0.9999 with stdin string '1 0' exited with non-zero status 1 log.txt: Exiting the slave because of a failed job on host hahnlab-pc01 log.txt: Due to failure we are reducing the remaining retry count of job /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job to 0 log.txt: We have set the default memory of the failed job to 8589934592bytes Job: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job is completely failed
Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/9#issuecomment-29080349 .
benedictpaten
commented
10 years ago Ahh.. Joel points out:
"Hi Benedict,
Just noticed--the trouble that the person on github is having is because their /bin/sh is different--I think on all our machines it's symlinked to bash, which can do fancier stuff than just regular sh.
The '>& /dev/null' is one of those fancy things-- I think you have to replace it with '> /dev/null 2>&1'--in that order, for some reason.
The relevant line is in matchingAlgorithms.c, l175."
I have made this change and committed it. Still need to make progressive cactus point at this update.
On Fri, Nov 22, 2013 at 2:06 PM, Benedict Paten benedict@soe.ucsc.eduwrote:
PS, you'll need to edit the file, then wipe out your earlier run and start a new one.
On Fri, Nov 22, 2013 at 2:06 PM, Benedict Paten benedict@soe.ucsc.eduwrote:
Hi,
Looks like the blossum tool went awry. Could you rerun, but edit the config file (should be in the foo/cactus/progressive/cactus_progressive_workflow_config.file, where foo is the path to the cactus submodule directory), to replace the line: matchingAlgorithm="blossom5" which matchingAlgorithm="greedy".
This will test if it is truly an issue with blossum, or a more general issue.
Thanks!
Benedict
On Fri, Nov 22, 2013 at 7:24 AM, Hussain notifications@github.comwrote:
Here is the rest of the cactus.log that preceded the ALERT shown above:
2013-11-21 20:26:40.344724: Beginning Progressive Cactus Alignment
Got message from job at time: 1385101026.67 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/work/jobTree/jobs/gTD2/tmp_ruCMcy0z5a/tmp_cgJhQMAdPy_kill.txt Got message from job at time: 1385101043.0 : Input-sample: ./work/sequenceData/dmel-all-chromosome-r5.53.fasta Total-sequences: 15 Total-length: 168736537 Proportion-repeat-masked: 0.131561 ProportionNs: 0.037744 N50: 142743 Median-sequence-length: 3288761 Got message from job at time: 1385101043.0 : Input-sample: ./work/sequenceData/dsim-all-chromosome-r1.4.fasta Total-sequences: 10005 Total-length: 137828247 Proportion-repeat-masked: 0.099978 ProportionNs: 0.076833 N50: 165255 Median-sequence-length: 1223 Got message from job at time: 1385101043.0 : Starting caf phase target with index 0 at 1385101032.26 seconds (recursing = 1) Got message from job at time: 1385101043.0 : Adding an oversize flower for target class and stats flower name: 0 total bases: 306564784 total-ends: 20040 total-caps: 20040 max-end-degree: 1 max-adjacency-length: 29004657 total-blocks: 0 total-groups: 1 total-edges: 10020 total-free-ends: 20040 total-attached-ends: 0 total-chains: 0 total-link groups: 0 Got message from job at time: 1385109215.83 : Pinch graph component with 10921 nodes and 11949 edges is being split up by breaking 964 edges to reduce size to less than 714 max, but found 0 pointless edges Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527232, header Uextra with length 29004656 and 2234908 total bases aligned and 0 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527227, header 3R with length 27905053 and 22146229 total bases aligned and 58635 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527212, header 3L with length 24543557 and 18315740 total bases aligned and 188857 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527202, header 2L with length 23011544 and 17348961 total bases aligned and 230802 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527207, header X with length 22422827 and 14155982 total bases aligned and 330194 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527222, header 2R with length 21146708 and 15262809 total bases aligned and 318738 bases aligned to other chromosome threads Got message from job at time: 1385109215.86 : Attaching the sequence to the cactus root 2699485764143527217, header 4 with length 1351857 and 552287 total bases aligned and 36014 bases aligned to other chromosome threads Got message from job at time: 1385110476.96 : Starting bar phase target with index 0 at 1385109216.1 seconds (recursing = 1) Got message from job at time: 1385110477.04 : Adding an oversize flower for target class and stats flower name: 2133298848490051438 total bases: 3454268 total-ends: 796 total-caps: 2162 max-end-degree: 30 max-adjacency-length: 123624 total-blocks: 0 total-groups: 1 total-edges: 798 total-free-ends: 62 total-attached-ends: 734 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889605 total bases: 3434216 total-ends: 840 total-caps: 2298 max-end-degree: 29 max-adjacency-length: 141481 total-blocks: 0 total-groups: 1 total-edges: 849 total-free-ends: 33 total-attached-ends: 807 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889681 total bases: 3335596 total-ends: 811 total-caps: 2216 max-end-degree: 22 max-adjacency-length: 70229 total-blocks: 0 total-groups: 1 total-edges: 818 total-free-ends: 30 total-attached-ends: 781 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889685 total bases: 3566142 total-ends: 902 total-caps: 2316 max-end-degree: 25 max-adjacency-length: 79970 total-blocks: 0 total-groups: 1 total-edges: 907 total-free-ends: 36 total-attached-ends: 866 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889689 total bases: 3670542 total-ends: 863 total-caps: 2306 max-end-degree: 22 max-adjacency-length: 54690 total-blocks: 0 total-groups: 1 total-edges: 869 total-free-ends: 35 total-attached-ends: 828 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889821 total bases: 2888874 total-ends: 824 total-caps: 2166 max-end-degree: 41 max-adjacency-length: 71147 total-blocks: 0 total-groups: 1 total-edges: 842 total-free-ends: 74 total-attached-ends: 750 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889825 total bases: 3629179 total-ends: 816 total-caps: 2194 max-end-degree: 23 max-adjacency-length: 95616 total-blocks: 0 total-groups: 1 total-edges: 820 total-free-ends: 31 total-attached-ends: 785 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889913 total bases: 3525679 total-ends: 911 total-caps: 2400 max-end-degree: 50 max-adjacency-length: 37444 total-blocks: 0 total-groups: 1 total-edges: 922 total-free-ends: 40 total-attached-ends: 871 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889917 total bases: 3347355 total-ends: 876 total-caps: 2368 max-end-degree: 29 max-adjacency-length: 39264 total-blocks: 0 total-groups: 1 total-edges: 886 total-free-ends: 31 total-attached-ends: 845 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889925 total bases: 3263116 total-ends: 802 total-caps: 2120 max-end-degree: 24 max-adjacency-length: 116911 total-blocks: 0 total-groups: 1 total-edges: 811 total-free-ends: 45 total-attached-ends: 757 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889929 total bases: 3061174 total-ends: 791 total-caps: 2038 max-end-degree: 19 max-adjacency-length: 106442 total-blocks: 0 total-groups: 1 total-edges: 794 total-free-ends: 117 total-attached-ends: 674 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889933 total bases: 3447168 total-ends: 828 total-caps: 2142 max-end-degree: 20 max-adjacency-length: 87702 total-blocks: 0 total-groups: 1 total-edges: 833 total-free-ends: 30 total-attached-ends: 798 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889937 total bases: 3649393 total-ends: 858 total-caps: 2206 max-end-degree: 23 max-adjacency-length: 77729 total-blocks: 0 total-groups: 1 total-edges: 862 total-free-ends: 29 total-attached-ends: 829 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821889941 total bases: 3226879 total-ends: 807 total-caps: 2090 max-end-degree: 15 max-adjacency-length: 42295 total-blocks: 0 total-groups: 1 total-edges: 810 total-free-ends: 58 total-attached-ends: 749 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110477.06 : Adding an oversize flower for target class and stats flower name: 6570892593821890009 total bases: 3480968 total-ends: 825 total-caps: 2254 max-end-degree: 50 max-adjacency-length: 98450 total-blocks: 0 total-groups: 1 total-edges: 828 total-free-ends: 33 total-attached-ends: 792 total-chains: 0 total-link groups: 0 Got message from job at time: 1385110584.69 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111585.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111616.42 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111616.51 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111616.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111641.68 : Breaking bar job into 5 separate jobs Got message from job at time: 1385111641.77 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111645.92 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.01 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.39 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.49 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111646.58 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.67 : Breaking bar job into 7 separate jobs Got message from job at time: 1385111646.83 : Breaking bar job into 6 separate jobs Got message from job at time: 1385111655.85 : Breaking bar job into 7 separate jobs Got message from job at time: 1385120612.16 : Starting avg phase target with index 0 at 1385120611.95 seconds (recursing = 0) Got message from job at time: 1385120612.16 : Starting reference phase target with index 0 at 1385120611.97 seconds (recursing = 1) Got message from job at time: 1385120622.59 : Blocking on ktserver with killPath /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/gTD1/tmp_yuJIaNmPsH/tmp_oZP91YPYsD_kill.txt The job seems to have left a log file, indicating failure: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job Reporting file: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/log.txt log.txt: ---JOBTREE SLAVE OUTPUT LOG--- log.txt: sh: 1: Syntax error: Bad fd number log.txt: Something went wrong with the command: blossom5 -e /tmp/stTmp17374_0 -w /tmp/stTmp17374_1 >& /dev/null log.txt: Traceback (most recent call last): log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/src/jobTreeSlave.py", line 271, in main log.txt: defaultMemory=defaultMemory, defaultCpu=defaultCpu, depth=depth) log.txt: File "/home/hussainather/progressiveCactus/submodules/jobTree/scriptTree/stack.py", line 153, in execute log.txt: self.target.run() log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/pipeline/cactus_workflow.py", line 676, in run log.txt: wiggle=self.getOptionalPhaseAttrib("wiggle", float)) log.txt: File "/home/hussainather/progressiveCactus/submodules/cactus/shared/common.py", line 346, in runCactusReference log.txt: popenPush(command, stdinString=flowerNames) log.txt: File "/home/hussainather/progressiveCactus/submodules/sonLib/bioio.py", line 210, in popenPush log.txt: raise RuntimeError("Command: %s with stdin string '%s' exited with non-zero status %i" % (command, stdinString, sts)) log.txt: RuntimeError: Command: cactus_reference --cactusDisk ' log.txt: log.txt: log.txt: ' --logLevel CRITICAL --matchingAlgorithm blossom5 --referenceEventString Anc0 --permutations 10 --useSimulatedAnnealing --theta 0.0001 --maxWalkForCalculatingZ 100000 --ignoreUnalignedGaps --wiggle 0.9999 with stdin string '1 0' exited with non-zero status 1 log.txt: Exiting the slave because of a failed job on host hahnlab-pc01 log.txt: Due to failure we are reducing the remaining retry count of job /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job to 0 log.txt: We have set the default memory of the failed job to 8589934592bytes Job: /home/hussainather/progressiveCactus/work/jobTree/jobs/t1/t1/job is completely failed
Reply to this email directly or view it on GitHubhttps://github.com/glennhickey/progressiveCactus/issues/9#issuecomment-29080349 .
Mcfoofa
commented
10 years ago The problem is resolved. It was the blossom issue. Thank you!
When running the aligner on full genomes, this message appears in the .out file. Also, when creating the work directory, "b00.hal:" is created rather than "b00.hal" The example files worked fine, but not the full genome sequence we enter.
\ ALERT **
The only jobs that I have detected running for at least the past 7800s are 2 ktservers. Furthermore, there appears to have been 1 failed jobs. It is likely that Progressive Cactus is in a deadlock state and will not finish until the servers or your batch system time out. Suggestions: