search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
529 stars 111 forks source link

cactus_caf: impl/pinchIterator.c:103: pairwiseAlignmentToPinch_getNext: Assertion `pA->xCoordinate == pA->pairwiseAlignment->end1' failed. #434

Open hehuiying1125 opened 3 years ago

hehuiying1125 commented 3 years ago

Hi, I used cactus to alignment plant genomes (cactus --nodeTypes c4.8xlarge:0.7,r3.8xlarge --minNodes 0,0 --maxNodes 10,5 --provisioner aws --batchSystem mesos --metrics aws:us-east-1:agis-2021-jobstore seqFile.txt result.hal --root ac --realTimeLogging --logFile cactus.log --targetTime 1 --restart), and I always met the bug "pairwiseAlignmentToPinch_getNext: Assertion `pA->xCoordinate == pA->pairwiseAlignment->end1". Did anyone meet the same bug? If anyone could give me some suggests, it will be apprecaited. the log was as following:

The job seems to have left a log file, indicating failure: 'CactusCafWrapper' 24fe9f5f-c914-4d40-86fd-5bdd8fc59062-- Log from job 24fe9f5f-c914-4d40-86fd-5bdd8fc59062 follows: =========> |   |   |     | [2021-02-13T12:42:54+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---   | [2021-02-13T12:42:54+0000] [MainThread] [I] [toil] Running Toil version 4.2.0-3aa1da130141039cb357efe36d7df9b9f6ae9b5b on host ip-172-31-45-12.us-east-2.compute.internal.   | [2021-02-13T12:43:00+0000] [MainThread] [I] [toil.lib.bioio] Alignments file: /var/lib/toil/node-2f6f2de8-df2e-424c-815b-db37a6a59679-4ad9c8e3b64c0936af8834905f342da0/tmpqd5bc18f/3776d191-62ff-445b-a28f-b51e6b3b0b79/tmp8x5yqsee.tmp   | [2021-02-13T12:43:04+0000] [MainThread] [I] [cactus.shared.common] Work dirs: {'/var/lib/toil/node-2f6f2de8-df2e-424c-815b-db37a6a59679-4ad9c8e3b64c0936af8834905f342da0/tmpqd5bc18f/3776d191-62ff-445b-a28f-b51e6b3b0b79'}   | [2021-02-13T12:43:04+0000] [MainThread] [I] [cactus.shared.common] Docker work dir: /var/lib/toil/node-2f6f2de8-df2e-424c-815b-db37a6a59679-4ad9c8e3b64c0936af8834905f342da0/tmpqd5bc18f/3776d191-62ff-445b-a28f-b51e6b3b0b79   | [2021-02-13T12:43:04+0000] [MainThread] [I] [cactus.shared.common] Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/node-2f6f2de8-df2e-424c-815b-db37a6a59679-4ad9c8e3b64c0936af8834905f342da0/tmpqd5bc18f/3776d191-62ff-445b-a28f-b51e6b3b0b79:/data', '--entrypoint', '/opt/cactus/wrapper.sh', '--name', '40a61ece-bff7-4ebe-89dd-5d6e65c012cd', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:b50cdcdc102582359d5319857436a3ad6f46a9a8', 'cactus_caf', '--logLevel', 'INFO', '--alignments', 'tmp8x5yqsee.tmp', '--cactusDisk', '\n\t\t\t\n\t\t\n\t', '--secondaryAlignments', 'tmp7atqj9he.tmp', '--annealingRounds', '128', '--deannealingRounds', '2 8', '--trim', '0 0', '--lastzArguments', '--step=1 --ambiguous=iupac,100,100 --ydrop=3000', '--minimumTreeCoverage', '0.0', '--blockTrim', '5.0', '--minimumDegree', '2', '--minimumSequenceLengthForBlast', '30', '--minimumIngroupDegree', '1', '--minimumOutgroupDegree', '0', '--alignmentFilter', 'filterSecondariesByMultipleSpecies', '--maxAdjacencyComponentSizeRatio', '50.0', '--realign', '--realignArguments', '--gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99', '--phylogenyNumTrees', '30', '--phylogenyRootingMethod', 'bestRecon', '--phylogenyScoringMethod', 'reconCost', '--phylogenyBreakpointScalingFactor', '1.0', '--phylogenySkipSingleCopyBlocks', '--phylogenyMaxBaseDistance', '100', '--phylogenyMaxBlockDistance', '50', '--phylogenyTreeBuildingMethod', 'guidedNeighborJoining,splitDecomposition', '--phylogenyCostPerDupPerBase', '0.00', '--phylogenyCostPerLossPerBase', '0.02', '--referenceEventHeader', 'Anc245', '--phylogenyDoSplitsWithSupportHigherThanThisAllAtOnce', '0.44', '--numTreeBuildingThreads', '2', '--minimumBlockDegreeToCheckSupport', '10', '--minimumBlockHomologySupport', '0.05', '--removeRecoverableChains', 'unequalNumberOfIngroupCopies', '--minimumNumberOfSpecies', '1', '--maxRecoverableChainsIterations', '5', '--maxRecoverableChainLength', '500000', '--phylogenyHomologyUnitType', 'chain', '--phylogenyDistanceCorrectionMethod', 'jukesCantor', '--minLengthForChromosome', '1000000', '--proportionOfUnalignedBasesForNewChromosome', '0.8', '--maximumMedianSequenceLengthBetweenLinkedEnds', '1000']   | [2021-02-13T12:43:04+0000] [MainThread] [I] [toil-rt] 2021-02-13 12:43:04.854822: Running the command: "docker run --interactive --net=host --log-driver=none -u 0:0 -v /var/lib/toil/node-2f6f2de8-df2e-424c-815b-db37a6a59679-4ad9c8e3b64c0936af8834905f342da0/tmpqd5bc18f/3776d191-62ff-445b-a28f-b51e6b3b0b79:/data --entrypoint /opt/cactus/wrapper.sh --name 40a61ece-bff7-4ebe-89dd-5d6e65c012cd --rm quay.io/comparative-genomics-toolkit/cactus:b50cdcdc102582359d5319857436a3ad6f46a9a8 cactus_caf --logLevel INFO --alignments tmp8x5yqsee.tmp --cactusDisk   |   |   |   |   |   |   |   |   | --secondaryAlignments tmp7atqj9he.tmp --annealingRounds 128 --deannealingRounds 2 8 --trim 0 0 --lastzArguments --step=1 --ambiguous=iupac,100,100 --ydrop=3000 --minimumTreeCoverage 0.0 --blockTrim 5.0 --minimumDegree 2 --minimumSequenceLengthForBlast 30 --minimumIngroupDegree 1 --minimumOutgroupDegree 0 --alignmentFilter filterSecondariesByMultipleSpecies --maxAdjacencyComponentSizeRatio 50.0 --realign --realignArguments --gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99 --phylogenyNumTrees 30 --phylogenyRootingMethod bestRecon --phylogenyScoringMethod <...> ectionMethod jukesCantor --minLengthForChromosome 1000000 --proportionOfUnalignedBasesForNewChromosome 0.8 --maximumMedianSequenceLengthBetweenLinkedEnds 1000" (features={'alignmentsSize': 406278350})   | Error: No such object: 40a61ece-bff7-4ebe-89dd-5d6e65c012cd   | Error: No such object: 40a61ece-bff7-4ebe-89dd-5d6e65c012cd   | Error: No such object: 40a61ece-bff7-4ebe-89dd-5d6e65c012cd   | Error: No such object: 40a61ece-bff7-4ebe-89dd-5d6e65c012cd   | Running command catchsegv 'cactus_caf' '--logLevel' 'INFO' '--alignments' 'tmp8x5yqsee.tmp' '--cactusDisk' '   |   |   |   |   |   |   |   |   | ' '--secondaryAlignments' 'tmp7atqj9he.tmp' '--annealingRounds' '128' '--deannealingRounds' '2 8' '--trim' '0 0' '--lastzArguments' '--step=1 --ambiguous=iupac,100,100 --ydrop=3000' '--minimumTreeCoverage' '0.0' '--blockTrim' '5.0' '--minimumDegree' '2' '--minimumSequenceLengthForBlast' '30' '--minimumIngroupDegree' '1' '--minimumOutgroupDegree' '0' '--alignmentFilter' 'filterSecondariesByMultipleSpecies' '--maxAdjacencyComponentSizeRatio' '50.0' '--realign' '--realignArguments' '--gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99' '--phylogenyNumTrees' '30' '--phylogenyRootingMethod' 'bestRecon' '--phylogenyScoringMethod' 'reconCost' '--phylogenyBreakpointScalingFactor' '1.0' '--phylogenySkipSingleCopyBlocks' '--phylogenyMaxBaseDistance' '100' '--phylogenyMaxBlockDistance' '50' '--phylogenyTreeBuildingMethod' 'guidedNeighborJoining,splitDecomposition' '--phylogenyCostPerDupPerBase' '0.00' '--phylogenyCostPerLossPerBase' '0.02' '--referenceEventHeader' 'Anc245' '--phylogenyDoSplitsWithSupportHigherThanThisAllAtOnce' '0.44' '--numTreeBuildingThreads' '2' '--minimumBlockDegreeToCheckSupport' '10' '--minimumBlockHomologySupport' '0.05' '--removeRecoverableChains' 'unequalNumberOfIngroupCopies' '--minimumNumberOfSpecies' '1' '--maxRecoverableChainsIterations' '5' '--maxRecoverableChainLength' '500000' '--phylogenyHomologyUnitType' 'chain' '--phylogenyDistanceCorrectionMethod' 'jukesCantor' '--minLengthForChromosome' '1000000' '--proportionOfUnalignedBasesForNewChromosome' '0.8' '--maximumMedianSequenceLengthBetweenLinkedEnds' '1000'   | Flower disk name :   |   |   |   |   |   |   |   |   |   |   |     | Set up the flower disk |     | cactus_caf: impl/pinchIterator.c:103: pairwiseAlignmentToPinch_getNext: Assertion `pA->xCoordinate == pA->pairwiseAlignment->end1' failed.   | Aborted (core dumped) |     | [2021-02-13T12:44:22+0000] [MainThread] [I] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Max memory used for job CactusCafWrapper (tool cactus_caf) on JSON features {"alignmentsSize": 406278350}: 0   | Traceback (most recent call last):   | File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 368, in workerScript   | job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)   | File "/tmp/tmp9r6yf3e9/0fe438d376c97140e22dfa1f0d2f8374/cactus/shared/common.py", line 1424, in _runner   | fileStore=fileStore, **kwargs)   | File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1424, in _runner   | returnValues = self._run(jobGraph, fileStore)   | File "/usr/local/lib/python3.6/dist-packages/toil/job.py", line 1361, in _run   | return self.run(fileStore) |     | File "/tmp/tmp9r6yf3e9/0fe438d376c97140e22dfa1f0d2f8374/cactus/pipeline/cactus_workflow.py", line 856, in run   | constraints=constraints) |     | File "/tmp/tmp9r6yf3e9/0fe438d376c97140e22dfa1f0d2f8374/cactus/pipeline/cactus_workflow.py", line 838, in runCactusCafInWorkflow   | maxRecoverableChainLength=self.getOptionalPhaseAttrib("maxRecoverableChainLength", int))   | File "/tmp/tmp9r6yf3e9/0fe438d376c97140e22dfa1f0d2f8374/cactus/shared/common.py", line 435, in runCactusCaf   | features=features, job_name=jobName, fileStore=fileStore)   | File "/tmp/tmp9r6yf3e9/0fe438d376c97140e22dfa1f0d2f8374/cactus/shared/common.py", line 1357, in cactus_call   | raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))   | RuntimeError: Command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/node-2f6f2de8-df2e-424c-815b-db37a6a59679-4ad9c8e3b64c0936af8834905f342da0/tmpqd5bc18f/3776d191-62ff-445b-a28f-b51e6b3b0b79:/data', '--entrypoint', '/opt/cactus/wrapper.sh', '--name', '40a61ece-bff7-4ebe-89dd-5d6e65c012cd', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:b50cdcdc102582359d5319857436a3ad6f46a9a8', 'cactus_caf', '--logLevel', 'INFO', '--alignments', 'tmp8x5yqsee.tmp', '--cactusDisk', '\n\t\t\t\n\t\t\n\t', '--secondaryAlignments', 'tmp7atqj9he.tmp', '--annealingRounds', '128', '--deannealingRounds', '2 8', '--trim', '0 0', '--lastzArguments', '--step=1 --ambiguous=iupac,100,100 --ydrop=3000', '--minimumTreeCoverage', '0.0', '--blockTrim', '5.0', '--minimumDegree', '2', '--minimumSequenceLengthForBlast', '30', '--minimumIngroupDegree', '1', '--minimumOutgroupDegree', '0', '--alignmentFilter', 'filterSecondariesByMultipleSpecies', '--maxAdjacencyComponentSizeRatio', '50.0', '--realign', '--realignArguments', '--gapGamma 0.0 --matchGamma 0.9 --diagonalExpansion 4 --splitMatrixBiggerThanThis 10 --constraintDiagonalTrim 0 --alignAmbiguityCharacters --splitIndelsLongerThanThis 99', '--phylogenyNumTrees', '30', '--phylogenyRootingMethod', 'bestRecon', '--phylogenyScoringMethod', 'reconCost', '--phylogenyBreakpointScalingFactor', '1.0', '--phylogenySkipSingleCopyBlocks', '--phylogenyMaxBaseDistance', '100', '--phylogenyMaxBlockDistance', '50', '--phylogenyTreeBuildingMethod', 'guidedNeighborJoining,splitDecomposition', '--phylogenyCostPerDupPerBase', '0.00', '--phylogenyCostPerLossPerBase', '0.02', '--referenceEventHeader', 'Anc245', '--phylogenyDoSplitsWithSupportHigherThanThisAllAtOnce', '0.44', '--numTreeBuildingThreads', '2', '--minimumBlockDegreeToCheckSupport', '10', '--minimumBlockHomologySupport', '0.05', '--removeRecoverableChains', 'unequalNumberOfIngroupCopies', '--minimumNumberOfSpecies', '1', '--maxRecoverableChainsIterations', '5', '--maxRecoverableChainLength', '500000', '--phylogenyHomologyUnitType', 'chain', '--phylogenyDistanceCorrectionMethod', 'jukesCantor', '--minLengthForChromosome', '1000000', '--proportionOfUnalignedBasesForNewChromosome', '0.8', '--maximumMedianSequenceLengthBetweenLinkedEnds', '1000'] exited 1: stdout=   | [2021-02-13T12:44:22+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host ip-172-31-45-12.us-east-2.compute.internal

hehuiying1125 commented 3 years ago

Hi， I run the same job on a single machine with 64 cores (cactus ./jobStore seqFile.txt ./result.hal --root ac --realTimeLogging --logFile cactus.log --targetTime 1 --maxCores 62), the bug never appears. The seqFile was exactly the same. I installed cactus-bin-v1.2.3.tar.gz following BIN-INSTALL.md. On AWS, I installed cactus following running-in-aws.md by git clone.

At beginning I thought this bug may be due to I installed different versions of Cactus on AWS and on my single machine. I installed cactus-bin-v1.2.3.tar.gz following BIN-INSTALL.md on AWS. Then I restart my failed run, the bug still came out. It seems that the bug was due to differences of AWS instance and my single machine.

If anyone could give me some suggestions to how to further find the reason of this bug, it will be appreciated.