CactusConsolidated Error

[francicco]

Hi,

I'm getting this type of error, maybe there's something wrong with my installation because I previously aligned the same genomes before...

Any help? Thanks a lot F CACTUS.errors.log

[2021-10-29T09:36:11+0100] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'CactusConsolidated' kind-CactusConsolidated/instance-1jxqgg9z v7
Exit reason: None
[2021-10-29T09:36:11+0100] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'CactusConsolidated' kind-CactusConsolidated/instance-1jxqgg9z v8
[2021-10-29T09:36:11+0100] [MainThread] [W] [toil.leader] Log from job "kind-CactusConsolidated/instance-1jxqgg9z" follows:
=========>
        [2021-10-29T09:36:04+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
        [2021-10-29T09:36:04+0100] [MainThread] [I] [toil] Running Toil version 5.5.0-b0ff5be051f2fd55352e00450b7848dcf8354a3b on host bp1-compute00194.data.bp.acrc.priv.
        [2021-10-29T09:36:04+0100] [MainThread] [I] [toil.worker] Working on job 'CactusConsolidated' kind-CactusConsolidated/instance-1jxqgg9z v7
        [2021-10-29T09:36:06+0100] [MainThread] [I] [toil.worker] Loaded body Job('CactusConsolidated' kind-CactusConsolidated/instance-1jxqgg9z v7) from description 'CactusConsolidated' kind-CactusConsolidated/instance-1jxqgg9z v7
        [2021-10-29T09:36:08+0100] [MainThread] [I] [toil.statsAndLogging] Alignments file: /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbwszp7le.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [W] [root] Deprecated toil method.  Please call "logging.getLevelName" directly.
        [2021-10-29T09:36:10+0100] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_consolidated', '--sequences', 'Hmel /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbc28_wx1.tmp Jcoe /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp7kqfowkg.tmp Mcin /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpm27i0mze.tmp Pdid /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpafhhrizy.tmp Smor /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpuq14pfnj.tmp', '--speciesTree', '(((Hmel:0.123292,Pdid:0.091286)Heliconiini:0.112744,Smor:0.138589)Heliconiinae:0.128586,(Jcoe:0.122226,Mcin:0.171988)Nymphalinae:0.104749)HeliconiinaeNymphalinae;', '--logLevel', 'INFO', '--alignments', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbwszp7le.tmp', '--params', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpad1icbvz.tmp', '--outputFile', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpeexabb91.tmp', '--outputHalFastaFile', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp16ngnyka.tmp', '--outputReferenceFile', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpzfgfimx4.tmp', '--outgroupEvents', 'Smor Pdid Hmel', '--referenceEvent', 'Nymphalinae', '--secondaryAlignments', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpcn1y7trz.tmp', '--threads', '64']
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] 2021-10-29 09:36:10.274747: Running the command: "cactus_consolidated --sequences Hmel /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbc28_wx1.tmp Jcoe /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp7kqfowkg.tmp Mcin /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpm27i0mze.tmp Pdid /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpafhhrizy.tmp Smor /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpuq14pfnj.tmp --speciesTree (((Hmel:0.123292,Pdid:0.091286)Heliconiini:0.112744,Smor:0.138589)Heliconiinae:0.128586,(Jcoe:0.122226,Mcin:0.171988)Nymphalinae:0.104749)HeliconiinaeNymphalinae; --logLevel INFO --alignments /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbwszp7le.tmp --params /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpad1icbvz.tmp --outputFile /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpeexabb91.tmp --outputHalFastaFile /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp16ngnyka.tmp --outputReferenceFile /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpzfgfimx4.tmp --outgroupEvents Smor Pdid Hmel --referenceEvent Nymphalinae --secondaryAlignments /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpcn1y7trz.tmp --threads 64"
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Params file: /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpad1icbvz.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Output file string : /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpeexabb91.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Output hal fasta file string : /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp16ngnyka.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Output reference fasta file string : /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpzfgfimx4.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Sequence files and events: Hmel /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbc28_wx1.tmp Jcoe /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp7kqfowkg.tmp Mcin /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpm27i0mze.tmp Pdid /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpafhhrizy.tmp Smor /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpuq14pfnj.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-f2o3n7d2/file-eef7d0f0e94b42dd8c6b696ccccaebaa/tmpse_69p_k.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbc28_wx1.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Alignments file: /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbwszp7le.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-rwegaq88/cleanup/file-0e864a92280f49d1ac6656a7804956fc/tmp4q02vzyz.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp7kqfowkg.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-rwegaq88/cleanup/file-1550347ecbc3458fa53b67f1d8a4a955/tmpgmwcqpcv.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpm27i0mze.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Secondary alignments file: /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpcn1y7trz.tmp
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-10og9fnn/file-ec83f077901c41259e3dc834a1f51d22/tmp_4ed0q1o.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpafhhrizy.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Constraint alignments file: (null)
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-hjo5oesx/file-0c96f15208c3471eaa0044d7f2cb4618/tmpiruvr337.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpuq14pfnj.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-nd406utm/file-3b6c8e9889fb48d4b20202ea615a08f4/tmph1idkwjw.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbwszp7le.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-nd406utm/file-02d3dd9c7db8444487d7ed04cf335c42/tmpo9xrrzsi.tmp' to path '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpcn1y7trz.tmp'
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Species tree: (((Hmel:0.123292,Pdid:0.091286)Heliconiini:0.112744,Smor:0.138589)Heliconiinae:0.128586,(Jcoe:0.122226,Mcin:0.171988)Nymphalinae:0.104749)HeliconiinaeNymphalinae;
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Outgroup events: Smor Pdid Hmel
        [2021-10-29T09:36:10+0100] [MainThread] [I] [toil-rt] cactus_consolidated(Nymphalinae): Reference event: Nymphalinae
        Traceback (most recent call last):
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/toil/worker.py", line 402, in workerScript
            job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/cactus/shared/common.py", line 933, in _runner
            fileStore=fileStore, **kwargs)
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/toil/job.py", line 2362, in _runner
            returnValues = self._run(jobGraph=None, fileStore=fileStore)
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/toil/job.py", line 2283, in _run
            return self.run(fileStore)
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/cactus/pipeline/cactus_workflow.py", line 420, in run
            cores=self.cores)
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/cactus/shared/common.py", line 198, in runCactusConsolidated
            parameters=["cactus_consolidated"] + args)[1] # Get just the standard error output
          File "/work/tk19812/software/cactus-bin-v2.0.3/venv/lib/python3.6/site-packages/cactus/shared/common.py", line 868, in cactus_call
            raise RuntimeError("Command {} signaled {}: {}".format(call, signal.Signals(-process.returncode).name, out))
        RuntimeError: Command ['cactus_consolidated', '--sequences', 'Hmel /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbc28_wx1.tmp Jcoe /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp7kqfowkg.tmp Mcin /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpm27i0mze.tmp Pdid /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpafhhrizy.tmp Smor /tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpuq14pfnj.tmp', '--speciesTree', '(((Hmel:0.123292,Pdid:0.091286)Heliconiini:0.112744,Smor:0.138589)Heliconiinae:0.128586,(Jcoe:0.122226,Mcin:0.171988)Nymphalinae:0.104749)HeliconiinaeNymphalinae;', '--logLevel', 'INFO', '--alignments', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpbwszp7le.tmp', '--params', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpad1icbvz.tmp', '--outputFile', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpeexabb91.tmp', '--outputHalFastaFile', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmp16ngnyka.tmp', '--outputReferenceFile', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpzfgfimx4.tmp', '--outgroupEvents', 'Smor Pdid Hmel', '--referenceEvent', 'Nymphalinae', '--secondaryAlignments', '/tmp/5fad7dfe9ffb54f990c112d5763ecc23/794f/3bf0/tmpcn1y7trz.tmp', '--threads', '64'] signaled SIGSEGV: stdout=, stderr=None
        [2021-10-29T09:36:10+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host bp1-compute00194.data.bp.acrc.priv
<=========

[francicco]

Hi,

The error persists :( F

=========>
    [2021-11-26T12:36:30+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
    [2021-11-26T12:36:30+0000] [MainThread] [I] [toil] Running Toil version 5.4.0-87293d63fa6c76f03bed3adf93414ffee67bf9a7 on host bp1-compute194.data.bp.acrc.priv.
    [2021-11-26T12:36:30+0000] [MainThread] [I] [toil.worker] Working on job 'CactusConsolidated' kind-CactusConsolidated/instance-fpv2frjm
    [2021-11-26T12:36:31+0000] [MainThread] [I] [toil.worker] Loaded body Job('CactusConsolidated' kind-CactusConsolidated/instance-fpv2frjm) from description 'CactusConsolidated' kind-CactusConsolidated/instance-fpv2frjm
    [2021-11-26T12:36:42+0000] [MainThread] [I] [toil.statsAndLogging] Alignments file: /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpzmgjn28m.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [W] [root] Deprecated toil method.  Please call "logging.getLevelName" directly.
    [2021-11-26T12:36:48+0000] [MainThread] [I] [cactus.shared.common] Running the command ['cactus_consolidated', '--sequences', 'Hcha /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpcxpg7pgb.tmp Hcly /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmphgzilige.tmp Herd /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp0syx4a7y.tmp Hhor /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp4oh52q2y.tmp Hsar /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpobmihzxq.tmp', '--speciesTree', '((Hsar:0.019742036200000035,Hcha:0.018929476599999995)SaraSapho:0.004005008000000032,(Herd:0.02043395960000005,(Hhor:0.00889558459999995,Hcly:0.011993779699999929)Anc15:0.008227025599999993)Erato:0.0013045297000000566)EratoSaraSapho;', '--logLevel', 'INFO', '--alignments', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpzmgjn28m.tmp', '--params', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmps1n_4ysg.tmp', '--outputFile', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpczbransj.tmp', '--outputHalFastaFile', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpkx9dvxy1.tmp', '--outputReferenceFile', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpeme9bpqq.tmp', '--outgroupEvents', 'Herd Hcha Hsar', '--referenceEvent', 'Anc15', '--secondaryAlignments', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp3hl7lcgt.tmp', '--threads', '200']
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] 2021-11-26 12:36:48.109161: Running the command: "cactus_consolidated --sequences Hcha /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpcxpg7pgb.tmp Hcly /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmphgzilige.tmp Herd /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp0syx4a7y.tmp Hhor /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp4oh52q2y.tmp Hsar /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpobmihzxq.tmp --speciesTree ((Hsar:0.019742036200000035,Hcha:0.018929476599999995)SaraSapho:0.004005008000000032,(Herd:0.02043395960000005,(Hhor:0.00889558459999995,Hcly:0.011993779699999929)Anc15:0.008227025599999993)Erato:0.0013045297000000566)EratoSaraSapho; --logLevel INFO --alignments /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpzmgjn28m.tmp --params /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmps1n_4ysg.tmp --outputFile /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpczbransj.tmp --outputHalFastaFile /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpkx9dvxy1.tmp --outputReferenceFile /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpeme9bpqq.tmp --outgroupEvents Herd Hcha Hsar --referenceEvent Anc15 --secondaryAlignments /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp3hl7lcgt.tmp --threads 200"
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Params file: /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmps1n_4ysg.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Output file string : /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpczbransj.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-ytifkod4/file-cb248f5cc9834fdbae6846d291588669/tmpnqlz9ytg.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpcxpg7pgb.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-4xu8u_88/cleanup/file-ad1b6d41cb9e4e49a61ecd30e7a11f0e/tmpmlpvbkxw.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmphgzilige.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Output hal fasta file string : /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpkx9dvxy1.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-etg0urwb/file-9e39b739ac3447de83f59bb82ef49418/tmpckjsbfv0.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp0syx4a7y.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CactusBlastPhase/instance-4xu8u_88/cleanup/file-b56436333a8e48798f804a8f31946293/tmpamwu_kxs.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp4oh52q2y.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-TrimAndRecurseOnOutgroups/instance-mgi3hg93/file-d01d31c9a60a4d41b4891bd5019516fe/tmpfhpcctmb.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpobmihzxq.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-rrnkij1z/file-a3a3d24b3f924cb2bfc993a4ff2ad891/tmptfdnxb22.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpzmgjn28m.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-JobFunctionWrappingJob/instance-rrnkij1z/file-b807d3377fcf41f49fff23d9c0bd3d1a/tmptfhk2154.tmp' to path '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp3hl7lcgt.tmp'
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Output reference fasta file string : /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpeme9bpqq.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Sequence files and events: Hcha /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpcxpg7pgb.tmp Hcly /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmphgzilige.tmp Herd /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp0syx4a7y.tmp Hhor /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp4oh52q2y.tmp Hsar /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpobmihzxq.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Alignments file: /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpzmgjn28m.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Secondary alignments file: /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp3hl7lcgt.tmp
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Constraint alignments file: (null)
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Species tree: ((Hsar:0.019742036200000035,Hcha:0.018929476599999995)SaraSapho:0.004005008000000032,(Herd:0.02043395960000005,(Hhor:0.00889558459999995,Hcly:0.011993779699999929)Anc15:0.008227025599999993)Erato:0.0013045297000000566)EratoSaraSapho;
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Outgroup events: Herd Hcha Hsar
    [2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Reference event: Anc15
    [2021-11-26T12:36:48+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] LOG-TO-MASTER: Job used more disk than requested. For CWL, consider increasing the outdirMin requirement, otherwise, consider increasing the disk requirement. Job files/for-job/kind-CactusConsolidated/instance-fpv2frjm/cleanup/file-28f3787f4a6b489caa654131bcf21538/stream used 140.78% disk (6.8 GiB [7303340032B] used, 4.8 GiB [5187740727B] requested).
    Traceback (most recent call last):
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/worker.py", line 393, in workerScript
        job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/shared/common.py", line 932, in _runner
        super(RoundedJob, self)._runner(*args, jobStore=jobStore,
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/job.py", line 2360, in _runner
        returnValues = self._run(jobGraph=None, fileStore=fileStore)
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/job.py", line 2281, in _run
        return self.run(fileStore)
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/pipeline/cactus_workflow.py", line 408, in run
        messages = runCactusConsolidated(seqMap=seqMap,
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/shared/common.py", line 197, in runCactusConsolidated
        masterMessages = cactus_call(check_output=True, returnStdErr=True, realtimeStderrPrefix='cactus_consolidated({})'.format(referenceEvent),
      File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/shared/common.py", line 868, in cactus_call
        raise RuntimeError("Command {} signaled {}: {}".format(call, signal.Signals(-process.returncode).name, out))
    RuntimeError: Command ['cactus_consolidated', '--sequences', 'Hcha /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpcxpg7pgb.tmp Hcly /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmphgzilige.tmp Herd /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp0syx4a7y.tmp Hhor /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp4oh52q2y.tmp Hsar /tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpobmihzxq.tmp', '--speciesTree', '((Hsar:0.019742036200000035,Hcha:0.018929476599999995)SaraSapho:0.004005008000000032,(Herd:0.02043395960000005,(Hhor:0.00889558459999995,Hcly:0.011993779699999929)Anc15:0.008227025599999993)Erato:0.0013045297000000566)EratoSaraSapho;', '--logLevel', 'INFO', '--alignments', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpzmgjn28m.tmp', '--params', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmps1n_4ysg.tmp', '--outputFile', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpczbransj.tmp', '--outputHalFastaFile', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpkx9dvxy1.tmp', '--outputReferenceFile', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmpeme9bpqq.tmp', '--outgroupEvents', 'Herd Hcha Hsar', '--referenceEvent', 'Anc15', '--secondaryAlignments', '/tmp/a11ca9dab50c567aa1f5afc26fe91504/e921/4bf2/tmp3hl7lcgt.tmp', '--threads', '200'] signaled SIGSEGV: stdout=, stderr=None
    [2021-11-26T12:36:48+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host bp1-compute194.data.bp.acrc.priv
<=========

[glennhickey]

A few clues from the log:

• it's crashing with SIGSEGV, which usually means bug or unhandled input error
• it's crashing extremely early in cactus_consolidated. The last message seems to be Reference event: Anc15. The next message would normally be Loaded the parameters files, 0 seconds have elapsed, but it doesn't get there due to the crash. So I think there's a decent chance it's failing to load the config here, even though it's usually good about spitting out a better error in such cases. One thing to try would be to run without specifying --configFile and seeing if the error persists...

[francicco]

Sorry, I can I run it without the --configFile flag? Where does it take in info of the genomes? I'm confused. Thanks for the reply. F

[glennhickey]

The info for the genomes comes on the second command line argument, and definitely not with --configFile. If you are not using --configFile, it may be the + signs in the ancestor names? You might try passing in a tree with only leaf names specified. The error does seem to be at the level of input/naming as it's happening before the first graph is even constructed.

[francicco]

this is the command line i'm using:

cactus $OUTPUTDIR/$CONFIG.scOG_ML.jobStore $OUTPUTDIR/$CONFIG.scOG_ML.conf $OUTPUTDIR/$CONFIG.scOG_ML.hal --maxMemory $MEM --workDir /tmp --binariesMode local --stats --logFile $OUTPUTDIR/$CONFIG.$REF.log --maxCores $THREADS --defaultMemory $MEM --realTimeLogging

There's actually no + in any of the ancestor names

(((((((((((((((((Htim:0.0038869669000000107,Hheu:0.0049981889000000335)Anc36:0.0033437624000000055,(Hpac:0.00986230450000003,Hcyd:0.0040755693000000814)Anc37:0.002230524999999983)Anc32:0.0020316084999999484,Hmel:0.007262151699999997)Anc27:0.004617452000000077,(((Hhel:0.00724042530000002,Hatt:0.008730127099999985)Anc38:0.0019464672000000238,(Helv:0.006136523899999968,Hpar:0.0061621217999999756)Anc39:0.002825525600000045)Anc33:0.001173160199999912,Heth:0.014242937799999966)Anc28:0.0015224138000000664)Anc21:0.0014648313999999774,((Hism:0.010934391799999998,Hnum:0.00853842589999998)Anc29:0.00145747340000002,Hbes:0.01286533379999999)Anc22:0.0008425928999999943)Anc16:0.0022114987000000363,Hnat:0.014734212999999996)MelpomeneSilvaniformis:0.01669843279999994,((Hwal:0.012354115599999993,Hbur:0.008478383399999956)Anc17:0.007174984200000067,Hege:0.012450162799999998)Wallacei:0.020106523499999973)WallaceiMelpomeneSilvaniformis:0.0027426320000000226,(((Hdor:0.010678678599999936,Hxan:0.0037508767999999693)Anc18:0.0050869265000000885,Hhie:0.008823579299999995)Anc11:0.009892071000000002,Hheb:0.01371957079999997)Doris:0.014872273399999969)DorisWallaceiMelpomeneSilvaniformis:0.002858650099999971,Haoe:0.05007555829999999)AoedesDorisWallaceiMelpomeneSilvaniformis:0.005401554400000008,((((((((Hsap:0.004773281099999971,Hhew:0.003146042200000032)Anc34:0.0004602516999999695,(Hcon:0.0024528891999999747,Hele:0.0030787116999999586)Anc35:0.0011447064999999812)Anc30:0.0023424109999999887,Hant:0.007829635300000026)Anc23:0.0028368990000000593,(Hsar:0.006339082500000037,Hleu:0.008220619599999979)Anc24:0.0031723685000000224)Anc19:0.007903264999999937,Hric:0.020677386200000036)Anc12:0.0009953779000000385,(Hdem:0.0008117181000000251,Hert:0.0007670346999999467)Anc13:0.010973798600000029)Anc07:0.0013319422999999997,(Hper:0.006816607199999991,Hcha:0.00507812409999997)Anc08:0.013851352500000025)SaraSapho:0.004005008000000032,((((((Hhim:0.006117437700000061,Hlat:0.007049078800000053)Anc31:0.002257163499999937,Heet:0.00903959919999997)Anc25:0.001100892200000092,(Herd:0.007866239500000094,Hpet:0.008716476899999992)Anc26:0.0017808431000000402)Anc20:0.004957653499999992,Hher:0.01114041110000008)Anc14:0.0038197568999999154,Hhec:0.013713833799999908)Anc09:0.0020094666000000094,((Hhor:0.00889558459999995,Hcly:0.011993779699999929)Anc15:0.0033572640000000264,Htel:0.009705159200000035)Anc10:0.004869761599999967)Erato:0.0013045297000000566)EratoSaraSapho:0.016220800099999932)Heliconius:0.029514828100000057,(((Elyb:0.0299148387,Etal:0.02194292200000003)Anc05:0.008363003200000052,Eali:0.034603311100000034)Anc02:0.00710912269999997,((Evib:0.00826686259999998,Elam:0.008032134100000032)Anc06:0.013568470000000055,Eisa:0.02526911860000003)Anc03:0.011943215199999968)Eueides:0.038730827700000026)EueidesHeliconius:0.030820143799999977,(((Avcr:0.010631127699999965,Avpe:0.014029160400000018)Anc04:0.00670582190000002,Avfl:0.016425588099999944)Agraulis:0.04397123749999998,Djun:0.04309329220000002)DioneAgraulis:0.04418981529999999)DioneAgraulisEueidesHeliconius:0.015906054799999958,(((Ptel:0.06271842510000003,Diul:0.06912531960000001)Anc01:0.013234128200000028,Dpha:0.08595198570000007)Anc00:0.006596193799999983,Pdid:0.07419015350000002)OtherGenera:0.014268624800000018)Heliconiini:0.10024188680000001,Smor:0.13099943050000007)Heliconiinae:0.11228131870000002,(Mcin:0.15858968770000004,Jcoe:0.11762040340000002)Nymphalinae:0.09381540420000001)HeliconiinaeNymphalinae:0.05243273250000002,Bany:0.25790259150000006)BanyHeliconiinaeNymphalinae:0.17065359,Dple:0.17065358999999994)Nymphalidae;

F

[francicco]

I could rename the nodes once the alignment is done I guess. But I can't see why that should be a problem, a string is a string, I guess... F

[glennhickey]

Could you share the full input file (tree and sequence names) please? I will try to quickly reproduce here with dummy sequences.

[francicco]

Thanks a lot Glen, very much apprecciated!

(((((((((((((((((Htim:0.0038869669000000107,Hheu:0.0049981889000000335):0.0033437624000000055,(Hpac:0.00986230450000003,Hcyd:0.0040755693000000814):0.002230524999999983):0.0020316084999999484,Hmel:0.007262151699999997):0.004617452000000077,(((Hhel:0.00724042530000002,Hatt:0.008730127099999985):0.0019464672000000238,(Helv:0.006136523899999968,Hpar:0.0061621217999999756):0.002825525600000045):0.001173160199999912,Heth:0.014242937799999966):0.0015224138000000664):0.0014648313999999774,((Hism:0.010934391799999998,Hnum:0.00853842589999998):0.00145747340000002,Hbes:0.01286533379999999):8.425928999999943E-4):0.0022114987000000363,Hnat:0.014734212999999996)MelpomeneSilvaniformis:0.01669843279999994,((Hwal:0.012354115599999993,Hbur:0.008478383399999956):0.007174984200000067,Hege:0.012450162799999998)Wallacei:0.020106523499999973)WallaceiMelpomeneSilvaniformis:0.0027426320000000226,(((Hdor:0.010678678599999936,Hxan:0.0037508767999999693):0.0050869265000000885,Hhie:0.008823579299999995):0.009892071000000002,Hheb:0.01371957079999997)Doris:0.014872273399999969)DorisWallaceiMelpomeneSilvaniformis:0.002858650099999971,Haoe:0.05007555829999999)AoedesDorisWallaceiMelpomeneSilvaniformis:0.005401554400000008,((((((((Hsap:0.004773281099999971,Hhew:0.003146042200000032):4.602516999999695E-4,(Hcon:0.0024528891999999747,Hele:0.0030787116999999586):0.0011447064999999812):0.0023424109999999887,Hant:0.007829635300000026):0.0028368990000000593,(Hsar:0.006339082500000037,Hleu:0.008220619599999979):0.0031723685000000224):0.007903264999999937,Hric:0.020677386200000036):9.953779000000385E-4,(Hdem:8.117181000000251E-4,Hert:7.670346999999467E-4):0.010973798600000029):0.0013319422999999997,(Hper:0.006816607199999991,Hcha:0.00507812409999997):0.013851352500000025)SaraSapho:0.004005008000000032,((((((Hhim:0.006117437700000061,Hlat:0.007049078800000053):0.002257163499999937,Heet:0.00903959919999997):0.001100892200000092,(Herd:0.007866239500000094,Hpet:0.008716476899999992):0.0017808431000000402):0.004957653499999992,Hher:0.01114041110000008):0.0038197568999999154,Hhec:0.013713833799999908):0.0020094666000000094,((Hhor:0.00889558459999995,Hcly:0.011993779699999929):0.0033572640000000264,Htel:0.009705159200000035):0.004869761599999967)Erato:0.0013045297000000566)EratoSaraSapho:0.016220800099999932)Heliconius:0.029514828100000057,(((Elyb:0.0299148387,Etal:0.02194292200000003):0.008363003200000052,Eali:0.034603311100000034):0.00710912269999997,((Evib:0.00826686259999998,Elam:0.008032134100000032):0.013568470000000055,Eisa:0.02526911860000003):0.011943215199999968)Eueides:0.038730827700000026)EueidesHeliconius:0.030820143799999977,(((Avcr:0.010631127699999965,Avpe:0.014029160400000018):0.00670582190000002,Avfl:0.016425588099999944)Agraulis:0.04397123749999998,Djun:0.04309329220000002)DioneAgraulis:0.04418981529999999)DioneAgraulisEueidesHeliconius:0.015906054799999958,(((Ptel:0.06271842510000003,Diul:0.06912531960000001):0.013234128200000028,Dpha:0.08595198570000007):0.006596193799999983,Pdid:0.07419015350000002)OtherGenera:0.014268624800000018)Heliconiini:0.10024188680000001,Smor:0.13099943050000007)Heliconiinae:0.11228131870000002,(Mcin:0.15858968770000004,Jcoe:0.11762040340000002)Nymphalinae:0.09381540420000001)HeliconiinaeNymphalinae:0.05243273250000002,Bany:0.25790259150000006)BanyHeliconiinaeNymphalinae:0.17065359,Dple:0.17065358999999994)Nymphalidae;

Bany /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Bany.assembly.v1.2.fasta.masked
Dple /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Dple.assembly.v3.0.fasta.masked
Jcoe /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Jcoe.assembly.v1.1.fasta.masked
*Mcin /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Mcin.assembly.v2.0.fasta.masked
*Smor /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Smor.assembly.v1.0.fasta.masked
*Ptel /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Ptel.assembly.v1.2.fasta.masked
*Dpha /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Dpha.assembly.v1.2.fasta.masked
*Diul /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Diul.assembly.v1.1.fasta.masked
*Pdid /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Pdid.assembly.v1.3.fasta.masked
*Avfl /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Avfl.assembly.v1.2.fasta.masked
Avcr /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Avcr.assembly.v1.1.fasta.masked
Avpe /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Avpe.assembly.v0.9.fasta.masked
*Djun /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Djun.assembly.v1.2.fasta.masked
Eali /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Eali.assembly.v0.9.fasta.masked
Etal /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Etal.assembly.v1.5.fasta.masked
Elyb /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Elyb.assembly.v0.9.fasta.masked
*Eisa /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Eisa.assembly.v1.1.fasta.masked
Elam /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Elam.assembly.v0.9.fasta.masked
Evib /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Evib.assembly.v0.9.fasta.masked
Hhor /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hhor.assembly.v0.9.fasta.masked
Hcly /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hcly.assembly.v0.9.fasta.masked
Htel /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Htel.assembly.v1.5.fasta.masked
Hpet /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hpet.assembly.v0.9.fasta.masked
Hlat /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hlat.assembly.v1.5.fasta.masked
Hhec /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hhec.assembly.v1.5.fasta.masked
Hher /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hher.assembly.v1.0.fasta.masked
Hhim /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hhim.assembly.v1.1.fasta.masked
*Herd /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Herd.assembly.v1.0.fasta.masked
Heet /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Heet.assembly.v0.9.fasta.masked
Hert /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hert.assembly.v0.9.fasta.masked
Hdem /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hdem.assembly.v1.5.fasta.masked
Hper /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hper.assembly.v0.9.fasta.masked
*Hcha /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hcha.assembly.v1.1.fasta.masked
Hric /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hric.assembly.v0.9.fasta.masked
Hant /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hant.assembly.v0.9.fasta.masked
Hleu /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hleu.assembly.v0.9.fasta.masked
*Hsar /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hsar.assembly.v1.2.fasta.masked
Hhew /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hhew.assembly.v0.9.fasta.masked
Hsap /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hsap.assembly.v0.9.fasta.masked
Hcon /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hcon.assembly.v0.9.fasta.masked
Hele /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hele.assembly.v0.9.fasta.masked
*Haoe /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Haoe.assembly.v1.2.fasta.masked
Hheb /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hheb.assembly.v0.9.fasta.masked
*Hdor /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hdor.assembly.v1.2.fasta.masked
Hhie /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hhie.assembly.v0.9.fasta.masked
Hxan /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hxan.assembly.v0.9.fasta.masked
Hege /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hege.assembly.v0.9.fasta.masked
Hbur /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hbur.assembly.v1.5.fasta.masked
Hwal /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hwal.assembly.v0.9.fasta.masked
Hnat /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hnat.assembly.v1.0.fasta.masked
*Hmel /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hmel.assembly.v2.5.fasta.masked
Hcyd /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hcyd.assembly.v1.0.fasta.masked
Hpac /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hpac.assembly.v1.0.fasta.masked
Htim /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Htim.assembly.v0.9.fasta.masked
Hheu /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hheu.assembly.v0.9.fasta.masked
Hism /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hism.assembly.v0.9.fasta.masked
Hnum /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hnum.assembly.v1.5.fasta.masked
Hbes /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hbes.assembly.v1.5.fasta.masked
Helv /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Helv.assembly.v1.5.fasta.masked
Heth /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Heth.assembly.v0.9.fasta.masked
Hhel /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hhel.assembly.v1.5.fasta.masked
Hatt /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hatt.assembly.v1.0.fasta.masked
Hpar /user/work/tk19812/HeliconiniiProject/CompleteAssemblies/AssembledGenomes/Hpar.assembly.v1.5.fasta.masked

[glennhickey]

Seems to be working fine here, so I guess it's nothing to do with the tree or species names. If I"m to look any further, you'd need to share some sequences. For the Anc15 crash you posted upthread that would be

Hcha
Hcly
Herd
Hhor
Hsar

[francicco]

do you have an email, I'll sed you links to them F

[glennhickey]

I'll let it run through to the end, but it doesn't look like I can reproduce your crash. From your log, the crash happens at

[2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Outgroup events: Herd Hcha Hsar
[2021-11-26T12:36:48+0000] [MainThread] [I] [toil-rt] cactus_consolidated(Anc15): Reference event: Anc15
(... segmentation fault ...)

But in my run (using v2.0.4 binaries) it's already at

cactus_consolidated(Anc15): Outgroup events: Herd Hcha Hsar
cactus_consolidated(Anc15): Reference event: Anc15
cactus_consolidated(Anc15): Loaded the parameters files, 0 seconds have elapsed
cactus_consolidated(Anc15): Set up the cactus disk, 0 seconds have elapsed
cactus_consolidated(Anc15): Constructed the first flower
cactus_consolidated(Anc15): Going to build the event tree with newick string: ((Hsar:0.019742036200000035,Hcha:0.018929476599999995)SaraSapho:0.0040050080000
00032,(Herd:0.02043395960000005,(Hhor:0.00889558459999995,Hcly:0.011993779699999929)Anc15:0.008227025599999993)Erato:0.0013045297000000566)EratoSaraSapho;
cactus_consolidated(Anc15): Parsed the tree
cactus_consolidated(Anc15): Constructed the basic event tree
cactus_consolidated(Anc15): Assigning sequence /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpn2lylyss.tmp to Hcha
cactus_consolidated(Anc15): Processing file: /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpn2lylyss.tmp
cactus_consolidated(Anc15): The file /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpn2lylyss.tmp is specified incomplete, the sequences will not be
 attached
1 jobs are running, 1 jobs are issued and waiting to run
cactus_consolidated(Anc15): Assigning sequence /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpzptz13dl.tmp to Hcly
cactus_consolidated(Anc15): Processing file: /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpzptz13dl.tmp
cactus_consolidated(Anc15): The file /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpzptz13dl.tmp is specified incomplete, the sequences will not be
 attached
cactus_consolidated(Anc15): Assigning sequence /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpcjbtpiut.tmp to Herd
cactus_consolidated(Anc15): Processing file: /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpcjbtpiut.tmp
cactus_consolidated(Anc15): The file /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpcjbtpiut.tmp is specified incomplete, the sequences will not be
 attached
cactus_consolidated(Anc15): Assigning sequence /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpcmh10hc3.tmp to Hhor
cactus_consolidated(Anc15): Processing file: /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpcmh10hc3.tmp
cactus_consolidated(Anc15): The file /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmpcmh10hc3.tmp is specified incomplete, the sequences will not be
 attached
cactus_consolidated(Anc15): Assigning sequence /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmp32u_3m2p.tmp to Hsar
cactus_consolidated(Anc15): Processing file: /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmp32u_3m2p.tmp
cactus_consolidated(Anc15): The file /work/work/b60a150d5d825c858d03eba40e522428/612f/fda6/tmp32u_3m2p.tmp is specified incomplete, the sequences will not be
 attached
cactus_consolidated(Anc15): Constructed the initial flower with 129308 sequences and 10 events with string: (((Hsar:0.019742,Hcha:0.0189295)SaraSapho:0.00400
501,(Herd:0.020434,(Hhor:0.00889558,Hcly:0.0119938)Anc15:0.00822703)Erato:0.00130453)EratoSaraSapho:9.22337e+18)ROOT:9.22337e+18;
cactus_consolidated(Anc15): Established the first Flower in the hierarchy, 9 seconds have elapsed
cactus_consolidated(Anc15): Converted alignment coordinates, 185 seconds have elapsed
cactus_consolidated(Anc15): Stripped the unique IDs, 185 seconds have elapsed
cactus_consolidated(Anc15): Starting annealing round with a minimum chain length of 64 and an alignment trim of 3

So unfortunately I don't have any answers for you. Either I made a mistake when trying to reproduce, or there is something with your computer and/or cactus installation that's causing the crash. I think it's conceivable that running out of disk or memory could be a factor (it's using 32G of both so far in my run fwiw), but it's impossible to say.

If you want to debug further on your own, you'd need to

• build your own cactus binaries
• run with --cleanWorkDir never
• then but the failing cactus_consolidated command into gdb to see where it's failing.

[francicco]

Thank you Glenn. That's a very bad news since I tried to compile the binaries so many times and I always failed at some point. It was a blessing when the precompiled version was available. I can run it with --cleanWorkDir never and see happens. Now I don't have any functional version since python compatibilities made it harder. Thank you F

[diekhans]

Compiled and running gdb requires some hard core C++ skils

Allocating more memory is well worth trying. Sometimes poorly handled memory errors don't result in obvious messages.

What python issues are blocking you

Francesco Cicconardi @.***> writes:

Thank you Glenn. That's a very bad news since I tried to compile the binaries so many times and I always failed at some point. It was a blessing when the precompiled version was available. I can run it with --cleanWorkDir never and see happens. Now I don't have any functional version since python compatibilities made it harder. Thank you

[francicco]

I mean, I'm not totally a newbie when it comes to complile, but it should not be my job. Compile cactus would require a lot of time and I'd rather do science. Also, the cluster atm is giving so much trouble that adding this is an extra complication.

I'll try once again to reinstall a pre-consolidate version and see if it works this time.

The nodes have a minimum of 180Gb and I have a node with 700Gb, I'm not sure if memory is a real issue

I'll let you know what happens. Thanks again. You're extra helpfull. F

[diekhans]

Hey Francesco,

I have been compiling C/C++ for like 30+ years, and I still have problems with cactus.

Could there be a memory limit below the minimum. Seeing as your phyloP run appears to have memory problems.

Francesco Cicconardi @.***> writes:

I mean, I'm not totally a newbie when it comes to complile, but it should not be my job. Compile cactus would require a lot of time and I'd rather do science. Also, the cluster atm is giving so much trouble that adding this is an extra complication.

I'll try once again to reinstall a pre-consolidate version and see if it works this time.

The nodes have a minimum of 180Gb and I have a node with 700Gb, I'm not sure if memory is a real issue

I'll let you know what happens. Thanks again. You're extra helpfull. F

[francicco]

Thank you for your support Mark. I installed the 1.3.0 without errors or warnings. I haven't tried yet, I wanna give a last try to consolidate. The problem I have now is that the cluster is in big problem. It's basically not working. I'll let you know as soon as I've tested it.

Thanks really a lot!!! F

[francicco]

So...

This is the command line I'm using... cactus 63Nymphalidae.Cactus.scOG_ML.jobStore 63Nymphalidae.Cactus.scOG_ML.conf 63Nymphalidae.Cactus.scOG_ML.hal --maxMemory 160G --workDir /tmp --binariesMode local --stats --logFile 63Nymphalidae.Cactus.Hmel.log --maxCores 48 --defaultMemory 160G --realTimeLogging --cleanWorkDir never --restart

and this is what I'm getting now:

[2021-12-03T14:28:47+0000] [MainThread] [I] [toil] Running Toil version 5.4.0-87293d63fa6c76f03bed3adf93414ffee67bf9a7 on host bp1-compute134.data.bp.acrc.priv.
[2021-12-03T14:28:47+0000] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2021-12-03T14:28:53+0000] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2021-12-03T14:28:53+0000] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
Traceback (most recent call last):
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/progressive/cactus_progressive.py", line 406, in main
    runCactusProgressive(options)
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/progressive/cactus_progressive.py", line 416, in runCactusProgressive
    halID = toil.restart()
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/common.py", line 874, in restart
    return self._runMainLoop(rootJobDescription)
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/common.py", line 1127, in _runMainLoop
    return Leader(config=self.config,
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/leader.py", line 229, in run
    self.innerLoop()
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/leader.py", line 604, in innerLoop
    self._processReadyJobs()
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/leader.py", line 513, in _processReadyJobs
    self._processReadyJob(updatedJob, resultStatus)
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/leader.py", line 461, in _processReadyJob
    self.issueJob(readyJob)
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/leader.py", line 732, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/batchSystems/singleMachine.py", line 615, in issueBatchJob
    self.checkResourceRequest(jobDesc.memory, cores, jobDesc.disk, job_name=jobDesc.jobName,
  File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/batchSystems/abstractBatchSystem.py", line 319, in checkResourceRequest
    raise InsufficientSystemResources(msg)
toil.batchSystems.abstractBatchSystem.InsufficientSystemResources: The job CactusConsolidated is requesting 200.0 cores, more than the maximum of 48 cores that SingleMachineBatchSystem was configured with, or enforced by --maxCores.Scale is set to 1.0.

why The job CactusConsolidated is requesting 200.0 cores

[glennhickey]

That's an annoying toil limitation. When you run with --restart it makes jobs with the same resource requirements as the first time you ran.

[francicco]

I'm getting this error...

[2021-12-05T12:39:44+0000] [Thread-1  ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job _toil_worker ProgressiveUp file:/user/work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/63Nymphalidae.Cactus.scOG_ML.jobStore kind-ProgressiveUp/instance-prjgryxt.
[2021-12-05T12:39:44+0000] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'ProgressiveUp' kind-ProgressiveUp/instance-prjgryxt
Exit reason: None
[2021-12-05T12:39:44+0000] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'ProgressiveUp' kind-ProgressiveUp/instance-prjgryxt
[2021-12-05T12:39:44+0000] [MainThread] [W] [toil.leader] Log from job "kind-ProgressiveUp/instance-prjgryxt" follows:
=========>
        [2021-12-05T12:39:43+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
        [2021-12-05T12:39:43+0000] [MainThread] [I] [toil] Running Toil version 5.4.0-87293d63fa6c76f03bed3adf93414ffee67bf9a7 on host bp1-compute221.data.bp.acrc.priv.
        [2021-12-05T12:39:43+0000] [MainThread] [I] [toil.worker] Working on job 'ProgressiveUp' kind-ProgressiveUp/instance-prjgryxt
        [2021-12-05T12:39:43+0000] [MainThread] [I] [toil.worker] Loaded body Job('ProgressiveUp' kind-ProgressiveUp/instance-prjgryxt) from description 'ProgressiveUp' kind-ProgressiveUp/instance-prjgryxt
        [2021-12-05T12:39:43+0000] [MainThread] [I] [cactus.progressive.cactus_progressive] Progressive Up: Nymphalinae
        [2021-12-05T12:39:43+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
        [2021-12-05T12:39:43+0000] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-00797214092546028b0a1592514acbfa/cactus_progressive_config.xml' to path '/tmp/80aecfee4bd35b10a4e88c546a106bcc/4d6d/e13b/tmp11mkqcxu.tmp'
        Traceback (most recent call last):
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/worker.py", line 393, in workerScript
            job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/shared/common.py", line 932, in _runner
            super(RoundedJob, self)._runner(*args, jobStore=jobStore,
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/job.py", line 2360, in _runner
            returnValues = self._run(jobGraph=None, fileStore=fileStore)
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/job.py", line 2281, in _run
            return self.run(fileStore)
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/cactus/progressive/cactus_progressive.py", line 156, in run
            experimentFile = fileStore.readGlobalFile(self.project.expIDMap[self.event])
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/fileStores/nonCachingFileStore.py", line 91, in readGlobalFile
            self.jobStore.readFile(fileStoreID, localFilePath, symlink=symlink)
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/jobStores/fileJobStore.py", line 440, in readFile
            self._checkJobStoreFileID(jobStoreFileID)
          File "/user/work/tk19812/software/cactus-bin-v2.0.4/venv/lib/python3.8/site-packages/toil/jobStores/fileJobStore.py", line 724, in _checkJobStoreFileID
            raise NoSuchFileException(jobStoreFileID)
        toil.jobStores.abstractJobStore.NoSuchFileException: File 'files/no-job/file-75733186c1a345f3989f7335d25ba28c/Nymphalinae_experiment.xml' does not exist.
        [2021-12-05T12:39:43+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host bp1-compute221.data.bp.acrc.priv
<=========

[glennhickey]

closed via #743