search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
529 stars 111 forks source link

Not enough memory! #310

Open francicco opened 4 years ago

francicco commented 4 years ago

Hi Mark,

I got this error:

Not enough memory! User limited to 429496729600 bytes but we only have 269441224704 bytes.

I tried to restart the job by setting less memory --defaultMemory, but apparently it's not working.

Traceback (most recent call last):
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/cactus/progressive/cactus_progressive.py", line 406, in main
    runCactusProgressive(options)
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/cactus/progressive/cactus_progressive.py", line 416, in runCactusProgressive
    halID = toil.restart()
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/toil/common.py", line 858, in restart
    self._cacheAllJobs()
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/toil/common.py", line 1042, in _cacheAllJobs
    self._jobCache = {jobGraph.jobStoreID: jobGraph for jobGraph in self._jobStore.jobs()}
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/toil/common.py", line 1042, in <dictcomp>
    self._jobCache = {jobGraph.jobStoreID: jobGraph for jobGraph in self._jobStore.jobs()}
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/toil/jobStores/fileJobStore.py", line 251, in jobs
    yield self.load(jobId)
  File "/mnt/storage/scratch/tk19812/software/cactus-bin-v1.1.1/venv/lib/python3.7/site-packages/toil/jobStores/fileJobStore.py", line 210, in load
    job = pickle.load(fileHandle)
EOFError: Ran out of input

Is there anything I could do to overcome this problem?

Thanks a lot F

diekhans commented 4 years ago

I think it needs more memory. It should scale it up. This is coming from Toil @DailyDreaming ??

However, this seems 500 GB seems insanely larger for a python process.

francicco commented 4 years ago

Should I give less memory since the beginning? I don't know what to do. F

diekhans commented 4 years ago

probably need to up the max memory, but might be worth asking Toil group.

Sorry, can't really look more now, deadline

Francesco Cicconardi notifications@github.com writes:

Should I give less memory since the beginning? I don't know what to do. F

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/ComparativeGenomicsToolkit/cactus/issues/310#issuecomment-687832333 Should I give less memory since the beginning? I don't know what to do. F

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.*

francicco commented 4 years ago

I just don't have more than 500Gb of memory F

diekhans commented 4 years ago

It seems very odd that a python process would use so much memory.

It is a long-weekend holiday here, so not many people to respond.

Francesco Cicconardi notifications@github.com writes:

I just don't have more than 500Gb of memory

francicco commented 4 years ago

I'm sorry to keep bothering you, Mark. I'm trying to test CACTUS on another cluster, which has more memory although the walltime is way shorter. Unfortunately, there's something wrong with cactus, I got this aligning the test example:

bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,470 MainThread INFO toil.leader: Job ended: 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,473 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with job batch system ID: 253 and cores: 1, disk: 2.0 G, and memory: 500.0 M
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,538 MainThread INFO toil.leader: Job ended: 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,542 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with job batch system ID: 254 and cores: 1, disk: 2.0 G, and memory: 500.0 M
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/worker_log.txt
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/worker_log.txt
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:16,469 MainThread INFO toil-rt: 2020-09-07 10:41:16.469003: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
            <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
        </st_kv_database_conf>
     --bed"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:20,838 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:24,712 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,016 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-f4z694yy follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
    INFO:toil-rt:2020-09-07 10:41:16.469003: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
                <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
            </st_kv_database_conf>
         --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f804a911555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp70058_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy to 1
    WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy to 2147483648 bytes
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,045 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with job batch system ID: 255 and cores: 1, disk: 2.0 G, and memory: 2.0 G
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,123 MainThread INFO toil-rt: 2020-09-07 10:41:25.123344: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
            <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
        </st_kv_database_conf>
     --bed"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,288 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,296 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,297 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-yyklg10i follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
    INFO:toil-rt:2020-09-07 10:41:25.123344: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
                <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
            </st_kv_database_conf>
         --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa518494555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp71684_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i to 1
    WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i to 2147483648 bytes
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,299 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with job batch system ID: 256 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/worker_log.txt
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:29,328 MainThread INFO toil-rt: 2020-09-07 10:41:29.328871: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
            <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
        </st_kv_database_conf>
     --bed"
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/worker_log.txt
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:34,329 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:37,268 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:37,404 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-f4z694yy follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
    INFO:toil-rt:2020-09-07 10:41:29.328871: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
                <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
            </st_kv_database_conf>
         --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fcc032cf555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp72449_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy to 0
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:37,844 MainThread WARNING toil.leader: Job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,071 MainThread INFO toil-rt: 2020-09-07 10:41:40.071790: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
            <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
        </st_kv_database_conf>
     --bed"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,327 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,332 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,332 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-yyklg10i follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
    INFO:toil-rt:2020-09-07 10:41:40.071790: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
                <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
            </st_kv_database_conf>
         --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7ddc198555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp74487_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i to 0
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,333 MainThread WARNING toil.leader: Job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:47,916 MainThread INFO toil-rt: 2020-09-07 10:41:47.916196: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:49,428 MainThread INFO toil-rt: 2020-09-07 10:41:49.428707: Running the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:49,444 MainThread INFO toil-rt: 2020-09-07 10:41:49.444236: Successfully ran the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1" in 0.0076715946197509766 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:50,018 MainThread INFO toil-rt: 2020-09-07 10:41:50.018908: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:51,826 MainThread INFO toil-rt: 2020-09-07 10:41:51.826316: Running the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:51,838 MainThread INFO toil-rt: 2020-09-07 10:41:51.838081: Successfully ran the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1" in 0.0073244571685791016 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:48,012 MainThread INFO toil-rt: 2020-09-07 10:42:48.012228: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:48,575 MainThread INFO toil-rt: 2020-09-07 10:42:48.575290: Successfully ran the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE" in 0.009398221969604492 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,730 MainThread INFO toil.leader: Job ended: 'KtServerService' kind-KtServerService/instance-hnakmgk1
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,735 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'KtServerService' kind-KtServerService/instance-hnakmgk1
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,736 MainThread WARNING toil.leader: Log from job kind-KtServerService/instance-hnakmgk1 follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
    INFO:toil-rt:2020-09-07 10:40:46.804704: Running the command: "netstat -tuplen"
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    INFO:toil-rt:2020-09-07 10:40:46.823367: Successfully ran the command: "netstat -tuplen" in 0.012812137603759766 seconds
    INFO:cactus.shared.common:Running the command ['ktserver', '-port', '29555', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
    INFO:toil-rt:2020-09-07 10:40:46.835407: Running the command: "ktserver -port 29555 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:40:47.845364: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:41:47.916196: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE', '1']
    INFO:toil-rt:2020-09-07 10:41:49.428707: Running the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1"
    INFO:toil-rt:2020-09-07 10:41:49.444236: Successfully ran the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1" in 0.0076715946197509766 seconds
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:42:48.012228: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
    1
    INFO:toil-rt:2020-09-07 10:42:48.575290: Successfully ran the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE" in 0.009398221969604492 seconds
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
        returnValues = self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
        raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
    RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-hnakmgk1 with ID kind-KtServerService/instance-hnakmgk1 to 1
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,738 MainThread WARNING toil.leader: Job 'KtServerService' kind-KtServerService/instance-hnakmgk1 with ID kind-KtServerService/instance-hnakmgk1 is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:50,089 MainThread INFO toil-rt: 2020-09-07 10:42:50.089057: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:50,101 MainThread INFO toil-rt: 2020-09-07 10:42:50.101638: Successfully ran the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE" in 0.005008220672607422 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,108 MainThread INFO toil.leader: Job ended: 'KtServerService' kind-KtServerService/instance-8pd4r5v0
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,112 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'KtServerService' kind-KtServerService/instance-8pd4r5v0
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,113 MainThread WARNING toil.leader: Log from job kind-KtServerService/instance-8pd4r5v0 follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
    INFO:toil-rt:2020-09-07 10:40:48.905109: Running the command: "netstat -tuplen"
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    INFO:toil-rt:2020-09-07 10:40:48.927017: Successfully ran the command: "netstat -tuplen" in 0.016075611114501953 seconds
    INFO:cactus.shared.common:Running the command ['ktserver', '-port', '5445', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
    INFO:toil-rt:2020-09-07 10:40:48.941000: Running the command: "ktserver -port 5445 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:40:49.949341: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:41:50.018908: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE', '1']
    INFO:toil-rt:2020-09-07 10:41:51.826316: Running the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1"
    INFO:toil-rt:2020-09-07 10:41:51.838081: Successfully ran the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1" in 0.0073244571685791016 seconds
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:42:50.089057: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
    1
    INFO:toil-rt:2020-09-07 10:42:50.101638: Successfully ran the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE" in 0.005008220672607422 seconds
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
        returnValues = self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
        raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
    RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 with ID kind-KtServerService/instance-8pd4r5v0 to 1
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,118 MainThread WARNING toil.leader: Job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 with ID kind-KtServerService/instance-8pd4r5v0 is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:30,883 MainThread INFO toil.leader: Finished toil run with 22 failed jobs.
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:30,884 MainThread INFO toil.leader: Failed jobs at end of the run: 'KtServerService' kind-KtServerService/instance-hnakmgk1 'StartPrimaryDB' kind-StartPrimaryDB/instance-_oa_qio2 'ProgressiveUp' kind-ProgressiveUp/instance-lokvicq8 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho 'StartPrimaryDB' kind-StartPrimaryDB/instance-4djpc7nx 'ProgressiveDown' kind-ProgressiveDown/instance-li8hv0fj 'ProgressiveNext' kind-ProgressiveNext/instance-a7irsvu6 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-cfldch16 'ProgressiveDown' kind-ProgressiveDown/instance-8kgdnuhg 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-c158ubud 'ProgressiveDown' kind-ProgressiveDown/instance-ycp1al2l 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-i3r410fe 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-8x084ped 'KtServerService' kind-KtServerService/instance-8pd4r5v0 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-abhxdgwj 'ProgressiveNext' kind-ProgressiveNext/instance-pju96b5_ 'ProgressiveDown' kind-ProgressiveDown/instance-hnb46cgh 'ProgressiveUp' kind-ProgressiveUp/instance-_p3wau6d 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-duyjfuj1
Workflow Progress 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 257/257 (0 failures) [06:08<00:00, 0.70 jobs/s]
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:30,899 MainThread INFO toil.realtimeLogger: Stopping real-time logging server.
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:31,154 MainThread INFO toil.realtimeLogger: Joining real-time logging server thread.
Traceback (most recent call last):
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/progressive/cactus_progressive.py", line 406, in main
    runCactusProgressive(options)
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/progressive/cactus_progressive.py", line 454, in runCactusProgressive
    halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/common.py", line 829, in start
    return self._runMainLoop(rootJobGraph)
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/common.py", line 1120, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/leader.py", line 269, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/work/tk19812/software/cactus-bin-v1.2.1/jobstore' contains 22 failed jobs: 'KtServerService' kind-KtServerService/instance-hnakmgk1, 'StartPrimaryDB' kind-StartPrimaryDB/instance-_oa_qio2, 'ProgressiveUp' kind-ProgressiveUp/instance-lokvicq8, 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho, 'StartPrimaryDB' kind-StartPrimaryDB/instance-4djpc7nx, 'ProgressiveDown' kind-ProgressiveDown/instance-li8hv0fj, 'ProgressiveNext' kind-ProgressiveNext/instance-a7irsvu6, 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-cfldch16, 'ProgressiveDown' kind-ProgressiveDown/instance-8kgdnuhg, 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-c158ubud, 'ProgressiveDown' kind-ProgressiveDown/instance-ycp1al2l, 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i, 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-i3r410fe, 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-8x084ped, 'KtServerService' kind-KtServerService/instance-8pd4r5v0, 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-abhxdgwj, 'ProgressiveNext' kind-ProgressiveNext/instance-pju96b5_, 'ProgressiveDown' kind-ProgressiveDown/instance-hnb46cgh, 'ProgressiveUp' kind-ProgressiveUp/instance-_p3wau6d, 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy, 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1, 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-duyjfuj1
Log from job 'KtServerService' kind-KtServerService/instance-hnakmgk1 follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
    INFO:toil-rt:2020-09-07 10:40:46.804704: Running the command: "netstat -tuplen"
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    INFO:toil-rt:2020-09-07 10:40:46.823367: Successfully ran the command: "netstat -tuplen" in 0.012812137603759766 seconds
    INFO:cactus.shared.common:Running the command ['ktserver', '-port', '29555', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
    INFO:toil-rt:2020-09-07 10:40:46.835407: Running the command: "ktserver -port 29555 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:40:47.845364: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:41:47.916196: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE', '1']
    INFO:toil-rt:2020-09-07 10:41:49.428707: Running the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1"
    INFO:toil-rt:2020-09-07 10:41:49.444236: Successfully ran the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1" in 0.0076715946197509766 seconds
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:42:48.012228: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
    1
    INFO:toil-rt:2020-09-07 10:42:48.575290: Successfully ran the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE" in 0.009398221969604492 seconds
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
        returnValues = self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
        raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
    RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-hnakmgk1 with ID kind-KtServerService/instance-hnakmgk1 to 1
<=========
Log from job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
    INFO:toil-rt:2020-09-07 10:41:40.071790: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
                <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
            </st_kv_database_conf>
         --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7ddc198555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp74487_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i to 0
<=========
Log from job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
    INFO:toil-rt:2020-09-07 10:40:48.905109: Running the command: "netstat -tuplen"
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    INFO:toil-rt:2020-09-07 10:40:48.927017: Successfully ran the command: "netstat -tuplen" in 0.016075611114501953 seconds
    INFO:cactus.shared.common:Running the command ['ktserver', '-port', '5445', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
    INFO:toil-rt:2020-09-07 10:40:48.941000: Running the command: "ktserver -port 5445 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:toil.lib.bioio:Ktserver running.
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:40:49.949341: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:41:50.018908: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE', '1']
    INFO:toil-rt:2020-09-07 10:41:51.826316: Running the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1"
    INFO:toil-rt:2020-09-07 10:41:51.838081: Successfully ran the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1" in 0.0073244571685791016 seconds
    INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
    INFO:toil-rt:2020-09-07 10:42:50.089057: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
    1
    INFO:toil-rt:2020-09-07 10:42:50.101638: Successfully ran the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE" in 0.005008220672607422 seconds
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
        returnValues = self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
        raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
    RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 with ID kind-KtServerService/instance-8pd4r5v0 to 1
<=========
Log from job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy follows:
=========>
    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
    INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
    INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
    INFO:toil-rt:2020-09-07 10:41:29.328871: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
                <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
            </st_kv_database_conf>
         --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fcc032cf555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp72449_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
        raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
    RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
    ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy to 0
<=========

Cheers F

diekhans commented 4 years ago

This is the first error I see: ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp70058_0: No such file or directory

Francesco Cicconardi notifications@github.com writes:

I'm sorry to keep bothering you, Mark. I'm trying to test CACTUS on another cluster, which has more memory although the walltime is way shorter. Unfortunately, there's something wrong with cactus, I got this aligning the test example:

bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,470 MainThread INFO toil.leader: Job ended: 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,473 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with job batch system ID: 253 and cores: 1, disk: 2.0 G, and memory: 500.0 M
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,538 MainThread INFO toil.leader: Job ended: 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,542 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with job batch system ID: 254 and cores: 1, disk: 2.0 G, and memory: 500.0 M
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/worker_log.txt
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/worker_log.txt
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:16,469 MainThread INFO toil-rt: 2020-09-07 10:41:16.469003: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
          <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
      </st_kv_database_conf>
   --bed"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:20,838 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:24,712 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,016 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-f4z694yy follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
  INFO:toil-rt:2020-09-07 10:41:16.469003: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
              <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
          </st_kv_database_conf>
       --bed"
  cactus_convertAlignmentsToInternalNames[0x420d91]
  cactus_convertAlignmentsToInternalNames[0x420cc4]
  cactus_convertAlignmentsToInternalNames[0x41daf9]
  cactus_convertAlignmentsToInternalNames[0x40711d]
  /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f804a911555]
  cactus_convertAlignmentsToInternalNames[0x407467]
  ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp70058_0: No such file or directory
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
      super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
      return self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
      runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
      parameters=["cactus_convertAlignmentsToInternalNames"] + args)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
      raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
  RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy to 1
  WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy to 2147483648 bytes
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,045 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with job batch system ID: 255 and cores: 1, disk: 2.0 G, and memory: 2.0 G
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,123 MainThread INFO toil-rt: 2020-09-07 10:41:25.123344: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
          <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
      </st_kv_database_conf>
   --bed"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,288 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,296 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,297 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-yyklg10i follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
  INFO:toil-rt:2020-09-07 10:41:25.123344: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
              <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
          </st_kv_database_conf>
       --bed"
  cactus_convertAlignmentsToInternalNames[0x420d91]
  cactus_convertAlignmentsToInternalNames[0x420cc4]
  cactus_convertAlignmentsToInternalNames[0x41daf9]
  cactus_convertAlignmentsToInternalNames[0x40711d]
  /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa518494555]
  cactus_convertAlignmentsToInternalNames[0x407467]
  ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp71684_0: No such file or directory
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
      super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
      return self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
      runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
      parameters=["cactus_convertAlignmentsToInternalNames"] + args)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
      raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
  RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpk7bbi0wa.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/a0b188d4-888b-4ade-9036-5776acc41547/tmpawb3v6_o.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i to 1
  WARNING:toil.jobGraph:We have increased the default memory of the failed job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i to 2147483648 bytes
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:26,299 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with job batch system ID: 256 and cores: 1, disk: 2.0 G, and memory: 2.0 G
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/worker_log.txt
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:29,328 MainThread INFO toil-rt: 2020-09-07 10:41:29.328871: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
          <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
      </st_kv_database_conf>
   --bed"
INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/worker_log.txt
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:34,329 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:37,268 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:37,404 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-f4z694yy follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
  INFO:toil-rt:2020-09-07 10:41:29.328871: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
              <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
          </st_kv_database_conf>
       --bed"
  cactus_convertAlignmentsToInternalNames[0x420d91]
  cactus_convertAlignmentsToInternalNames[0x420cc4]
  cactus_convertAlignmentsToInternalNames[0x41daf9]
  cactus_convertAlignmentsToInternalNames[0x40711d]
  /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fcc032cf555]
  cactus_convertAlignmentsToInternalNames[0x407467]
  ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp72449_0: No such file or directory
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
      super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
      return self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
      runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
      parameters=["cactus_convertAlignmentsToInternalNames"] + args)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
      raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
  RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy to 0
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:37,844 MainThread WARNING toil.leader: Job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,071 MainThread INFO toil-rt: 2020-09-07 10:41:40.071790: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
          <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
      </st_kv_database_conf>
   --bed"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,327 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,332 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,332 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-yyklg10i follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
  INFO:toil-rt:2020-09-07 10:41:40.071790: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
              <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
          </st_kv_database_conf>
       --bed"
  cactus_convertAlignmentsToInternalNames[0x420d91]
  cactus_convertAlignmentsToInternalNames[0x420cc4]
  cactus_convertAlignmentsToInternalNames[0x41daf9]
  cactus_convertAlignmentsToInternalNames[0x40711d]
  /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7ddc198555]
  cactus_convertAlignmentsToInternalNames[0x407467]
  ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp74487_0: No such file or directory
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
      super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
      return self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
      runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
      parameters=["cactus_convertAlignmentsToInternalNames"] + args)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
      raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
  RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i to 0
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:40,333 MainThread WARNING toil.leader: Job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:47,916 MainThread INFO toil-rt: 2020-09-07 10:41:47.916196: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:49,428 MainThread INFO toil-rt: 2020-09-07 10:41:49.428707: Running the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:49,444 MainThread INFO toil-rt: 2020-09-07 10:41:49.444236: Successfully ran the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1" in 0.0076715946197509766 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:50,018 MainThread INFO toil-rt: 2020-09-07 10:41:50.018908: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:51,826 MainThread INFO toil-rt: 2020-09-07 10:41:51.826316: Running the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:51,838 MainThread INFO toil-rt: 2020-09-07 10:41:51.838081: Successfully ran the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1" in 0.0073244571685791016 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:48,012 MainThread INFO toil-rt: 2020-09-07 10:42:48.012228: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:48,575 MainThread INFO toil-rt: 2020-09-07 10:42:48.575290: Successfully ran the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE" in 0.009398221969604492 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,730 MainThread INFO toil.leader: Job ended: 'KtServerService' kind-KtServerService/instance-hnakmgk1
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,735 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'KtServerService' kind-KtServerService/instance-hnakmgk1
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,736 MainThread WARNING toil.leader: Log from job kind-KtServerService/instance-hnakmgk1 follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  INFO:toil-rt:2020-09-07 10:40:46.804704: Running the command: "netstat -tuplen"
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  INFO:toil-rt:2020-09-07 10:40:46.823367: Successfully ran the command: "netstat -tuplen" in 0.012812137603759766 seconds
  INFO:cactus.shared.common:Running the command ['ktserver', '-port', '29555', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  INFO:toil-rt:2020-09-07 10:40:46.835407: Running the command: "ktserver -port 29555 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:40:47.845364: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:41:47.916196: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE', '1']
  INFO:toil-rt:2020-09-07 10:41:49.428707: Running the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1"
  INFO:toil-rt:2020-09-07 10:41:49.444236: Successfully ran the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1" in 0.0076715946197509766 seconds
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:42:48.012228: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
  1
  INFO:toil-rt:2020-09-07 10:42:48.575290: Successfully ran the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE" in 0.009398221969604492 seconds
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
      returnValues = self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
      raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-hnakmgk1 with ID kind-KtServerService/instance-hnakmgk1 to 1
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:49,738 MainThread WARNING toil.leader: Job 'KtServerService' kind-KtServerService/instance-hnakmgk1 with ID kind-KtServerService/instance-hnakmgk1 is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:50,089 MainThread INFO toil-rt: 2020-09-07 10:42:50.089057: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:50,101 MainThread INFO toil-rt: 2020-09-07 10:42:50.101638: Successfully ran the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE" in 0.005008220672607422 seconds
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,108 MainThread INFO toil.leader: Job ended: 'KtServerService' kind-KtServerService/instance-8pd4r5v0
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,112 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'KtServerService' kind-KtServerService/instance-8pd4r5v0
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,113 MainThread WARNING toil.leader: Log from job kind-KtServerService/instance-8pd4r5v0 follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  INFO:toil-rt:2020-09-07 10:40:48.905109: Running the command: "netstat -tuplen"
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  INFO:toil-rt:2020-09-07 10:40:48.927017: Successfully ran the command: "netstat -tuplen" in 0.016075611114501953 seconds
  INFO:cactus.shared.common:Running the command ['ktserver', '-port', '5445', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  INFO:toil-rt:2020-09-07 10:40:48.941000: Running the command: "ktserver -port 5445 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:40:49.949341: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:41:50.018908: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE', '1']
  INFO:toil-rt:2020-09-07 10:41:51.826316: Running the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1"
  INFO:toil-rt:2020-09-07 10:41:51.838081: Successfully ran the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1" in 0.0073244571685791016 seconds
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:42:50.089057: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
  1
  INFO:toil-rt:2020-09-07 10:42:50.101638: Successfully ran the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE" in 0.005008220672607422 seconds
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
      returnValues = self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
      raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 with ID kind-KtServerService/instance-8pd4r5v0 to 1
<=========
bp1-login01.data.bp.acrc.priv 2020-09-07 10:42:52,118 MainThread WARNING toil.leader: Job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 with ID kind-KtServerService/instance-8pd4r5v0 is completely failed
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:30,883 MainThread INFO toil.leader: Finished toil run with 22 failed jobs.
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:30,884 MainThread INFO toil.leader: Failed jobs at end of the run: 'KtServerService' kind-KtServerService/instance-hnakmgk1 'StartPrimaryDB' kind-StartPrimaryDB/instance-_oa_qio2 'ProgressiveUp' kind-ProgressiveUp/instance-lokvicq8 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho 'StartPrimaryDB' kind-StartPrimaryDB/instance-4djpc7nx 'ProgressiveDown' kind-ProgressiveDown/instance-li8hv0fj 'ProgressiveNext' kind-ProgressiveNext/instance-a7irsvu6 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-cfldch16 'ProgressiveDown' kind-ProgressiveDown/instance-8kgdnuhg 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-c158ubud 'ProgressiveDown' kind-ProgressiveDown/instance-ycp1al2l 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-i3r410fe 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-8x084ped 'KtServerService' kind-KtServerService/instance-8pd4r5v0 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-abhxdgwj 'ProgressiveNext' kind-ProgressiveNext/instance-pju96b5_ 'ProgressiveDown' kind-ProgressiveDown/instance-hnb46cgh 'ProgressiveUp' kind-ProgressiveUp/instance-_p3wau6d 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-duyjfuj1
Workflow Progress 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 257/257 (0 failures) [06:08<00:00, 0.70 jobs/s]
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:30,899 MainThread INFO toil.realtimeLogger: Stopping real-time logging server.
bp1-login01.data.bp.acrc.priv 2020-09-07 10:43:31,154 MainThread INFO toil.realtimeLogger: Joining real-time logging server thread.
Traceback (most recent call last):
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/progressive/cactus_progressive.py", line 406, in main
    runCactusProgressive(options)
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/progressive/cactus_progressive.py", line 454, in runCactusProgressive
    halID = toil.start(RunCactusPreprocessorThenProgressiveDown(options, project, memory=configWrapper.getDefaultMemory()))
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/common.py", line 829, in start
    return self._runMainLoop(rootJobGraph)
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/common.py", line 1120, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/leader.py", line 269, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/work/tk19812/software/cactus-bin-v1.2.1/jobstore' contains 22 failed jobs: 'KtServerService' kind-KtServerService/instance-hnakmgk1, 'StartPrimaryDB' kind-StartPrimaryDB/instance-_oa_qio2, 'ProgressiveUp' kind-ProgressiveUp/instance-lokvicq8, 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho, 'StartPrimaryDB' kind-StartPrimaryDB/instance-4djpc7nx, 'ProgressiveDown' kind-ProgressiveDown/instance-li8hv0fj, 'ProgressiveNext' kind-ProgressiveNext/instance-a7irsvu6, 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-cfldch16, 'ProgressiveDown' kind-ProgressiveDown/instance-8kgdnuhg, 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-c158ubud, 'ProgressiveDown' kind-ProgressiveDown/instance-ycp1al2l, 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i, 'RunCactusPreprocessorThenProgressiveDown' kind-RunCactusPreprocessorThenProgressiveDown/instance-i3r410fe, 'RunCactusPreprocessorThenProgressiveDown2' kind-RunCactusPreprocessorThenProgressiveDown2/instance-8x084ped, 'KtServerService' kind-KtServerService/instance-8pd4r5v0, 'CactusSetupCheckpoint' kind-CactusSetupCheckpoint/instance-abhxdgwj, 'ProgressiveNext' kind-ProgressiveNext/instance-pju96b5_, 'ProgressiveDown' kind-ProgressiveDown/instance-hnb46cgh, 'ProgressiveUp' kind-ProgressiveUp/instance-_p3wau6d, 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy, 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1, 'CactusTrimmingBlastPhase' kind-CactusTrimmingBlastPhase/instance-duyjfuj1
Log from job 'KtServerService' kind-KtServerService/instance-hnakmgk1 follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  INFO:toil-rt:2020-09-07 10:40:46.804704: Running the command: "netstat -tuplen"
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  INFO:toil-rt:2020-09-07 10:40:46.823367: Successfully ran the command: "netstat -tuplen" in 0.012812137603759766 seconds
  INFO:cactus.shared.common:Running the command ['ktserver', '-port', '29555', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  INFO:toil-rt:2020-09-07 10:40:46.835407: Running the command: "ktserver -port 29555 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tk_dy799y/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp69z8ymvv/7cddb6fb-b0cd-4c2f-9c51-e35c301f7648/tmp5oipm3w4.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:40:47.845364: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:41:47.916196: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE', '1']
  INFO:toil-rt:2020-09-07 10:41:49.428707: Running the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1"
  INFO:toil-rt:2020-09-07 10:41:49.444236: Successfully ran the command: "ktremotemgr set -port 29555 -host 172.25.9.28 TERMINATE 1" in 0.0076715946197509766 seconds
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '29555', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:42:48.012228: Running the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE"
  1
  INFO:toil-rt:2020-09-07 10:42:48.575290: Successfully ran the command: "ktremotemgr get -port 29555 -host 172.25.9.28 TERMINATE" in 0.009398221969604492 seconds
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
      returnValues = self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
      raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-hnakmgk1 with ID kind-KtServerService/instance-hnakmgk1 to 1
<=========
Log from job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
  INFO:toil-rt:2020-09-07 10:41:40.071790: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
              <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />
          </st_kv_database_conf>
       --bed"
  cactus_convertAlignmentsToInternalNames[0x420d91]
  cactus_convertAlignmentsToInternalNames[0x420cc4]
  cactus_convertAlignmentsToInternalNames[0x41daf9]
  cactus_convertAlignmentsToInternalNames[0x40711d]
  /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7ddc198555]
  cactus_convertAlignmentsToInternalNames[0x407467]
  ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp74487_0: No such file or directory
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
      super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
      return self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
      runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
      parameters=["cactus_convertAlignmentsToInternalNames"] + args)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
      raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
  RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpr_gjao7w.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp__r5a8l7/ab9ec3ed-3daa-4898-b535-c668ac9fbf90/tmpobd6wopd.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="5445" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with ID kind-CactusCafPhase/instance-yyklg10i to 0
<=========
Log from job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['netstat', '-tuplen']
  INFO:toil-rt:2020-09-07 10:40:48.905109: Running the command: "netstat -tuplen"
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  INFO:toil-rt:2020-09-07 10:40:48.927017: Successfully ran the command: "netstat -tuplen" in 0.016075611114501953 seconds
  INFO:cactus.shared.common:Running the command ['ktserver', '-port', '5445', '-ls', '-tout', '200000', '-th', '64', '-bgs', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot', '-bgsc', 'lzo', '-bgsi', '1000000', '-log', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp', ':#opts=ls#bnum=30m#msiz=50g#ktopts=p']
  INFO:toil-rt:2020-09-07 10:40:48.941000: Running the command: "ktserver -port 5445 -ls -tout 200000 -th 64 -bgs /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tnkuw5b6x/snapshot -bgsc lzo -bgsi 1000000 -log /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp6vi0iu3v/d0a3b9bf-385a-4b37-b144-3f1e85fba49e/tmp3ov4_3i1.tmp :#opts=ls#bnum=30m#msiz=50g#ktopts=p"
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:toil.lib.bioio:Ktserver running.
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:40:49.949341: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:41:50.018908: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'set', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE', '1']
  INFO:toil-rt:2020-09-07 10:41:51.826316: Running the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1"
  INFO:toil-rt:2020-09-07 10:41:51.838081: Successfully ran the command: "ktremotemgr set -port 5445 -host 172.25.9.28 TERMINATE 1" in 0.0073244571685791016 seconds
  INFO:cactus.shared.common:Running the command ['ktremotemgr', 'get', '-port', '5445', '-host', '172.25.9.28', 'TERMINATE']
  INFO:toil-rt:2020-09-07 10:42:50.089057: Running the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE"
  1
  INFO:toil-rt:2020-09-07 10:42:50.101638: Successfully ran the command: "ktremotemgr get -port 5445 -host 172.25.9.28 TERMINATE" in 0.005008220672607422 seconds
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1748, in _run
      returnValues = self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1722, in run
      raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
  RuntimeError: Detected the error jobStoreID has been removed so exiting with an error
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'KtServerService' kind-KtServerService/instance-8pd4r5v0 with ID kind-KtServerService/instance-8pd4r5v0 to 1
<=========
Log from job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy follows:
=========>
  INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
  INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe.
  INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed']
  INFO:toil-rt:2020-09-07 10:41:29.328871: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp --cactusDisk <st_kv_database_conf type="kyoto_tycoon">
              <kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />
          </st_kv_database_conf>
       --bed"
  cactus_convertAlignmentsToInternalNames[0x420d91]
  cactus_convertAlignmentsToInternalNames[0x420cc4]
  cactus_convertAlignmentsToInternalNames[0x41daf9]
  cactus_convertAlignmentsToInternalNames[0x40711d]
  /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fcc032cf555]
  cactus_convertAlignmentsToInternalNames[0x407467]
  ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp72449_0: No such file or directory
  Traceback (most recent call last):
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
      job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
      super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
      returnValues = self._run(jobGraph, fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
      return self.run(fileStore)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
      runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
      parameters=["cactus_convertAlignmentsToInternalNames"] + args)
    File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1280, in cactus_call
      raise RuntimeError("Command {} exited {}: {}".format(call, process.returncode, out))
  RuntimeError: Command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpowi02szy.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpf7plu84p/18c0539a-04e5-4b5b-b698-d41e05c155a6/tmpvhj_mrrc.tmp', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="172.25.9.28" port="29555" />\n\t\t</st_kv_database_conf>\n\t', '--bed'] exited 1: stdout=None
  ERROR:toil.worker:Exiting the worker because of a failed job on host bp1-login01.data.bp.acrc.priv
  WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with ID kind-CactusCafPhase/instance-f4z694yy to 0
<=========

Cheers F

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/ComparativeGenomicsToolkit/cactus/issues/310#issuecomment-688210071I'm sorry to keep bothering you, Mark. I'm trying to test CACTUS on another cluster, which has more memory although the walltime is way shorter. Unfortunately, there's something wrong with cactus, I got this aligning the test example:

bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,470 MainThread INFO toil.leader: Job ended: 'CactusSetupPhase' kind-CactusSetupPhase/instance-olnax8p1 bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,473 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy with job batch system ID: 253 and cores: 1, disk: 2.0 G, and memory: 500.0 M bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,538 MainThread INFO toil.leader: Job ended: 'CactusSetupPhase' kind-CactusSetupPhase/instance-gk07lqho bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:15,542 MainThread INFO toil.leader: Issued job 'CactusCafPhase' kind-CactusCafPhase/instance-yyklg10i with job batch system ID: 254 and cores: 1, disk: 2.0 G, and memory: 500.0 M INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/worker_log.txt INFO:toil.worker:Redirecting logging to /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmpzimt8x_m/worker_log.txt bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:16,469 MainThread INFO toil-rt: 2020-09-07 10:41:16.469003: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp --cactusDisk 

            </st_kv_database_conf>
     --bed"

bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:20,838 MainThread INFO toil.leader: Job ended: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:24,712 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CactusCafPhase' kind-CactusCafPhase/instance-f4z694yy bp1-login01.data.bp.acrc.priv 2020-09-07 10:41:25,016 MainThread WARNING toil.leader: Log from job kind-CactusCafPhase/instance-f4z694yy follows: =========> INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- INFO:toil:Running Toil version 4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe. INFO:cactus.shared.common:Running the command ['cactus_convertAlignmentsToInternalNames', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp', '/tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp', '--cactusDisk', '\n\t\t\t\n\t\t\n\t', '--bed'] INFO:toil-rt:2020-09-07 10:41:16.469003: Running the command: "cactus_convertAlignmentsToInternalNames /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpsy99ihd0.tmp /tmp/node-7d61eb05-2720-43e2-bb0a-84e08354173b-81657330c30e4f38a6b8e3469f401cce/tmp1_0vvot0/bb893aea-053b-46ae-ac55-91e34330199e/tmpml60qicl.tmp --cactusDisk 

                    </st_kv_database_conf>
             --bed"
    cactus_convertAlignmentsToInternalNames[0x420d91]
    cactus_convertAlignmentsToInternalNames[0x420cc4]
    cactus_convertAlignmentsToInternalNames[0x41daf9]
    cactus_convertAlignmentsToInternalNames[0x40711d]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f804a911555]
    cactus_convertAlignmentsToInternalNames[0x407467]
    ERROR: Couldn't create temporary file's file descriptor for /work/tk19812/HeliconiniiProject/HeliconGenomeAlignmentAnnotation/Test.ANN.pipeline.CAT.workingDir.Test/stTmp70058_0: No such file or directory
    Traceback (most recent call last):
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/worker.py", line 366, in workerScript
        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 1343, in _runner
        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1392, in _runner
        returnValues = self._run(jobGraph, fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/toil/job.py", line 1329, in _run
        return self.run(fileStore)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/pipeline/cactus_workflow.py", line 692, in run
        runConvertAlignmentsToInternalNames(self.cactusWorkflowArguments.cactusDiskDatabaseString, tempFile, ingroupCoverageFile, self.topFlowerName, isBedFile=True)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py", line 250, in runConvertAlignmentsToInternalNames
        parameters=["cactus_convertAlignmentsToInternalNames"] + args)
      File "/work/tk19812/software/cactus-bin-v1.2.1/venv/lib/python3.7/site-packages/cactus/shared/common.py"
LemoAlex commented 3 years ago

Hello,

Was this issue ever resolved ?

I have the exact same problem with my work and I am trying to find why the temporary file cannot be found. I prepared the command lines with cactus-prepare, and up until this point (Round 0 : cactus-align), everything went smoothly so I don't understand why a fille/directory is not found here.

Cheers,

Alexandre