search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
507 stars 112 forks source link

KTServer and CactusHalGeneratorUpWrapper RuntimeError on AWS #68

Open xieduo7 opened 5 years ago

xieduo7 commented 5 years ago

Hello, I try to run the primate genome on the AWS using such command:

TOIL_AWS_AMI=ami-aff944d5 nohup cactus --configFile primate_progressive_config.xml --stats --nodeTypes c4.8xlarge:0.6,r4.8xlarge --minNodes 0,0 --maxNodes 70,4 --provisioner aws --batchSystem mesos --metrics aws:us-east-1:primate-jobstore-9 seqFile_ensembl.txt s3://primate-out90320.log >cactus.log 2>&1 &

And the job report error when it run the KtServerService:

Job ended successfully: 'KtServerService' f24176c7-82dc-48d7-a98a-fa0021a97d73
The job seems to have left a log file, indicating failure: 'KtServerService' f24176c7-82dc-48d7-a98a-fa0021a97d73
f24176c7-82dc-48d7-a98a-fa0021a97d73    ', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'be601255-a0d4-47a5-8d9c-a75a47e9c8f8', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '9add65a0-5837-43fc-b194-7e501efee82e', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '59f35a9a-cf7a-4464-95de-57838464ea99', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '10290e8d-6ddd-4697-a75a-6933c02b7e54', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '044967e5-1fc9-4a35-b7c2-75b3820c07f3', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '52838eb6-bdbd-41e6-9ca3-389343cd8de8', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a37fec1b-dd19-4991-92b4-ce1536e8ac82', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'c9b68343-075d-4052-9946-f9b48d522edb', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a7957346-b7d1-456f-8f65-43f25026d37e', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '3d3bf469-431b-430a-9b10-e319a10dfb40', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'c251df05-0a86-42ed-b3a8-fc86ba644648', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '7f5262a3-b2fd-4ac9-87ed-90eab5caca24', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '88bdaf74-8459-4b3e-b43e-6f24ebf5b6da', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '6518d355-8c8a-42be-9fb4-540c85da9054', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '34ba7dd4-0a3f-4802-9993-3238b78ac350', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '49c106e2-8360-4348-9d90-139b105933d8', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '4fc7e9c0-a257-4703-a4d3-0a593d83a4a3', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '0dd23714-d674-411c-9e76-c798e494f1eb', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a4962e4d-13e5-4bea-871d-f0f3e244f277', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'aa9126cf-1740-4cb7-be9a-8e3cec6f0a87', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '81a5dd62-8583-41a5-90e2-389f1d3a6a69', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '47b504ed-a516-4a40-a94d-0ce3df6914fa', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'c537957c-84a3-4c2f-b1e6-3aadd62e8a07', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'd98e0845-e8bc-4424-962a-5dac3214f70d', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '4993f72f-42c6-4772-b75f-0e6e63abfcf4', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '48ecdb6e-fb8c-4e6a-ba98-4e716b86548d', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'd4713489-153a-489b-be98-bf13170731f4', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '10bac09a-9f2a-44f3-a248-76e713ffeef5', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '6f110ecb-910e-456e-a0a5-789c2aaf73ed', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'b8d179e3-a7d6-40d1-81b3-31b570fba66d', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'b60ec31b-2bd3-4673-a2ab-41cf2a108541', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'c909a001-8b1d-4e46-8532-a559aa7b2477', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '39d5a530-b197-4dac-87fd-f8c13c10fa3d', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '0e5b35f8-9492-49b5-8567-3223db29ee75', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'fb35e4ec-941e-4fc0-a6ae-c529b682ca81', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'bdfe13a7-ad98-4515-aa3a-c3ca330ce809', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '638fe428-b594-46b3-8d56-c75c8231fdb0', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '7991546d-e53a-4241-890f-0690062d7406', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '513f5131-4240-480d-8fd0-e9d697faf849', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'dc17fd1d-d6d0-4c9c-82ae-789992a63cf1', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'cd982872-2d1b-4aa4-bfe7-d3d41278881e', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '8672bd62-bd3b-4e7e-8ae0-57b90d6a27a1', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'df8527e1-71c9-4849-b648-3deff3201ddd', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '6b3acda3-73a7-4df8-a438-08caa44df400', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'cab6a370-b8a4-4b1f-b306-4364cf26ceef', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '553e1af7-d434-464c-bb6b-a6e30a63a947', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'fbf354e9-70ab-401e-a44f-254984086d2a', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a80968a6-0eb1-43c7-ae6c-0d6e2cc2ea9a', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'e046e74c-521b-4fd7-b1dc-37e4f41bdb99', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a1b5e9c9-ab59-46fa-ac23-33dcab76e69f', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'fd552886-7aae-4dc5-9ccb-74b11385cd17', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'aec241fd-ab97-4e24-8caa-7420eb4c7897', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '9d9e4125-3c8d-4f4a-97c0-99011239ea15', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'b03562d4-9f7c-4e7c-8e22-b521146fa5e0', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '50405ba3-a0d0-4c0b-b7db-1bb3d9a59704', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '881665aa-e3c7-4733-850d-ca9a62b2d475', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'ad92dd32-fefe-4368-a0aa-4b3ca6d2e762', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a7454fa7-3013-452e-8d7e-21f276c4f025', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'd407fa30-a3c2-4205-9f50-4877acf46132', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'eff1b6b5-4a0a-42af-a88c-4ce908a6cde2', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '479b7b2e-ff03-4350-b6ae-aea85eb70099', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'a5bbe586-f63a-4ef2-89f4-e8fa669ed055', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '2d6a2e22-52bd-408f-a899-9313af11dbed', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '392043f1-75b5-4580-bcaa-bc5c69d43971', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '2b0ba8e3-bf4d-4a5f-b589-a35963a30df3', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '6e529526-e6eb-4222-a60b-24c353edf21b', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'e51e0aa9-484b-4aee-9ee6-46dbaed2c514', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '712317a2-bcc2-48e0-ad2e-23ebf432e907', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '76455c33-ddf7-4293-8f08-9bdecf05c9fe', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '8cc5bef0-0e7e-42ee-be5d-a010b7d40903', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'd46190a0-a60f-4d39-a105-2db770bdf630', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '5a1e0093-85ea-4d21-8797-cfa218bab4f5', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '4b6f83f9-3a40-4af0-9c78-034b150c3e55', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'c794ea8b-b681-4be1-95ef-8f464addd6b1', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'f714b1a9-0f40-40bd-bb49-f005c45510e2', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'cfc9ea2e-4f3f-4fcb-8815-d03db7c0f943', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '4a23e7e9-5f41-49f6-9c72-cd1b3161f3fb', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'eb39b7f6-9a37-4ff1-ad6a-6251f2a2d329', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '990f3d01-6f3c-4035-898a-5ad2f3e9b031', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'd63c29b2-da77-42a2-8525-8b41c58c884c', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '12402cc4-37e8-4fd9-9151-f64875117acb', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'f6a2631c-4a1c-478a-8651-5c5587d652eb', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '6beea2e8-66bc-4d9e-b0b3-87b5bf900ed8', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'daca9407-a167-49f1-bdec-74466d1d55ce', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '0eb6130b-dd65-4de1-97e6-54e4ca5ab303', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'b8ff05a8-b886-4103-942a-63f3bccc09e0', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '691568a7-417a-46ee-adcf-a240216c8677', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '9eda72a2-335c-4217-859f-ce6dad3a7601', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'e927ac45-e3c1-43cf-a091-39a9e2ba9995', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'ecf698a3-ea12-46e8-a91d-bcdb658d76cf', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '4a8e0db3-8763-4749-a7aa-4a7aa0416833', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'dc9a55a1-27cf-44ad-9d2b-0ed5069d7ce7', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'aad14033-3917-407f-b281-a3cea870b7f3', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'e7059161-149a-4094-bca8-dfdfa253531c', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '595241e6-780f-4647-9f47-071683dba8c7', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'faec8a90-a10c-41ae-ba8a-4879358446c6', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '8da89661-c029-47f0-adfd-e8dc05d4ac29', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '9101692f-9c3a-4227-9e07-62d63e43027d', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'baf740aa-d026-4fda-aaf0-88cce5f21acf', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'ba4da18e-395e-4454-9dc4-8cf552e5565a', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '22d67e08-6815-432d-8d37-9e8e40305ad4', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '0416a2ab-6d79-47d0-92ac-8caf12e91cc9', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '69c983f4-3468-4230-8cb5-caac7c9d2f76', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', 'd7fe84a0-6147-47af-8abc-f8db3a1cc6c7', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'set', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE', '1']
f24176c7-82dc-48d7-a98a-fa0021a97d73    Running command catchsegv 'ktremotemgr' 'set' '-port' '6092' '-host' '10.0.130.252' 'TERMINATE' '1'
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Work dirs: set([])
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Docker work dir: .
f24176c7-82dc-48d7-a98a-fa0021a97d73    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-d5bc9059-f08a-42b3-89db-76b9f28471f1/tmp_wcTZ9/68e3dff1-72f6-4ab4-8b7e-7fde0eba4c50:/data', '--name', '2c25c323-5f94-47ef-af37-c8723d037eaf', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'ktremotemgr', 'get', '-port', '6092', '-host', '10.0.130.252', 'TERMINATE']
f24176c7-82dc-48d7-a98a-fa0021a97d73    1
f24176c7-82dc-48d7-a98a-fa0021a97d73    [1]+  Interrupt               eval "${options}" 0<&0
f24176c7-82dc-48d7-a98a-fa0021a97d73    WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 1bd5b2e1-8e1e-4a44-9342-7403fc24c07f used 1130.61% (22.6 GB [24279564288B] used, 2.0 GB [2147483648B] requested) at the end of its run.
f24176c7-82dc-48d7-a98a-fa0021a97d73    Traceback (most recent call last):
f24176c7-82dc-48d7-a98a-fa0021a97d73      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 314, in workerScript
f24176c7-82dc-48d7-a98a-fa0021a97d73        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
f24176c7-82dc-48d7-a98a-fa0021a97d73      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1351, in _runner
f24176c7-82dc-48d7-a98a-fa0021a97d73        returnValues = self._run(jobGraph, fileStore)
f24176c7-82dc-48d7-a98a-fa0021a97d73      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1694, in _run
f24176c7-82dc-48d7-a98a-fa0021a97d73        returnValues = self.run(fileStore)
f24176c7-82dc-48d7-a98a-fa0021a97d73      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1668, in run
f24176c7-82dc-48d7-a98a-fa0021a97d73        raise RuntimeError("Detected the error jobStoreID has been removed so exiting with an error")
f24176c7-82dc-48d7-a98a-fa0021a97d73    RuntimeError: Detected the error jobStoreID has been removed so exiting with an error

and The error of CactusHalGeneratorUpWrapper

Issued job 'CactusHalGeneratorUpWrapper' eb0aee2b-4107-40db-8bca-cde3ed0eda59 with job batch system ID: 1248381 and cores: 1, disk: 2.0 G, and memory: 3.8 G
Job ended successfully: 'CactusHalGeneratorUpWrapper' eb0aee2b-4107-40db-8bca-cde3ed0eda59
The job seems to have left a log file, indicating failure: 'CactusHalGeneratorUpWrapper' eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
eb0aee2b-4107-40db-8bca-cde3ed0eda59    INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be.
eb0aee2b-4107-40db-8bca-cde3ed0eda59    INFO:cactus.shared.common:Work dirs: set([])
eb0aee2b-4107-40db-8bca-cde3ed0eda59    INFO:cactus.shared.common:Docker work dir: .
eb0aee2b-4107-40db-8bca-cde3ed0eda59    INFO:cactus.shared.common:Running the command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-109549f5-fea1-46ce-b47e-f14588626e09/tmp_jeExw/37faddac-9006-45a1-8400-35f4258ef61c:/data', '--name', '2e1bf0f3-a247-4896-bf05-885d013f97ca', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'cactus_halGenerator', '--logLevel', 'INFO', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="10.0.130.252" port="6092" />\n\t\t</st_kv_database_conf>\n\t', '--secondaryDisk', '<st_kv_database_conf type="kyoto_tycoon">\n   <kyoto_tycoon database_dir="fakepath" host="10.0.130.252" port="30916" />\n  </st_kv_database_conf>\n ', '--referenceEventString', 'Anc2']
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Running command catchsegv 'cactus_halGenerator' '--logLevel' 'INFO' '--cactusDisk' '<st_kv_database_conf type="kyoto_tycoon">
eb0aee2b-4107-40db-8bca-cde3ed0eda59                            <kyoto_tycoon database_dir="fakepath" host="10.0.130.252" port="6092" />
eb0aee2b-4107-40db-8bca-cde3ed0eda59                    </st_kv_database_conf>
eb0aee2b-4107-40db-8bca-cde3ed0eda59            ' '--secondaryDisk' '<st_kv_database_conf type="kyoto_tycoon">
eb0aee2b-4107-40db-8bca-cde3ed0eda59       <kyoto_tycoon database_dir="fakepath" host="10.0.130.252" port="30916" />
eb0aee2b-4107-40db-8bca-cde3ed0eda59      </st_kv_database_conf>
eb0aee2b-4107-40db-8bca-cde3ed0eda59     ' '--referenceEventString' 'Anc2'
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Set log level to INFO
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Set up the flower disk
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Set up the secondary database
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Segmentation fault (core dumped)
eb0aee2b-4107-40db-8bca-cde3ed0eda59    *** Segmentation fault
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Register dump:
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59     RAX: 00000000022197e0   RBX: 000000000221b4e0   RCX: 000000000221b300
eb0aee2b-4107-40db-8bca-cde3ed0eda59     RDX: 0000000000000000   RSI: 00007f1afded2b28   RDI: 0000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     RBP: 0000000000000000   R8 : 00000000022199f0   R9 : 0000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     R10: 0000000000000001   R11: 0000000000000000   R12: 00000000022197e0
eb0aee2b-4107-40db-8bca-cde3ed0eda59     R13: 0000000000000003   R14: 0000000000404630   R15: 000000000221b4e0
eb0aee2b-4107-40db-8bca-cde3ed0eda59     RSP: 00007ffe6ebc1a98
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59     RIP: 00000000004159e0   EFLAGS: 00010202
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59     CS: 0033   FS: 0000   GS: 0000
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59     Trap: 0000000e   Error: 00000004   OldMask: 00001000   CR2: 00000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59     FPUCW: 0000037f   FPUSW: 00000000   TAG: 00000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     RIP: 00000000   RDP: 00000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59     ST(0) 0000 0000000000000000   ST(1) 0000 0000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     ST(2) 0000 0000000000000000   ST(3) 0000 0000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     ST(4) 0000 0000000000000000   ST(5) 0000 0000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     ST(6) 0000 0000000000000000   ST(7) 0000 0000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     mxcsr: 1fa0
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM0:  00000000000000000000000000000000 XMM1:  00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM2:  00000000000000000000000000000000 XMM3:  00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM4:  00000000000000000000000000000000 XMM5:  00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM6:  00000000000000000000000000000000 XMM7:  00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM8:  00000000000000000000000000000000 XMM9:  00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59     XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Backtrace:
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x4159e0]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x40466a]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x404fdd]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x405173]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x40498b]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x404127]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f1afdb2e830]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    cactus_halGenerator[0x4041d9]
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Memory map:
eb0aee2b-4107-40db-8bca-cde3ed0eda59
eb0aee2b-4107-40db-8bca-cde3ed0eda59    00400000-00464000 r-xp 00000000 ca:09 1062417 /usr/local/bin/cactus_halGenerator
eb0aee2b-4107-40db-8bca-cde3ed0eda59    00663000-00664000 r--p 00063000 ca:09 1062417 /usr/local/bin/cactus_halGenerator
eb0aee2b-4107-40db-8bca-cde3ed0eda59    00664000-00665000 rw-p 00064000 ca:09 1062417 /usr/local/bin/cactus_halGenerator
eb0aee2b-4107-40db-8bca-cde3ed0eda59    00665000-00696000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    021fe000-02230000 rw-p 00000000 00:00 0 [heap]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd2b6000-7f1afd2d7000 r-xp 00000000 ca:09 1580691 /lib/x86_64-linux-gnu/liblzo2.so.2.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd2d7000-7f1afd4d6000 ---p 00021000 ca:09 1580691 /lib/x86_64-linux-gnu/liblzo2.so.2.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd4d6000-7f1afd4d7000 r--p 00020000 ca:09 1580691 /lib/x86_64-linux-gnu/liblzo2.so.2.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd4d7000-7f1afd4d8000 rw-p 00021000 ca:09 1580691 /lib/x86_64-linux-gnu/liblzo2.so.2.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd4d8000-7f1afd4f9000 r-xp 00000000 ca:09 1575436 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd4f9000-7f1afd6f8000 ---p 00021000 ca:09 1575436 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd6f8000-7f1afd6f9000 r--p 00020000 ca:09 1575436 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd6f9000-7f1afd6fa000 rw-p 00021000 ca:09 1575436 /lib/x86_64-linux-gnu/liblzma.so.5.0.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd6fa000-7f1afd6fd000 r-xp 00000000 ca:09 1575418 /lib/x86_64-linux-gnu/libdl-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd6fd000-7f1afd8fc000 ---p 00003000 ca:09 1575418 /lib/x86_64-linux-gnu/libdl-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd8fc000-7f1afd8fd000 r--p 00002000 ca:09 1575418 /lib/x86_64-linux-gnu/libdl-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd8fd000-7f1afd8fe000 rw-p 00003000 ca:09 1575418 /lib/x86_64-linux-gnu/libdl-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd8fe000-7f1afd90d000 r-xp 00000000 ca:09 1575404 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afd90d000-7f1afdb0c000 ---p 0000f000 ca:09 1575404 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afdb0c000-7f1afdb0d000 r--p 0000e000 ca:09 1575404 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afdb0d000-7f1afdb0e000 rw-p 0000f000 ca:09 1575404 /lib/x86_64-linux-gnu/libbz2.so.1.0.4
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afdb0e000-7f1afdcce000 r-xp 00000000 ca:09 1575405 /lib/x86_64-linux-gnu/libc-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afdcce000-7f1afdece000 ---p 001c0000 ca:09 1575405 /lib/x86_64-linux-gnu/libc-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afdece000-7f1afded2000 r--p 001c0000 ca:09 1575405 /lib/x86_64-linux-gnu/libc-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afded2000-7f1afded4000 rw-p 001c4000 ca:09 1575405 /lib/x86_64-linux-gnu/libc-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afded4000-7f1afded8000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afded8000-7f1afdeee000 r-xp 00000000 ca:09 1575426 /lib/x86_64-linux-gnu/libgcc_s.so.1
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afdeee000-7f1afe0ed000 ---p 00016000 ca:09 1575426 /lib/x86_64-linux-gnu/libgcc_s.so.1
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe0ed000-7f1afe0ee000 rw-p 00015000 ca:09 1575426 /lib/x86_64-linux-gnu/libgcc_s.so.1
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe0ee000-7f1afe0fa000 r-xp 00000000 ca:09 1586794 /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe0fa000-7f1afe2f9000 ---p 0000c000 ca:09 1586794 /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe2f9000-7f1afe2fa000 r--p 0000b000 ca:09 1586794 /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe2fa000-7f1afe2fb000 rw-p 0000c000 ca:09 1586794 /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe2fb000-7f1afe46d000 r-xp 00000000 ca:09 1576308 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe46d000-7f1afe66d000 ---p 00172000 ca:09 1576308 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe66d000-7f1afe677000 r--p 00172000 ca:09 1576308 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe677000-7f1afe679000 rw-p 0017c000 ca:09 1576308 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe679000-7f1afe67d000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe67d000-7f1afe769000 r-xp 00000000 ca:09 1586835 /usr/lib/x86_64-linux-gnu/libkyotocabinet.so.16.13.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe769000-7f1afe969000 ---p 000ec000 ca:09 1586835 /usr/lib/x86_64-linux-gnu/libkyotocabinet.so.16.13.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe969000-7f1afe96d000 r--p 000ec000 ca:09 1586835 /usr/lib/x86_64-linux-gnu/libkyotocabinet.so.16.13.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe96d000-7f1afe96e000 rw-p 000f0000 ca:09 1586835 /usr/lib/x86_64-linux-gnu/libkyotocabinet.so.16.13.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe96e000-7f1afe97d000 r-xp 00000000 ca:09 1586839 /usr/lib/x86_64-linux-gnu/libkyototycoon.so.2.19.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afe97d000-7f1afeb7d000 ---p 0000f000 ca:09 1586839 /usr/lib/x86_64-linux-gnu/libkyototycoon.so.2.19.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afeb7d000-7f1afeb7e000 r--p 0000f000 ca:09 1586839 /usr/lib/x86_64-linux-gnu/libkyototycoon.so.2.19.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afeb7e000-7f1afeb7f000 rw-p 00010000 ca:09 1586839 /usr/lib/x86_64-linux-gnu/libkyototycoon.so.2.19.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afeb7f000-7f1afec87000 r-xp 00000000 ca:09 1575437 /lib/x86_64-linux-gnu/libm-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afec87000-7f1afee86000 ---p 00108000 ca:09 1575437 /lib/x86_64-linux-gnu/libm-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afee86000-7f1afee87000 r--p 00107000 ca:09 1575437 /lib/x86_64-linux-gnu/libm-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afee87000-7f1afee88000 rw-p 00108000 ca:09 1575437 /lib/x86_64-linux-gnu/libm-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afee88000-7f1afeea0000 r-xp 00000000 ca:09 1575473 /lib/x86_64-linux-gnu/libpthread-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1afeea0000-7f1aff09f000 ---p 00018000 ca:09 1575473 /lib/x86_64-linux-gnu/libpthread-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff09f000-7f1aff0a0000 r--p 00017000 ca:09 1575473 /lib/x86_64-linux-gnu/libpthread-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff0a0000-7f1aff0a1000 rw-p 00018000 ca:09 1575473 /lib/x86_64-linux-gnu/libpthread-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff0a1000-7f1aff0a5000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff0a5000-7f1aff0be000 r-xp 00000000 ca:09 1575504 /lib/x86_64-linux-gnu/libz.so.1.2.8
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff0be000-7f1aff2bd000 ---p 00019000 ca:09 1575504 /lib/x86_64-linux-gnu/libz.so.1.2.8
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff2bd000-7f1aff2be000 r--p 00018000 ca:09 1575504 /lib/x86_64-linux-gnu/libz.so.1.2.8
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff2be000-7f1aff2bf000 rw-p 00019000 ca:09 1575504 /lib/x86_64-linux-gnu/libz.so.1.2.8
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff2bf000-7f1aff3c4000 r-xp 00000000 ca:09 1586918 /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff3c4000-7f1aff5c4000 ---p 00105000 ca:09 1586918 /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff5c4000-7f1aff5c5000 r--p 00105000 ca:09 1586918 /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff5c5000-7f1aff5c6000 rw-p 00106000 ca:09 1586918 /usr/lib/x86_64-linux-gnu/libtokyocabinet.so.9.11.0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff5c6000-7f1aff5ca000 r-xp 00000000 ca:09 1575389 /lib/x86_64-linux-gnu/libSegFault.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff5ca000-7f1aff7c9000 ---p 00004000 ca:09 1575389 /lib/x86_64-linux-gnu/libSegFault.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff7c9000-7f1aff7ca000 r--p 00003000 ca:09 1575389 /lib/x86_64-linux-gnu/libSegFault.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff7ca000-7f1aff7cb000 rw-p 00004000 ca:09 1575389 /lib/x86_64-linux-gnu/libSegFault.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff7cb000-7f1aff7f1000 r-xp 00000000 ca:09 1575385 /lib/x86_64-linux-gnu/ld-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff9e0000-7f1aff9ea000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff9ef000-7f1aff9f0000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff9f0000-7f1aff9f1000 r--p 00025000 ca:09 1575385 /lib/x86_64-linux-gnu/ld-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff9f1000-7f1aff9f2000 rw-p 00026000 ca:09 1575385 /lib/x86_64-linux-gnu/ld-2.23.so
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7f1aff9f2000-7f1aff9f3000 rw-p 00000000 00:00 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7ffe6eba3000-7ffe6ebc4000 rw-p 00000000 00:00 0 [stack]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7ffe6ebc4000-7ffe6ebc7000 r--p 00000000 00:00 0 [vvar]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    7ffe6ebc7000-7ffe6ebc9000 r-xp 00000000 00:00 0 [vdso]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
eb0aee2b-4107-40db-8bca-cde3ed0eda59    INFO:toil.fileStore:LOG-TO-MASTER: Max memory used for job CactusHalGeneratorUpWrapper (tool cactus_halGenerator) on JSON features {"maxFlowerSize": 4, "flowerGroupSize": 4, "numFlowers": 1}: 0
eb0aee2b-4107-40db-8bca-cde3ed0eda59    Traceback (most recent call last):
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 314, in workerScript
eb0aee2b-4107-40db-8bca-cde3ed0eda59        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/tmp/tmpS1UzIC/01db939c0566a9b6f72a2360703a405e/cactus/shared/common.py", line 1096, in _runner
eb0aee2b-4107-40db-8bca-cde3ed0eda59        super(RoundedJob, self)._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1351, in _runner
eb0aee2b-4107-40db-8bca-cde3ed0eda59        returnValues = self._run(jobGraph, fileStore)
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1296, in _run
eb0aee2b-4107-40db-8bca-cde3ed0eda59        return self.run(fileStore)
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/tmp/tmpS1UzIC/01db939c0566a9b6f72a2360703a405e/cactus/pipeline/cactus_workflow.py", line 1324, in run
eb0aee2b-4107-40db-8bca-cde3ed0eda59        self.getOptionalPhaseAttrib("showOnlySubstitutionsWithRespectToReference", bool))
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/tmp/tmpS1UzIC/01db939c0566a9b6f72a2360703a405e/cactus/shared/common.py", line 720, in runCactusHalGenerator
eb0aee2b-4107-40db-8bca-cde3ed0eda59        job_name=jobName, features=features, fileStore=fileStore)
eb0aee2b-4107-40db-8bca-cde3ed0eda59      File "/tmp/tmpS1UzIC/01db939c0566a9b6f72a2360703a405e/cactus/shared/common.py", line 1040, in cactus_call
eb0aee2b-4107-40db-8bca-cde3ed0eda59        raise RuntimeError("Command %s failed with output: %s" % (call, output))
eb0aee2b-4107-40db-8bca-cde3ed0eda59    RuntimeError: Command ['docker', 'run', '--interactive', '--net=host', '--log-driver=none', '-u', '0:0', '-v', '/var/lib/toil/toil-b89e206a-ebc8-499a-a0f7-dae0a7ce14f9-109549f5-fea1-46ce-b47e-f14588626e09/tmp_jeExw/37faddac-9006-45a1-8400-35f4258ef61c:/data', '--name', '2e1bf0f3-a247-4896-bf05-885d013f97ca', '--rm', 'quay.io/comparative-genomics-toolkit/cactus:d4a0e82ebc75dcb7a260a31cf8682d971ea509bf', 'cactus_halGenerator', '--logLevel', 'INFO', '--cactusDisk', '<st_kv_database_conf type="kyoto_tycoon">\n\t\t\t<kyoto_tycoon database_dir="fakepath" host="10.0.130.252" port="6092" />\n\t\t</st_kv_database_conf>\n\t', '--secondaryDisk', '<st_kv_database_conf type="kyoto_tycoon">\n   <kyoto_tycoon database_dir="fakepath" host="10.0.130.252" port="30916" />\n  </st_kv_database_conf>\n ', '--referenceEventString', 'Anc2'] failed with output: None
eb0aee2b-4107-40db-8bca-cde3ed0eda59    ERROR:toil.worker:Exiting the worker because of a failed job on host ip-10-0-97-160.ec2.internal

I have two questions about these error.

  1. Are these errors related?
  2. Is it caused by the shortage of mermoy? Because I run the example sequence of evolverMammals successfully using such the almost same commond: 
    TOIL_AWS_AMI=ami-aff944d5 nohup cactus --stats --nodeTypes c4.8xlarge:0.6,r3.8xlarge --minNodes 0,0 --maxNodes 1,1 --provisioner aws --batchSystem mesos --metrics aws:us-east-1:test-jobstore-8 evolverMammals.txt evolverMammals.hal &

    The difference between the example sequences and primate genome is the size of data, so whether it is possible that the increase of data size increase the request of memory? Thank you.

joelarmstrong commented 5 years ago

Interesting. The server appears to still be up at the time the crash occurs. This may be an out-of-memory issue, though I'm not sure. The backtrace for future reference is:

segment_getStrand
/home/cactus/api/impl/cactusSegment.c:96
writeSegment
/home/cactus/hal/impl/hal.c:117
compress
/home/cactus/reference/impl/recursiveThreadBuilder.c:16
buildRecursiveThreads
/home/cactus/reference/impl/recursiveThreadBuilder.c:185
makeHalFormat
/home/cactus/hal/impl/hal.c:182
main
/home/cactus/hal/cactus_halGenerator.c:136
hehuiying1125 commented 3 years ago

Hello, When running cactus on AWS, I also met the same error to the first error. How did you solve it? If you could give me some suggestion, it will be appreciated.

Thank you.