BD2KGenomics / toil-scripts

Toil workflows for common genomic pipelines

Apache License 2.0

33 stars 18 forks source link

ip-172-31-22-168: 2016-07-24 23:09:37,127 INFO: toil.lib.bioio: Logging set at level: INFO ip-172-31-22-168: 2016-07-24 23:09:37,127 INFO: toil.lib.bioio: Logging set at level: INFO ip-172-31-22-168: 2016-07-24 23:09:37,129 INFO: toil.jobStores.fileJobStore: Path to job store directory is '/mnt/ephemeral/exome/jobstore'. ip-172-31-22-168: 2016-07-24 23:09:37,129 INFO: toil.jobStores.abstractJobStore: The workflow ID is: 'b3c4da6a-fcc3-4197-9608-26d5c161a790' ip-172-31-22-168: 2016-07-24 23:09:37,131 INFO: toil.common: Using the single machine batch system ip-172-31-22-168: 2016-07-24 23:09:37,131 WARNING: toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (32). ip-172-31-22-168: 2016-07-24 23:09:37,131 WARNING: toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (63321128960). ip-172-31-22-168: 2016-07-24 23:09:37,131 INFO: toil.batchSystems.singleMachine: Setting up the thread pool with 320 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 32. ip-172-31-22-168: 2016-07-24 23:09:37,187 INFO: toil.common: Written the environment for the jobs to the environment file ip-172-31-22-168: 2016-07-24 23:09:37,187 INFO: toil.common: Caching all jobs in job store ip-172-31-22-168: 2016-07-24 23:09:37,187 INFO: toil.common: 0 jobs downloaded. ip-172-31-22-168: 2016-07-24 23:09:37,684 INFO: toil.realtimeLogger: Real-time logging disabled ip-172-31-22-168: 2016-07-24 23:09:37,686 INFO: toil.leader: (Re)building internal scheduler state ip-172-31-22-168: 2016-07-24 23:09:37,687 INFO: toil.leader: Checked batch system has no running jobs and no updated jobs ip-172-31-22-168: 2016-07-24 23:09:37,687 INFO: toil.leader: Found 1 jobs to start and 0 jobs with successors to run ip-172-31-22-168: 2016-07-24 23:09:37,691 INFO: toil.leader: Starting the main loop ip-172-31-22-168: 2016-07-24 23:09:37,692 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore u/z/jobo9cL4i'. INFO:toil.common:Created the workflow directory at /mnt/ephemeral/exome/data/toil-b3c4da6a-fcc3-4197-9608-26d5c161a790 ip-172-31-22-168: 2016-07-24 23:09:38,160 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore M/5/jobQSuXkC'. ip-172-31-22-168: 2016-07-24 23:09:38,160 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 6/J/jobmgWBGT'. ip-172-31-22-168: 2016-07-24 23:09:38,164 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 4/F/job_N7yVe'. ip-172-31-22-168: 2016-07-24 23:09:38,165 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore e/z/jobz9ojxk'. ip-172-31-22-168: 2016-07-24 23:09:38,165 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore P/0/jobVdbhlt'. ip-172-31-22-168: 2016-07-24 23:09:38,217 INFO: toil.leader: Got message from job at time 07-24-2016 23:09:38: Downloaded shared files Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f5092ff6e10>> ignored ip-172-31-22-168: 2016-07-24 23:09:40,379 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 6/J/jobmgWBGT ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: Reporting file: 6/J/jobmgWBGT ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:40,382 WARNING: toil.leader: 6/J/jobmgWBGT: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 6/J/jobmgWBGT to 1 ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,383 WARNING: toil.leader: 6/J/jobmgWBGT: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,383 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 6/J/jobmgWBGT'. Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fcd00de2e10>> ignored ip-172-31-22-168: 2016-07-24 23:09:40,417 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 4/F/job_N7yVe ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: Reporting file: 4/F/job_N7yVe ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,418 WARNING: toil.leader: 4/F/job_N7yVe: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,419 WARNING: toil.leader: 4/F/job_N7yVe: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,420 WARNING: toil.leader: 4/F/job_N7yVe: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 4/F/job_N7yVe to 1 ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,421 WARNING: toil.leader: 4/F/job_N7yVe: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,421 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 4/F/job_N7yVe'. Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f3a38195e10>> ignored Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7ff9939c2e10>> ignored ip-172-31-22-168: 2016-07-24 23:09:40,455 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: e/z/jobz9ojxk ip-172-31-22-168: 2016-07-24 23:09:40,455 WARNING: toil.leader: Reporting file: e/z/jobz9ojxk ip-172-31-22-168: 2016-07-24 23:09:40,455 WARNING: toil.leader: e/z/jobz9ojxk: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:40,455 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,455 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,456 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,457 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:40,458 WARNING: toil.leader: e/z/jobz9ojxk: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,459 WARNING: toil.leader: e/z/jobz9ojxk: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,459 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job e/z/jobz9ojxk to 1 ip-172-31-22-168: 2016-07-24 23:09:40,459 WARNING: toil.leader: e/z/jobz9ojxk: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,459 WARNING: toil.leader: e/z/jobz9ojxk: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,459 WARNING: toil.leader: e/z/jobz9ojxk: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,459 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore e/z/jobz9ojxk'. ip-172-31-22-168: 2016-07-24 23:09:40,466 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: P/0/jobVdbhlt ip-172-31-22-168: 2016-07-24 23:09:40,466 WARNING: toil.leader: Reporting file: P/0/jobVdbhlt ip-172-31-22-168: 2016-07-24 23:09:40,466 WARNING: toil.leader: P/0/jobVdbhlt: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:40,466 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: ip-172-31-22-168: 2016-07-24 23:09:40,467 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,468 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:40,469 WARNING: toil.leader: P/0/jobVdbhlt: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job P/0/jobVdbhlt to 1 ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,470 WARNING: toil.leader: P/0/jobVdbhlt: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:40,470 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore P/0/jobVdbhlt'. Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f2c8c16de10>> ignored ip-172-31-22-168: 2016-07-24 23:09:42,389 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: M/5/jobQSuXkC ip-172-31-22-168: 2016-07-24 23:09:42,389 WARNING: toil.leader: Reporting file: M/5/jobQSuXkC ip-172-31-22-168: 2016-07-24 23:09:42,389 WARNING: toil.leader: M/5/jobQSuXkC: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:42,389 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,389 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,390 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,391 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:42,392 WARNING: toil.leader: M/5/jobQSuXkC: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,393 WARNING: toil.leader: M/5/jobQSuXkC: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,393 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job M/5/jobQSuXkC to 1 ip-172-31-22-168: 2016-07-24 23:09:42,393 WARNING: toil.leader: M/5/jobQSuXkC: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,393 WARNING: toil.leader: M/5/jobQSuXkC: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,393 WARNING: toil.leader: M/5/jobQSuXkC: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,393 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore M/5/jobQSuXkC'. Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f8480326e10>> ignored ip-172-31-22-168: 2016-07-24 23:09:42,610 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 6/J/jobmgWBGT ip-172-31-22-168: 2016-07-24 23:09:42,610 WARNING: toil.leader: Reporting file: 6/J/jobmgWBGT ip-172-31-22-168: 2016-07-24 23:09:42,610 WARNING: toil.leader: 6/J/jobmgWBGT: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:42,610 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,610 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,610 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,611 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,612 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,613 WARNING: toil.leader: 6/J/jobmgWBGT: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,614 WARNING: toil.leader: 6/J/jobmgWBGT: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 6/J/jobmgWBGT to 0 ip-172-31-22-168: 2016-07-24 23:09:42,614 WARNING: toil.leader: 6/J/jobmgWBGT: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,614 WARNING: toil.leader: 6/J/jobmgWBGT: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,614 WARNING: toil.leader: 6/J/jobmgWBGT: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,614 WARNING: toil.leader: Job: 6/J/jobmgWBGT is completely failed Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fc7dcbe6e10>> ignored Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f13d7f9ee10>> ignored Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f2c0f0ebe10>> ignored ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: e/z/jobz9ojxk ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: Reporting file: e/z/jobz9ojxk ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: e/z/jobz9ojxk: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,678 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,679 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,680 WARNING: toil.leader: e/z/jobz9ojxk: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:42,681 WARNING: toil.leader: e/z/jobz9ojxk: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job e/z/jobz9ojxk to 0 ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: e/z/jobz9ojxk: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: Job: e/z/jobz9ojxk is completely failed ip-172-31-22-168: 2016-07-24 23:09:42,682 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: P/0/jobVdbhlt ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: Reporting file: P/0/jobVdbhlt ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,683 WARNING: toil.leader: P/0/jobVdbhlt: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,684 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:42,685 WARNING: toil.leader: P/0/jobVdbhlt: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job P/0/jobVdbhlt to 0 ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: P/0/jobVdbhlt: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: Job: P/0/jobVdbhlt is completely failed ip-172-31-22-168: 2016-07-24 23:09:42,686 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 4/F/job_N7yVe ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: Reporting file: 4/F/job_N7yVe ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,687 WARNING: toil.leader: 4/F/job_N7yVe: ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: main( args ) ip-172-31-22-168: 2016-07-24 23:09:42,688 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,689 WARNING: toil.leader: 4/F/job_N7yVe: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 4/F/job_N7yVe to 0 ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: 4/F/job_N7yVe: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:42,690 WARNING: toil.leader: Job: 4/F/job_N7yVe is completely failed Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fd8f571be10>> ignored ip-172-31-22-168: 2016-07-24 23:09:44,608 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: M/5/jobQSuXkC ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: Reporting file: M/5/jobQSuXkC ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: ---TOIL WORKER OUTPUT LOG--- ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:The localize() method should only be invoked on a worker. ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts' ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls') ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: main( args ) ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:44,609 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: main( args ) ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:44,610 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: main( args ) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: returnValues = self._run(jobWrapper, fileStore) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: return self.run(fileStore) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) ip-172-31-22-168: 2016-07-24 23:09:44,611 WARNING: toil.leader: M/5/jobQSuXkC: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: s3_key_path=s3_key_path, cghub_key_path=cghub_key_path) ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path) ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count)) ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: RuntimeError: S3AM failed to download after 3 retries. ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168 ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job M/5/jobQSuXkC to 0 ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: M/5/jobQSuXkC: S3AM failed with status code: 1 ip-172-31-22-168: 2016-07-24 23:09:44,612 WARNING: toil.leader: Job: M/5/jobQSuXkC is completely failed ip-172-31-22-168: 2016-07-24 23:09:46,612 INFO: toil.leader: No jobs left to run so exiting. ip-172-31-22-168: 2016-07-24 23:09:46,613 INFO: toil.leader: Finished the main loop ip-172-31-22-168: 2016-07-24 23:09:46,613 INFO: toil.leader: Waiting for stats and logging collator process to finish ... ip-172-31-22-168: 2016-07-24 23:09:46,736 INFO: toil.leader: ... finished collating stats and logs. Took 0.123184919357 seconds ip-172-31-22-168: 2016-07-24 23:09:46,736 INFO: toil.leader: Waiting for service manager thread to finish ... ip-172-31-22-168: 2016-07-24 23:09:47,689 INFO: toil.leader: ... finished shutting down the service manager. Took 0.95231795311 seconds ip-172-31-22-168: 2016-07-24 23:09:47,689 INFO: toil.leader: Finished toil run with 6 failed jobs ip-172-31-22-168: 2016-07-24 23:09:47,689 INFO: toil.leader: Failed jobs at end of the run: set(['M/5/jobQSuXkC', 'u/z/jobo9cL4i', '6/J/jobmgWBGT', 'e/z/jobz9ojxk', '4/F/job_N7yVe', 'P/0/jobVdbhlt']) Traceback (most recent call last): File "/home/ubuntu/toil-scripts/bin/toil-exome", line 11, in <module> sys.exit(main()) File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/exome_variant_pipeline/exome_variant_pipeline.py", line 424, in main Job.Runner.startToil(Job.wrapJobFn(download_shared_files, samples, config), args) File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 500, in startToil return toil.start(job) File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 561, in start return self._runMainLoop(job) File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 775, in _runMainLoop jobCache=self._jobCache) File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 694, in mainLoop raise FailedJobsException( config.jobStore, len(toilState.totalFailedJobs) ) toil.leader.FailedJobsException: The job store '/mnt/ephemeral/exome/jobstore' contains 6 failed jobs

ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: Traceback (most recent call last): ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/usr/local/bin/s3am", line 11, in <module> ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: sys.exit(try_main()) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: main( args ) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: result = operation.run( ) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: key = bucket.get_key( self.key_name, headers=headers ) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: key, resp = self._get_key_internal(key_name, headers, query_args_l) ip-172-31-22-168: 2016-07-24 23:09:40,380 WARNING: toil.leader: 6/J/jobmgWBGT: File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: response.status, response.reason, '') ip-172-31-22-168: 2016-07-24 23:09:40,381 WARNING: toil.leader: 6/J/jobmgWBGT: boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request

Test toil-exome at 2016-07-25 22:37:48.791056

Command:

toil-exome run /mnt/ephemeral/exome/jobstore --retryCount=1 --workDir=/mnt/ephemeral/exome/

Config:

# CGL Exome Pipeline configuration file
# This configuration file is formatted in YAML. Simply write the value (at least one space) after the colon.
# Edit the values in this configuration file and then rerun the pipeline: "toil-variant run"
# URLs can take the form: http://, file://, s3://, gnos://.
# Comments (beginning with #) do not need to be removed. Optional parameters may be left blank
####################################################################################################################
# Required: URL to reference genome
reference: s3://cgl-pipeline-inputs/variant_hg19/hg19.fa

# Required: URL to phase indels VCF
phase: s3://cgl-pipeline-inputs/variant_hg19/1000G_phase1.indels.hg19.sites.vcf

# Required: URL to Mills indel VCF
mills: s3://cgl-pipeline-inputs/variant_hg19/Mills_and_1000G_gold_standard.indels.hg19.sites.vcf

# Required: URL to dbsnp VCF
dbsnp: s3://cgl-pipeline-inputs/variant_hg19/dbsnp_138.hg19.vcf

# Required: URL to cosmic VCF
cosmic: s3://cgl-pipeline-inputs/variant_hg19/cosmic.hg19.vcf

# Optional: If true, will run MuTect to do mutation calls
run-mutect: true

# Optional: If true, will run pindel to analyze indel
run-pindel: true

# Optional: If true, will run MuSe to do mutation calls
run-muse: true

# Optional: If true, will perform indel realignment and base quality score recalibration
preprocessing: true

# Optional: Provide a full path to where results will appear
output-dir: /home/ubuntu/pipelines/exome/out

# Optional: Provide an s3 path (s3://bucket/dir) where results will appear
s3-output-dir:

# Optional: Provide a full path to a 32-byte key used for SSE-C Encryption in Amazon
ssec: /home/ubuntu/.key

# Optional: Provide a full path to a CGHub Key used to access GNOS hosted data
gtkey:

# Optional: If true, uses resource requirements appropriate for continuous integration
ci-test:

Manifest:

#   Edit this manifest to include information pertaining to each sample pair to be run.
#   There are 3 tab-separated columns: UUID, Normal BAM URL, Tumor BAM URL
#
#   UUID            This should be a unique identifier for the sample to be processed
#   Normal URL      A URL (http://, file://, s3://, gnos://) pointing to the normal bam
#   Tumor URL       A URL (http://, file://, s3://, gnos://) pointing to the tumor bam
#
#   Examples of several combinations are provided below. Lines beginning with # are ignored.
#
#   UUID_1  file:///path/to/normal.bam  file:///path/to/tumor.bam
#   UUID_2  http://sample-depot.com/normal.bam  http://sample-depot.com/tumor.bam
#   UUID_3  s3://my-bucket-name/directory/normal.bam    file:///path/to/tumor.bam
#
#   Place your samples below, one per line.
jake-test   s3://cgl-driver-projects-encrypted/ucsf-pnoc/ucsf_issue52_input/DTB-005-BL-T-DNA-HSmerge_S1.bam s3://cgl-driver-projects-encrypted/ucsf-pnoc/ucsf_issue52_input/DTB-005-N-DNA-HS1_S1.bam

stdout:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f3561d58dd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fe3e83ccdd0>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7feaba7fbdd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f4f81afddd0>> ignored
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fdeb944ddd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fd862d8cdd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7ffbd560fdd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7f9173450dd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7fd31d223dd0>> ignored
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/toil/fileStore.py", line 1215, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method FileStore.__del__ of <toil.fileStore.FileStore object at 0x7faefe301dd0>> ignored

stderr:

ip-172-31-22-168: 2016-07-25 22:37:48,882 INFO: toil.lib.bioio: Logging set at level: INFO
ip-172-31-22-168: 2016-07-25 22:37:48,882 INFO: toil.lib.bioio: Logging set at level: INFO
ip-172-31-22-168: 2016-07-25 22:37:48,884 INFO: toil.jobStores.fileJobStore: Path to job store directory is '/mnt/ephemeral/exome/jobstore'.
ip-172-31-22-168: 2016-07-25 22:37:48,884 INFO: toil.jobStores.abstractJobStore: The workflow ID is: '80b962a5-3603-4849-9780-f2308cab8e67'
ip-172-31-22-168: 2016-07-25 22:37:48,886 INFO: toil.common: Using the single machine batch system
ip-172-31-22-168: 2016-07-25 22:37:48,887 WARNING: toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (32).
ip-172-31-22-168: 2016-07-25 22:37:48,887 WARNING: toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (63321128960).
ip-172-31-22-168: 2016-07-25 22:37:48,887 INFO: toil.batchSystems.singleMachine: Setting up the thread pool with 320 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 32.
ip-172-31-22-168: 2016-07-25 22:37:48,946 INFO: toil.common: Written the environment for the jobs to the environment file
ip-172-31-22-168: 2016-07-25 22:37:48,946 INFO: toil.common: Caching all jobs in job store
ip-172-31-22-168: 2016-07-25 22:37:48,946 INFO: toil.common: 0 jobs downloaded.
ip-172-31-22-168: 2016-07-25 22:37:48,952 INFO: toil.realtimeLogger: Real-time logging disabled
ip-172-31-22-168: 2016-07-25 22:37:48,995 INFO: toil.leader: (Re)building internal scheduler state
ip-172-31-22-168: 2016-07-25 22:37:48,995 INFO: toil.leader: Checked batch system has no running jobs and no updated jobs
ip-172-31-22-168: 2016-07-25 22:37:48,995 INFO: toil.leader: Found 1 jobs to start and 0 jobs with successors to run
ip-172-31-22-168: 2016-07-25 22:37:49,000 INFO: toil.leader: Starting the main loop
ip-172-31-22-168: 2016-07-25 22:37:49,001 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore y/q/jobvQQoOR'.
INFO:toil.common:Created the workflow directory at /mnt/ephemeral/exome/toil-80b962a5-3603-4849-9780-f2308cab8e67
ip-172-31-22-168: 2016-07-25 22:37:49,267 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore w/U/job7c6pc0'.
ip-172-31-22-168: 2016-07-25 22:37:49,268 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore s/T/jobnxuFPx'.
ip-172-31-22-168: 2016-07-25 22:37:49,268 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 3/p/job8GGmgR'.
ip-172-31-22-168: 2016-07-25 22:37:49,273 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 4/9/job2hFwQY'.
ip-172-31-22-168: 2016-07-25 22:37:49,273 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore h/F/jobZ8Y3NI'.
ip-172-31-22-168: 2016-07-25 22:37:49,527 INFO: toil.leader: Got message from job at time 07-25-2016 22:37:49: Downloaded shared files
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: s/T/jobnxuFPx
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: Reporting file: s/T/jobnxuFPx
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: s/T/jobnxuFPx:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,487 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,488 WARNING: toil.leader: s/T/jobnxuFPx:  
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:  
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,489 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:  
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:51,490 WARNING: toil.leader: s/T/jobnxuFPx:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job s/T/jobnxuFPx to 1
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,491 WARNING: toil.leader: s/T/jobnxuFPx:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,491 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore s/T/jobnxuFPx'.
ip-172-31-22-168: 2016-07-25 22:37:51,495 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: w/U/job7c6pc0
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: Reporting file: w/U/job7c6pc0
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,496 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:  
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,497 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:  
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:  
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:51,498 WARNING: toil.leader: w/U/job7c6pc0:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job w/U/job7c6pc0 to 1
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,499 WARNING: toil.leader: w/U/job7c6pc0:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,500 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore w/U/job7c6pc0'.
ip-172-31-22-168: 2016-07-25 22:37:51,532 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 3/p/job8GGmgR
ip-172-31-22-168: 2016-07-25 22:37:51,532 WARNING: toil.leader: Reporting file: 3/p/job8GGmgR
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,533 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:  
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:  
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,534 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:  
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,535 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 3/p/job8GGmgR to 1
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,536 WARNING: toil.leader: 3/p/job8GGmgR:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,537 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 3/p/job8GGmgR'.
ip-172-31-22-168: 2016-07-25 22:37:51,552 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 4/9/job2hFwQY
ip-172-31-22-168: 2016-07-25 22:37:51,552 WARNING: toil.leader: Reporting file: 4/9/job2hFwQY
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,553 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:  
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:  
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,554 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,555 WARNING: toil.leader: 4/9/job2hFwQY:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,555 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,555 WARNING: toil.leader: 4/9/job2hFwQY:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,555 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,558 WARNING: toil.leader: 4/9/job2hFwQY:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:  
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:51,559 WARNING: toil.leader: 4/9/job2hFwQY:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 4/9/job2hFwQY to 1
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,560 WARNING: toil.leader: 4/9/job2hFwQY:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,561 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore 4/9/job2hFwQY'.
ip-172-31-22-168: 2016-07-25 22:37:51,593 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: h/F/jobZ8Y3NI
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: Reporting file: h/F/jobZ8Y3NI
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,594 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:  
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,595 WARNING: toil.leader: h/F/jobZ8Y3NI:  
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:  
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:51,596 WARNING: toil.leader: h/F/jobZ8Y3NI:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job h/F/jobZ8Y3NI to 1
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,597 WARNING: toil.leader: h/F/jobZ8Y3NI:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:51,598 INFO: toil.batchSystems.singleMachine: Executing command: '_toil_worker /mnt/ephemeral/exome/jobstore h/F/jobZ8Y3NI'.
ip-172-31-22-168: 2016-07-25 22:37:53,712 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: s/T/jobnxuFPx
ip-172-31-22-168: 2016-07-25 22:37:53,712 WARNING: toil.leader: Reporting file: s/T/jobnxuFPx
ip-172-31-22-168: 2016-07-25 22:37:53,712 WARNING: toil.leader: s/T/jobnxuFPx:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:53,712 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,712 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,712 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  
ip-172-31-22-168: 2016-07-25 22:37:53,713 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:  
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,714 WARNING: toil.leader: s/T/jobnxuFPx:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:  
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:53,715 WARNING: toil.leader: s/T/jobnxuFPx:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job s/T/jobnxuFPx to 0
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: s/T/jobnxuFPx:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,716 WARNING: toil.leader: Job: s/T/jobnxuFPx is completely failed
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: w/U/job7c6pc0
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: Reporting file: w/U/job7c6pc0
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,724 WARNING: toil.leader: w/U/job7c6pc0:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:  
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,725 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:  
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:  
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:53,726 WARNING: toil.leader: w/U/job7c6pc0:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job w/U/job7c6pc0 to 0
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,727 WARNING: toil.leader: w/U/job7c6pc0:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,728 WARNING: toil.leader: Job: w/U/job7c6pc0 is completely failed
ip-172-31-22-168: 2016-07-25 22:37:53,759 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 3/p/job8GGmgR
ip-172-31-22-168: 2016-07-25 22:37:53,759 WARNING: toil.leader: Reporting file: 3/p/job8GGmgR
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,760 WARNING: toil.leader: 3/p/job8GGmgR:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:  
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:  
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,761 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:  
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:53,762 WARNING: toil.leader: 3/p/job8GGmgR:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 3/p/job8GGmgR to 0
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: 3/p/job8GGmgR:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,763 WARNING: toil.leader: Job: 3/p/job8GGmgR is completely failed
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: 4/9/job2hFwQY
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: Reporting file: 4/9/job2hFwQY
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,779 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:  
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,780 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:  
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,781 WARNING: toil.leader: 4/9/job2hFwQY:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:  
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,782 WARNING: toil.leader: 4/9/job2hFwQY:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,783 WARNING: toil.leader: 4/9/job2hFwQY:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job 4/9/job2hFwQY to 0
ip-172-31-22-168: 2016-07-25 22:37:53,783 WARNING: toil.leader: 4/9/job2hFwQY:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,783 WARNING: toil.leader: 4/9/job2hFwQY:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,783 WARNING: toil.leader: 4/9/job2hFwQY:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,783 WARNING: toil.leader: Job: 4/9/job2hFwQY is completely failed
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: The jobWrapper seems to have left a log file, indicating failure: h/F/jobZ8Y3NI
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: Reporting file: h/F/jobZ8Y3NI
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  ---TOIL WORKER OUTPUT LOG---
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:The localize() method should only be invoked on a worker.
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't find resource for leader path '/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts'
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.resource:Can't localize module ModuleDescriptor(dirPath='/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages', name='toil_scripts.lib.urls')
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,815 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:  
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,816 WARNING: toil.leader: h/F/jobZ8Y3NI:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:  
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/bin/s3am", line 11, in <module>
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      sys.exit(try_main())
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 43, in try_main
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      main( args )
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/cli.py", line 103, in main
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      result = operation.run( )
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/s3am/operations.py", line 1044, in run
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      key = bucket.get_key( self.key_name, headers=headers )
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      key, resp = self._get_key_internal(key_name, headers, query_args_l)
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/opt/s3am/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      response.status, response.reason, '')
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:  boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:  
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:  Traceback (most recent call last):
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/worker.py", line 330, in main
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      job._runner(jobWrapper=jobWrapper, jobStore=jobStore, fileStore=fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1043, in _runner
ip-172-31-22-168: 2016-07-25 22:37:53,817 WARNING: toil.leader: h/F/jobZ8Y3NI:      returnValues = self._run(jobWrapper, fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 991, in _run
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:      return self.run(fileStore)
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 1136, in run
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:      rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 40, in download_url_job
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:      s3_key_path=s3_key_path, cghub_key_path=cghub_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 27, in download_url
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:      _s3am_with_retry(num_cores=1, file_path=file_path, s3_url=url, mode='download', s3_key_path=s3_key_path)
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:    File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/lib/urls.py", line 111, in _s3am_with_retry
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:      raise RuntimeError('S3AM failed to {} after {} retries.'.format(mode, retry_count))
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  RuntimeError: S3AM failed to download after 3 retries.
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  ERROR:toil.worker:Exiting the worker because of a failed jobWrapper on host ip-172-31-22-168
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  WARNING:toil.jobWrapper:Due to failure we are reducing the remaining retry count of job h/F/jobZ8Y3NI to 0
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: h/F/jobZ8Y3NI:  S3AM failed with status code: 1
ip-172-31-22-168: 2016-07-25 22:37:53,818 WARNING: toil.leader: Job: h/F/jobZ8Y3NI is completely failed
ip-172-31-22-168: 2016-07-25 22:37:55,819 INFO: toil.leader: No jobs left to run so exiting.
ip-172-31-22-168: 2016-07-25 22:37:55,819 INFO: toil.leader: Finished the main loop
ip-172-31-22-168: 2016-07-25 22:37:55,819 INFO: toil.leader: Waiting for stats and logging collator process to finish ...
ip-172-31-22-168: 2016-07-25 22:37:56,043 INFO: toil.leader: ... finished collating stats and logs. Took 0.224394798279 seconds
ip-172-31-22-168: 2016-07-25 22:37:56,043 INFO: toil.leader: Waiting for service manager thread to finish ...
ip-172-31-22-168: 2016-07-25 22:37:56,997 INFO: toil.leader: ... finished shutting down the service manager. Took 0.953212976456 seconds
ip-172-31-22-168: 2016-07-25 22:37:56,997 INFO: toil.leader: Finished toil run with 6 failed jobs
ip-172-31-22-168: 2016-07-25 22:37:56,997 INFO: toil.leader: Failed jobs at end of the run: set(['w/U/job7c6pc0', 'y/q/jobvQQoOR', '3/p/job8GGmgR', '4/9/job2hFwQY', 'h/F/jobZ8Y3NI', 's/T/jobnxuFPx'])
Traceback (most recent call last):
  File "/home/ubuntu/toil-scripts/bin/toil-exome", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/toil-scripts/local/lib/python2.7/site-packages/toil_scripts/exome_variant_pipeline/exome_variant_pipeline.py", line 424, in main
    Job.Runner.startToil(Job.wrapJobFn(download_shared_files, samples, config), args)
  File "/usr/local/lib/python2.7/dist-packages/toil/job.py", line 500, in startToil
    return toil.start(job)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 561, in start
    return self._runMainLoop(job)
  File "/usr/local/lib/python2.7/dist-packages/toil/common.py", line 775, in _runMainLoop
    jobCache=self._jobCache)
  File "/usr/local/lib/python2.7/dist-packages/toil/leader.py", line 694, in mainLoop
    raise FailedJobsException( config.jobStore, len(toilState.totalFailedJobs) )
toil.leader.FailedJobsException: The job store '/mnt/ephemeral/exome/jobstore' contains 6 failed jobs