search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
332 stars 229 forks source link

tmp Dir exists already #1036

Open ionox0 opened 5 years ago

ionox0 commented 5 years ago

I see a stochastic error for certain jobs when kicking off many jobs at the same time (~100 or so).

Expected Behavior

tmp dir should be created once and subsequently checked for existence.

Actual Behavior

OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'

Command issued

toil-cwl-runner \
--logFile /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/log/cwltoil.log \
--jobStore file:///ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/tmp//jobstore-e18d4790-1ab8-11e9-bf33-70106fb1697e \
--batchSystem lsf \
--workDir /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/tmp/ \
--outdir /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV \
--writeLogs /ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/ValidationRedo-0.1.0-DEV/log \
--logLevel INFO \
--stats   \
--cleanWorkDir onSuccess  \
--defaultDisk 10G  \
--preserve-environment PATH PYTHONPATH TOIL_LSF_ARGS  \
--retryCount 2  \
--no-container   \
--defaultMem 10G  \
--disableCaching   \
--maxLogFileSize 20000000  \
/home/johnsoni/pipeline_0.1.0-DEV/ACCESS-Pipeline/workflows/subworkflows/snps_and_indels.cwl inputs.yaml

Note that the values that I am supplying for the --workDir and --outdir parameters are not in agreement with the tmp directory that is causing the error.

This mysterious tmp directory shows up in the current working directory from which the command was submitted.

I am using toil, and this directory has appeared since upgrading from 3.15 to 3.18.

However it seems like an issue with cwltool.

Workflow Code

The pipeline I'm running is quite large, but I believe this issue is instead caused by submitting a large number of jobs concurrently.

I'm not familiar with all of the particular ways in which temporary directories are created by cwltool, and it seems there are several different ways of specifying the tmp dir location (TMPDIR environment variable, --tmp-dir-prefix, --tmp-outdir-prefix).

But my best guess is that there may be a race condition when creating the tmp dir because of this line:

https://github.com/common-workflow-language/cwltool/blob/1.0.20180819175200/cwltool/job.py#L381

        if not os.path.exists(self.tmpdir):
            os.makedirs(self.tmpdir)

Which will error if two jobs reach the same point at the same time without synchronization (similarly for any other places where this tmp dir is going to be created).

Full Traceback

WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'file:///home/johnsoni/pipeline_0.1.0-DEV/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java w/n/jobhgbJto
WARNING:toil.leader:w/n/jobhgbJto    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:w/n/jobhgbJto    INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:w/n/jobhgbJto    Got workflow error
WARNING:toil.leader:w/n/jobhgbJto    Traceback (most recent call last):
WARNING:toil.leader:w/n/jobhgbJto      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:w/n/jobhgbJto        job.run(runtime_context)
WARNING:toil.leader:w/n/jobhgbJto      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 378, in run
WARNING:toil.leader:w/n/jobhgbJto        os.makedirs(self.tmpdir)
WARNING:toil.leader:w/n/jobhgbJto      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:w/n/jobhgbJto        mkdir(name, mode)
WARNING:toil.leader:w/n/jobhgbJto    OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'
WARNING:toil.leader:w/n/jobhgbJto    ERROR:cwltool:Got workflow error
WARNING:toil.leader:w/n/jobhgbJto    Traceback (most recent call last):
WARNING:toil.leader:w/n/jobhgbJto      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:w/n/jobhgbJto        job.run(runtime_context)
WARNING:toil.leader:w/n/jobhgbJto      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 378, in run
WARNING:toil.leader:w/n/jobhgbJto        os.makedirs(self.tmpdir)
WARNING:toil.leader:w/n/jobhgbJto      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-DEV/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:w/n/jobhgbJto        mkdir(name, mode)
WARNING:toil.leader:w/n/jobhgbJto    OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/pipeline_0.1.0-DEV-TEST/tmp'

Your Environment

mr-c commented 5 years ago

@ionox0 Thank you for your issue, I believe you have made a correct assessment of the source of the problem.

As for a fix, we can certainly put a lock here

ionox0 commented 5 years ago

Thanks, I hope I've provided enough info, I'm still working towards being able to contribute more effectively to Toil. But I'd like to be able to run the test suite locally so I can add tests.

I'm having trouble installing the google.cloud, htcondor dependencies on my mac. It seems htcondor is only available for python on linux. Is the recommended way to develop for toil to be running on a linux machine?

Or perhaps it's ok for the htcondor and google-cloud tests to fail (8 / 485 failing)

mr-c commented 5 years ago

Can you test https://github.com/common-workflow-language/cwltool/pull/1037 ?

ionox0 commented 5 years ago

I went in and manually put the lock around my version of cwltool that I'm currently using the same way as in #1037 , just because it was easier, but I'll take a look at the version from the PR specifically when I get a chance.

Putting the lock around those lines unfortunately did not fix it for me.

|   378 class CommandLineJob(JobBase):
|   379 
|   380     def run(self,
|   381             runtimeContext     # type: RuntimeContext
|   382            ):  # type: (...) -> None
|   383 
|   384         print('before lock')
|   385         with TMPDIR_LOCK:
|   386             print('lock acquired')
|   387             if not os.path.exists(self.tmpdir):
|   388                 os.makedirs(self.tmpdir)
|   389         print('lock released')

I also noticed that it is consistently the first one or two jobs of the workflow that have this issue, and subsequent jobs are ok. Perhaps it is less stochastic than I thought.

Will get back with more info.

INFO:cwltool:Resolved '/home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/workflows/subworkflows/snps_and_indels.cwl' to 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/workflows/subworkflows/snps_and_indels.cwl'
WARNING:cwltool:Workflow checker warning:
../../../../../../../home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/workflows/module-4.cwl:156:7: 'access_filters_params'
                                                                                                    is not an input
                                                                                                    parameter of
                                                                                                    file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/python/ACCESS_filters.cwl,
                                                                                                    expected
                                                                                                    DS_curated_detect_alt_thres,
                                                                                                    DS_tumor_detect_alt_thres,
                                                                                                    anno_maf,
                                                                                                    curated_detect_alt_thres,
                                                                                                    fillout_maf,
                                                                                                    min_n_curated_samples_alt_detected,
                                                                                                    normal_TD_min,
                                                                                                    normal_samplename,
                                                                                                    normal_vaf_germline_thres,
                                                                                                    tier_one_alt_min,
                                                                                                    tn_ratio_thres,
                                                                                                    tumor_TD_min,
                                                                                                    tumor_detect_alt_thres,
                                                                                                    tumor_samplename,
                                                                                                    tumor_vaf_germline_thres
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (270748786688).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (840479052136448).
INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:A result seems to already have been processed for job 0
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/N/jobDblyr_ with job batch system ID: 103 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' S/G/jobVdqxr9 with job batch system ID: 104 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java Y/H/jobg5UM6H with job batch system ID: 105 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' H/T/jobq1C3oo with job batch system ID: 106 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java S/E/jobxyanUX with job batch system ID: 107 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' d/A/jobWIqgxL with job batch system ID: 108 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java U/Y/job2If_0v with job batch system ID: 109 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' A/y/jobmMy6YI with job batch system ID: 110 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java s/E/job96aiz8 with job batch system ID: 111 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' q/Q/jobG4AHA3 with job batch system ID: 112 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java T/n/jobPGF6CK with job batch system ID: 113 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' p/O/job8HF6uu with job batch system ID: 114 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java t/9/jobaOao7z with job batch system ID: 115 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' L/F/jobUtKm2w with job batch system ID: 116 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java D/Y/job3KMWXK with job batch system ID: 117 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' m/B/jobQq0Q6k with job batch system ID: 118 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java M/s/jobuiMkfY with job batch system ID: 119 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' Y/J/jobQmDCHl with job batch system ID: 120 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java B/s/jobLUjpo3 with job batch system ID: 121 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 9/V/jobvA_bxX with job batch system ID: 122 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java M/F/jobsXuV4k with job batch system ID: 123 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' X/T/jobEad7wW with job batch system ID: 124 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java a/p/jobBKDl4I with job batch system ID: 125 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' z/g/jobAEkumf with job batch system ID: 126 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 8/5/jobZDafJX with job batch system ID: 127 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' I/9/jobj7nLzp with job batch system ID: 128 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java s/L/jobkR3d5y with job batch system ID: 129 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' W/P/jobqWOH7V with job batch system ID: 130 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/E/jobE9BdRO with job batch system ID: 131 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' i/c/jobUsHqzl with job batch system ID: 132 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java j/s/jobeCCAL7 with job batch system ID: 133 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' c/5/job_XRt4w with job batch system ID: 134 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/y/jobJHNbPD with job batch system ID: 135 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' B/V/jobqDYUaz with job batch system ID: 136 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/8/jobiR0PN4 with job batch system ID: 137 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' c/g/jobXdcEQv with job batch system ID: 138 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/p/jobR80Kdb with job batch system ID: 139 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' F/h/job4JLZ5u with job batch system ID: 140 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java x/q/job_UG97E with job batch system ID: 141 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 2/V/jobPE165D with job batch system ID: 142 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java L/2/jobnxOsve with job batch system ID: 143 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' j/C/jobFw3wJb with job batch system ID: 144 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java A/B/job9OaqRe with job batch system ID: 145 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' o/D/job5XYkj_ with job batch system ID: 146 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java O/r/jobUYELPY with job batch system ID: 147 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' f/G/jobcCHHOi with job batch system ID: 148 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java n/C/jobxTSRw7 with job batch system ID: 149 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' D/i/job5OKYQy with job batch system ID: 150 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 6/X/jobqLgrKS with job batch system ID: 151 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' p/i/job5YzgzZ with job batch system ID: 152 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 8/O/job5WD1Mz with job batch system ID: 153 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' c/x/jobrfdFmi with job batch system ID: 154 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java R/4/joboMSIAC with job batch system ID: 155 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' g/L/jobRrs8lA with job batch system ID: 156 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java g/U/jobLYlA8D with job batch system ID: 157 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' E/S/jobHuLVCW with job batch system ID: 158 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java w/A/jobt6ln8G with job batch system ID: 159 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' d/Z/jobl2JWI0 with job batch system ID: 160 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java R/K/job7tfDIf with job batch system ID: 161 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' Y/s/jobVbtd4K with job batch system ID: 162 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java r/k/jobjvCVh5 with job batch system ID: 163 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' t/h/jobgjXKkY with job batch system ID: 164 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java L/e/jobNTwxqh with job batch system ID: 165 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 0/k/jobUvknBa with job batch system ID: 166 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java f/4/jobEbtATn with job batch system ID: 167 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' M/u/jobURhcHB with job batch system ID: 168 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java I/g/jobs6TCiU with job batch system ID: 169 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' P/o/jobPJlyEX with job batch system ID: 170 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 7/X/jobKLZV0f with job batch system ID: 171 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' T/c/jobdBC8lg with job batch system ID: 172 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 6/I/jobxOzyVb with job batch system ID: 173 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' Z/O/jobN7Kufm with job batch system ID: 174 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java q/4/jobJyUed6 with job batch system ID: 175 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' V/W/jobKqXzft with job batch system ID: 176 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java W/x/jobTlHBgR with job batch system ID: 177 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' K/n/job8856cU with job batch system ID: 178 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java N/G/jobEHs27n with job batch system ID: 179 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' P/c/jobALLNjN with job batch system ID: 180 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java e/G/jobFl9XGt with job batch system ID: 181 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' n/D/jobnLilyq with job batch system ID: 182 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java W/q/joboOmzNb with job batch system ID: 183 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' m/r/jobpJI8d_ with job batch system ID: 184 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java B/d/jobzHnh0J with job batch system ID: 185 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' X/r/jobmuBndZ with job batch system ID: 186 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java f/d/jobT_J0CW with job batch system ID: 187 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' k/R/jobv77z3t with job batch system ID: 188 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 1/c/job4W3O61 with job batch system ID: 189 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' R/V/jobErVRtI with job batch system ID: 190 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java Y/J/jobyYd6i0 with job batch system ID: 191 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' T/H/jobSYB0k9 with job batch system ID: 192 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 9/w/jobSWxe4t with job batch system ID: 193 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' a/7/jobblwyn4 with job batch system ID: 194 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java S/L/job0Z893I with job batch system ID: 195 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' i/m/jobZIeqph with job batch system ID: 196 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/P/jobyaE8ka with job batch system ID: 197 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' A/9/jobqV_MeW with job batch system ID: 198 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java b/i/jobTV9CXu with job batch system ID: 199 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' a/n/jobTHyWyr with job batch system ID: 200 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java t/f/job_hOkTM with job batch system ID: 201 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 6/B/jobYSQ2th with job batch system ID: 202 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java U/K/jobEFxMzb with job batch system ID: 203 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' f/J/jobyxo9PL with job batch system ID: 204 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java n/m/job_On1_9 with job batch system ID: 205 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 3/T/jobXlfr1x with job batch system ID: 206 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/o/jobQmVsjd with job batch system ID: 207 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' w/k/jobHrGAd9 with job batch system ID: 208 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java I/D/job3Ouhye with job batch system ID: 209 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' m/1/jobxhD1Cq with job batch system ID: 210 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 8/p/jobFxyfba with job batch system ID: 211 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' D/p/job2njUe4 with job batch system ID: 212 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/z/jobuzWT09 with job batch system ID: 213 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' w/e/jobFVFvKm with job batch system ID: 214 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 2/T/job_5uwQG with job batch system ID: 215 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' K/i/jobCeNgoZ with job batch system ID: 216 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java l/A/jobMgPIGU with job batch system ID: 217 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' L/f/jobThtcTp with job batch system ID: 218 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java l/p/job6ke8Ld with job batch system ID: 219 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 1/d/jobK7d0H7 with job batch system ID: 220 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 3/b/jobNLu6TB with job batch system ID: 221 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' C/4/jobSEZMFa with job batch system ID: 222 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java M/A/job7WqRTt with job batch system ID: 223 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 0/K/jobwUNVOS with job batch system ID: 224 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 3/M/jobjP4hsU with job batch system ID: 225 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' w/n/jobyvzEK1 with job batch system ID: 226 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java r/0/jobCMmNr6 with job batch system ID: 227 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' d/n/jobqI05IG with job batch system ID: 228 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java B/3/job49mb9z with job batch system ID: 229 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' H/B/job0Mfcat with job batch system ID: 230 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java w/z/job3ZD0aW with job batch system ID: 231 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' W/a/jobgWtDwM with job batch system ID: 232 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java H/g/jobFvIKlb with job batch system ID: 233 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' g/x/jobxb4AnI with job batch system ID: 234 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java v/M/jobZQLAUo with job batch system ID: 235 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' f/4/jobZFMhsh with job batch system ID: 236 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java o/y/jobj25UDF with job batch system ID: 237 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' v/j/jobqR3jK3 with job batch system ID: 238 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java w/6/jobg9gFLn with job batch system ID: 239 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 2/7/job9u_Jpm with job batch system ID: 240 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java P/R/jobC0tRZP with job batch system ID: 241 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 4/g/joboUj0co with job batch system ID: 242 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java m/f/jobrh_eem with job batch system ID: 243 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 7/Q/jobCm6qYw with job batch system ID: 244 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/V/job1VYzY3 with job batch system ID: 245 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 1/z/job1iHCsX with job batch system ID: 246 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 1/B/jobOA9JQy with job batch system ID: 247 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' B/r/jobyHcbVD with job batch system ID: 248 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java l/U/jobSxcJk3 with job batch system ID: 249 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' W/G/jobfP7c07 with job batch system ID: 250 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java I/n/jobesYK3q with job batch system ID: 251 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 3/D/jobkKbieV with job batch system ID: 252 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java V/K/jobAydIJR with job batch system ID: 253 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' O/l/jobqtCExO with job batch system ID: 254 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java H/D/jobXnXEul with job batch system ID: 255 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' t/d/jobvSW7Fl with job batch system ID: 256 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 8/Q/jobiIjL9B with job batch system ID: 257 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' r/T/jobHhOxdf with job batch system ID: 258 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java L/Q/jobjw97jS with job batch system ID: 259 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 3/v/jobbbMMkP with job batch system ID: 260 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java C/3/jobgijhZp with job batch system ID: 261 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' q/Q/jobVf439d with job batch system ID: 262 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java m/J/job7nhMI5 with job batch system ID: 263 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' o/N/job6KL2fv with job batch system ID: 264 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java N/F/jobmIP69q with job batch system ID: 265 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' L/c/jobPsrTkb with job batch system ID: 266 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 2/P/jobo7SzEX with job batch system ID: 267 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' w/7/jobQGvg9c with job batch system ID: 268 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java C/Y/jobkuesrq with job batch system ID: 269 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' S/l/jobNNYj8z with job batch system ID: 270 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java v/T/jobswJr4M with job batch system ID: 271 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' f/v/jobgpzcK4 with job batch system ID: 272 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 5/r/jobIkSWcZ with job batch system ID: 273 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' I/L/jobppqsmT with job batch system ID: 274 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java P/D/job6ZsWNI with job batch system ID: 275 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 4/b/job_zqrhn with job batch system ID: 276 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java x/6/jobR0Tao6 with job batch system ID: 277 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' i/D/jobM8SsKi with job batch system ID: 278 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 7/c/jobwoBcHK with job batch system ID: 279 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' N/1/jobH51yl1 with job batch system ID: 280 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java V/4/job0emUzB with job batch system ID: 281 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 6/A/jobN01ETV with job batch system ID: 282 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java t/f/jobtuRWNP with job batch system ID: 283 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 2/R/job3A688Y with job batch system ID: 284 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java D/2/jobRbWWJG with job batch system ID: 285 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 2/z/jobgdT6VE with job batch system ID: 286 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java P/b/job7p0ciZ with job batch system ID: 287 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' O/o/job6xhCIy with job batch system ID: 288 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java R/5/jobeTROyX with job batch system ID: 289 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' g/l/jobYmbJNE with job batch system ID: 290 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java h/i/job0dcwkx with job batch system ID: 291 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' c/D/jobmYDQ9o with job batch system ID: 292 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 7/X/jobK3Y45L with job batch system ID: 293 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' W/k/jobhcJIUn with job batch system ID: 294 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java H/s/job8yHzMV with job batch system ID: 295 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' d/3/jobAYjAKY with job batch system ID: 296 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 5/O/jobiXAAm7 with job batch system ID: 297 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' A/c/jobwPUyyl with job batch system ID: 298 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java C/e/jobYI7O49 with job batch system ID: 299 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' Y/P/job9EbCs1 with job batch system ID: 300 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/N/jobDblyr_
WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/N/jobDblyr_
WARNING:toil.leader:K/N/jobDblyr_    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:K/N/jobDblyr_    INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:K/N/jobDblyr_    Got workflow error
WARNING:toil.leader:K/N/jobDblyr_    Traceback (most recent call last):
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:K/N/jobDblyr_        job.run(runtime_context)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 388, in run
WARNING:toil.leader:K/N/jobDblyr_        os.makedirs(self.tmpdir)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:K/N/jobDblyr_        mkdir(name, mode)
WARNING:toil.leader:K/N/jobDblyr_    OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/cwltool_TMPDIR_LOCK_test_2/tmp'
WARNING:toil.leader:K/N/jobDblyr_    ERROR:cwltool:Got workflow error
WARNING:toil.leader:K/N/jobDblyr_    Traceback (most recent call last):
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:K/N/jobDblyr_        job.run(runtime_context)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 388, in run
WARNING:toil.leader:K/N/jobDblyr_        os.makedirs(self.tmpdir)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:K/N/jobDblyr_        mkdir(name, mode)
WARNING:toil.leader:K/N/jobDblyr_    OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/cwltool_TMPDIR_LOCK_test_2/tmp'
WARNING:toil.leader:K/N/jobDblyr_    WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job K/S/jobcVNwU1/g/tmp2hqSHH-_serialiseJob-stream used 352.13% (7.0 GB [7561955328B] used, 2.0 GB [2147483648B] requested) at the end of its run.
WARNING:toil.leader:K/N/jobDblyr_    Traceback (most recent call last):
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/worker.py", line 314, in workerScript
WARNING:toil.leader:K/N/jobDblyr_        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/job.py", line 1351, in _runner
WARNING:toil.leader:K/N/jobDblyr_        returnValues = self._run(jobGraph, fileStore)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/job.py", line 1296, in _run
WARNING:toil.leader:K/N/jobDblyr_        return self.run(fileStore)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/cwl/cwltoil.py", line 565, in run
WARNING:toil.leader:K/N/jobDblyr_        self.cwltool, cwljob, runtime_context, cwllogger)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 90, in execute
WARNING:toil.leader:K/N/jobDblyr_        self.run_jobs(process, job_order_object, logger, runtime_context)
WARNING:toil.leader:K/N/jobDblyr_      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 173, in run_jobs
WARNING:toil.leader:K/N/jobDblyr_        raise WorkflowException(Text(err))
WARNING:toil.leader:K/N/jobDblyr_    WorkflowException: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/cwltool_TMPDIR_LOCK_test_2/tmp'
WARNING:toil.leader:K/N/jobDblyr_    ERROR:toil.worker:Exiting the worker because of a failed job on host u22.cbio.private
WARNING:toil.leader:K/N/jobDblyr_    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/N/jobDblyr_ with ID K/N/jobDblyr_ to 2
WARNING:toil.leader:K/N/jobDblyr_    before lock
WARNING:toil.leader:K/N/jobDblyr_    lock acquired
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java K/N/jobDblyr_ with job batch system ID: 301 and cores: 1, disk: 2.0 G, and memory: 31.2 G
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' S/G/jobVdqxr9
WARNING:toil.leader:The job seems to have left a log file, indicating failure: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' S/G/jobVdqxr9
WARNING:toil.leader:S/G/jobVdqxr9    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:S/G/jobVdqxr9    INFO:toil:Running Toil version 3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be-dirty.
WARNING:toil.leader:S/G/jobVdqxr9    Got workflow error
WARNING:toil.leader:S/G/jobVdqxr9    Traceback (most recent call last):
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:S/G/jobVdqxr9        job.run(runtime_context)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 388, in run
WARNING:toil.leader:S/G/jobVdqxr9        os.makedirs(self.tmpdir)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:S/G/jobVdqxr9        mkdir(name, mode)
WARNING:toil.leader:S/G/jobVdqxr9    OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/cwltool_TMPDIR_LOCK_test_2/tmp'
WARNING:toil.leader:S/G/jobVdqxr9    ERROR:cwltool:Got workflow error
WARNING:toil.leader:S/G/jobVdqxr9    Traceback (most recent call last):
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 165, in run_jobs
WARNING:toil.leader:S/G/jobVdqxr9        job.run(runtime_context)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/job.py", line 388, in run
WARNING:toil.leader:S/G/jobVdqxr9        os.makedirs(self.tmpdir)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/os.py", line 157, in makedirs
WARNING:toil.leader:S/G/jobVdqxr9        mkdir(name, mode)
WARNING:toil.leader:S/G/jobVdqxr9    OSError: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/cwltool_TMPDIR_LOCK_test_2/tmp'
WARNING:toil.leader:S/G/jobVdqxr9    Traceback (most recent call last):
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/worker.py", line 314, in workerScript
WARNING:toil.leader:S/G/jobVdqxr9        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/job.py", line 1351, in _runner
WARNING:toil.leader:S/G/jobVdqxr9        returnValues = self._run(jobGraph, fileStore)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/job.py", line 1296, in _run
WARNING:toil.leader:S/G/jobVdqxr9        return self.run(fileStore)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/toil_ionox0-0.0.4-py2.7.egg/toil/cwl/cwltoil.py", line 565, in run
WARNING:toil.leader:S/G/jobVdqxr9        self.cwltool, cwljob, runtime_context, cwllogger)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 90, in execute
WARNING:toil.leader:S/G/jobVdqxr9        self.run_jobs(process, job_order_object, logger, runtime_context)
WARNING:toil.leader:S/G/jobVdqxr9      File "/home/johnsoni/virtualenvs/pipeline_0.1.0-rc2/lib/python2.7/site-packages/cwltool-1.0.20180820141117-py2.7.egg/cwltool/executors.py", line 173, in run_jobs
WARNING:toil.leader:S/G/jobVdqxr9        raise WorkflowException(Text(err))
WARNING:toil.leader:S/G/jobVdqxr9    WorkflowException: [Errno 17] File exists: '/ifs/work/bergerm1/Innovation/sandbox/ian/cwltool_TMPDIR_LOCK_test_2/tmp'
WARNING:toil.leader:S/G/jobVdqxr9    ERROR:toil.worker:Exiting the worker because of a failed job on host u25.cbio.private
WARNING:toil.leader:S/G/jobVdqxr9    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' S/G/jobVdqxr9 with ID S/G/jobVdqxr9 to 2
WARNING:toil.leader:S/G/jobVdqxr9    before lock
WARNING:toil.leader:S/G/jobVdqxr9    lock acquired
INFO:toil.leader:Issued job 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' S/G/jobVdqxr9 with job batch system ID: 302 and cores: 4, disk: 20.5 G, and memory: 31.2 G
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' d/A/jobWIqgxL
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' A/y/jobmMy6YI
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' q/Q/jobG4AHA3
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' p/O/job8HF6uu
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' H/T/jobq1C3oo
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' Y/J/jobQmDCHl
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:23:59: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job U/N/jobWnFJMP/g/tmp0T5x1B-_serialiseJob-stream used 352.18% (7.0 GB [7563069440B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java D/Y/job3KMWXK
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:24:40: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job X/9/jobV_8NL_/g/tmpLE48KK-_serialiseJob-stream used 352.19% (7.0 GB [7563167744B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java t/9/jobaOao7z
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' X/T/jobEad7wW
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' L/F/jobUtKm2w
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' m/B/jobQq0Q6k
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' 9/V/jobvA_bxX
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' i/c/jobUsHqzl
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:26:24: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 8/8/jobO5S0Ba/g/tmpk6Z4Xr-_serialiseJob-stream used 352.11% (7.0 GB [7561463808B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java T/n/jobPGF6CK
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' W/P/jobqWOH7V
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' z/g/jobAEkumf
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/vardict/vardict_paired.cwl' I/9/jobj7nLzp
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:28:09: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job w/m/jobsL4SwU/g/tmp9uGgVp-_serialiseJob-stream used 352.19% (7.0 GB [7563167744B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java B/s/jobLUjpo3
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:29:25: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job p/3/joby9kV1i/g/tmpgcvRiK-_serialiseJob-stream used 352.19% (7.0 GB [7563167744B] used, 2.0 GB [2147483648B] requested) at the end of its run.
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:29:32: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job i/2/jobZvwI1e/g/tmpYuaAUE-_serialiseJob-stream used 352.20% (7.0 GB [7563429888B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java U/Y/job2If_0v
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java M/s/jobuiMkfY
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:29:41: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job C/8/jobru1ofo/g/tmpCzHFaG-_serialiseJob-stream used 352.18% (7.0 GB [7562905600B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java 4/E/jobE9BdRO
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:30:03: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job o/J/jobOjdTb3/g/tmpx67nLP-_serialiseJob-stream used 352.19% (7.0 GB [7563134976B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java s/E/job96aiz8
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:30:49: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job I/O/jobvx4cbf/g/tmpfy2qCa-_serialiseJob-stream used 352.21% (7.0 GB [7563659264B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java Y/H/jobg5UM6H
WARNING:toil.statsAndLogging:Got message from job at time 01-18-2019 19:31:19: Job used more disk than requested. Consider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job U/s/job9qdltm/g/tmpiH0IR8-_serialiseJob-stream used 352.21% (7.0 GB [7563724800B] used, 2.0 GB [2147483648B] requested) at the end of its run.
INFO:toil.leader:Job ended successfully: 'file:///home/johnsoni/pipeline_0.1.0-rc2/ACCESS-Pipeline/cwl_tools/mutect/mutect.cwl' /opt/common/CentOS_6/java/jdk1.7.0_75/bin/java S/E/jobxyanUX
abhishekgahlot2 commented 1 month ago

I still feel this happening for me on version: /Library/Frameworks/Python.framework/Versions/3.10/bin/cwltool 3.1.20240508115724

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cwltool/executors.py", line 331, in _runner
    job.run(runtime_context, TMPDIR_LOCK)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cwltool/job.py", line 575, in run
    stage_files(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/cwltool/process.py", line 277, in stage_files
    os.symlink(entry.resolved, entry.target)
FileExistsError: [Errno 17] File exists: '/Users/abhishek/dir/unit.py' -> '/private/tmp/docker_tmp4w_jc0v3/unit.py'

Only with --parallel flag sequential execution still works fine.

abhishekgahlot2 commented 1 month ago

I also believe it's related to the number of cores on my machine. I haven't checked the code in detail, but I will give it a try. However, if I have 9 identical nodes and a 10-core machine, it works fine.

Suppose the number of identical nodes exceeds the core count minus one; it gives this error.

abhishekgahlot2 commented 1 month ago

Looking deeper into codebase. I see

something like this should help in stage_files function

if unique_id:
            new_target = os.path.join(os.path.dirname(entry.target), f"{unique_id}/{os.path.basename(entry.target)}")
            pathmapper.update(key, entry.resolved, new_target, entry.type, entry.staged) 
        unique_id = f"{self.name}_{datetime.datetime.now().timestamp()}"

        stage_files(
            self.pathmapper,
            ignore_writable=True,
            symlink=False,
            secret_store=runtimeContext.secret_store,
            unique_id=unique_id
        )
        if self.generatemapper is not None:
            stage_files(
                self.generatemapper,
                ignore_writable=self.inplace_update,
                symlink=True,
                secret_store=runtimeContext.secret_store,
                unique_id=unique_id
            )
            relink_initialworkdir(
                self.generatemapper,
                self.outdir,
                self.builder.outdir,
                inplace_update=self.inplace_update,
            )

so the problem that is happening is its reusing the same directory created for the previous job and in entry.target causing it to fall.