search

mskcc / ACCESS-Pipeline

cfDNA Sequencing Pipeline with UMI
MIT License
10 stars 3 forks source link

need to keep ALL lsf logging #199

Open ionox0 opened 5 years ago

ionox0 commented 5 years ago

Need to make this change to lsf.py

But also, need to put all the logs in their own directory in a logical place such as the --writeLogs folder

Go off of @dkennetz recent change to Toil (the one with the self inside of a staticmethod issue)

Also need to get the %J thing to work 

  inputs_5.yaml¶                                            |   178             bsubline.extend(bsubCpu)¶
  inputs_6.yaml¶                                            |   179             #stdoutfile = self.boss.formatStdOutErrPath(jobID, 'lsf', '%J', 'std_output')¶
  inputs_7.yaml¶                                            |   180             #stderrfile = self.boss.formatStdOutErrPath(jobID, 'lsf', '%J', 'std_error')¶
  inputs_8.yaml¶                                            |   181             stdout_file = str(jobID) + '_lsf_' + '%J' + '_out'¶
  TD-dmplib-1_title_file.txt¶                               |   182             stderr_file = str(jobID) + '_lsf_' + '%J' + '_err'¶
  TD-dmplib-2_title_file.txt¶                               |   183             bsubline.extend(['-o', stdout_file, '-e', stderr_file])¶
ionox0 commented 5 years ago 
WARNING:toil.leader:y/V/job0bQJlp    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
WARNING:toil.leader:y/V/job0bQJlp    INFO:toil:Running Toil version 3.19.0-16e477c322f5cbd8bc14bfe726f895540b1aa3e6.
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:toil:Configuration: {'maxLocalJobs': 40, 'rescueJobsFrequency': 3600, 'logLevel': 'DEBUG', 'minNodes': None, 'maxMemory': 9223372036854775807, 'jobStore': 'file:/home/johnsoni/juno_ACCESS/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/jobstore-d1c01e8e-aa30-11e9-98d4-ec0d9a88a232', 'linkImports': True, 'manualMemArgs': False, 'forceDockerAppliance': False, 'nodeOptions': None, 'nodeTypes': [], 'servicePollingInterval': 60, 'workDir': '/home/johnsoni/juno_ACCESS/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp', 'stats': True, 'disableCaching': True, 'maxPreemptableServiceJobs': 9223372036854775807, 'environment': {}, 'parasolMaxBatches': 10000, 'cleanWorkDir': 'onSuccess', 'disableChaining': False, 'maxCores': 9223372036854775807, 'targetTime': 1800, 'sseKey': None, 'cwl': False, 'scaleInterval': 60, 'deadlockWait': 60, 'defaultPreemptable': False, 'maxJobDuration': 9223372036854775807, 'clusterStats': None, 'defaultCores': 1, 'cseKey': None, 'betaInertia': 0.1, 'metrics': False, 'maxNodes': [10], 'scale': 1, 'writeLogs': '/home/johnsoni/juno_ACCESS/5500-FY/5500-FY_TD-dmplib-2-1.1.13/log', 'runCwlInternalJobsOnWorkers': False, 'disableAutoDeployment': False, 'badWorker': 0.0, 'defaultDisk': 10737418240, 'mesosMasterAddress': '10.0.201.24:5050', 'restart': True, 'useAsync': True, 'preemptableCompensation': 0.0, 'parasolCommand': 'parasol', 'workflowID': '234e5d3f-4ec8-470b-a343-e9b4a12e1ed3', 'maxServiceJobs': 9223372036854775807, 'readGlobalFileMutableByDefault': False, 'badWorkerFailInterval': 0.01, 'statePollingWait': 1, 'debugWorker': False, 'maxLogFileSize': 20000000000, 'defaultMemory': 10737418240, 'workflowAttemptNumber': 4, 'maxDisk': 9223372036854775807, 'clean': 'never', 'provisioner': None, 'batchSystem': 'lsf', 'retryCount': 2, 'writeLogsGzip': None, 'nodeStorage': 50}
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:toil.worker:Parsed job wrapper
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:toil.worker:Got a command to run: _toil s/t/job1Pw5kF/g/tmpmejaCP-_serialiseJob-stream /home/johnsoni/virtualenvs/pipeline_1.1.14/lib/python2.7/site-packages toil.cwl.cwltoil True
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:toil.job:Loading user module ModuleDescriptor(dirPath='/home/johnsoni/virtualenvs/pipeline_1.1.14/lib/python2.7/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=True).
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:rdflib:RDFLib Version: 4.2.2
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/home/johnsoni/juno_ACCESS/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/jobstore-d1c01e8e-aa30-11e9-98d4-ec0d9a88a232'.
WARNING:toil.leader:y/V/job0bQJlp    [job plots_module.cwl] /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpsL3Tnn/f/e/out_tmpdirFDxXXM$ plots_module.r \
WARNING:toil.leader:y/V/job0bQJlp        --family_sizes_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg825cdad0-68f0-4c99-8f8f-e2c9f3661892/family-sizes.txt \
WARNING:toil.leader:y/V/job0bQJlp        --family_types_A_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg6e83312f-f776-4e08-9c8c-a3d7ccfd22c3/family-types-A.txt \
WARNING:toil.leader:y/V/job0bQJlp        --family_types_B_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg17db5f4a-e175-472f-ba50-381c08240eb5/family-types-B.txt \
WARNING:toil.leader:y/V/job0bQJlp        --inputs_yaml_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stge77efc8f-d57e-4088-91ec-974b3053d049/inputs_2.yaml \
WARNING:toil.leader:y/V/job0bQJlp        --tables_output_dir \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg7ab10472-dfcb-4ba4-880c-a49fa8093a8b/aggregate_tables \
WARNING:toil.leader:y/V/job0bQJlp        --title_file_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stge492bdbf-9dda-41f2-afdd-ffddc3bf8429/TD-dmplib-2_title_file.txt
WARNING:toil.leader:y/V/job0bQJlp    INFO:cwltool:[job plots_module.cwl] /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpsL3Tnn/f/e/out_tmpdirFDxXXM$ plots_module.r \
WARNING:toil.leader:y/V/job0bQJlp        --family_sizes_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg825cdad0-68f0-4c99-8f8f-e2c9f3661892/family-sizes.txt \
WARNING:toil.leader:y/V/job0bQJlp        --family_types_A_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg6e83312f-f776-4e08-9c8c-a3d7ccfd22c3/family-types-A.txt \
WARNING:toil.leader:y/V/job0bQJlp        --family_types_B_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg17db5f4a-e175-472f-ba50-381c08240eb5/family-types-B.txt \
WARNING:toil.leader:y/V/job0bQJlp        --inputs_yaml_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stge77efc8f-d57e-4088-91ec-974b3053d049/inputs_2.yaml \
WARNING:toil.leader:y/V/job0bQJlp        --tables_output_dir \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stg7ab10472-dfcb-4ba4-880c-a49fa8093a8b/aggregate_tables \
WARNING:toil.leader:y/V/job0bQJlp        --title_file_path \
WARNING:toil.leader:y/V/job0bQJlp        /juno/work/bergerm1/MSK-ACCESS/ACCESS-Projects/5500-FY/5500-FY_TD-dmplib-2-1.1.13/tmp/tmpbsKTIS/stge492bdbf-9dda-41f2-afdd-ffddc3bf8429/TD-dmplib-2_title_file.txt
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp    Attaching package: ‘reshape2’
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp    The following object is masked from ‘package:tidyr’:
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp        smiths
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp    Attaching package: ‘data.table’
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp    The following objects are masked from ‘package:reshape2’:
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp        dcast, melt
WARNING:toil.leader:y/V/job0bQJlp
WARNING:toil.leader:y/V/job0bQJlp    Error in scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  :
WARNING:toil.leader:y/V/job0bQJlp      line 1 did not have 16 elements
WARNING:toil.leader:y/V/job0bQJlp    Calls: main -> read.table -> scan
WARNING:toil.leader:y/V/job0bQJlp    In addition: Warning message:
WARNING:toil.leader:y/V/job0bQJlp    In yaml.load(readLines(con), error.label = error.label, ...) :
WARNING:toil.leader:y/V/job0bQJlp      NAs introduced by coercion: . is not a real
WARNING:toil.leader:y/V/job0bQJlp    4: (function ()
WARNING:toil.leader:y/V/job0bQJlp       traceback(2))()
WARNING:toil.leader:y/V/job0bQJlp    3: scan(file = file, what = what, sep = sep, quote = quote, dec = dec,
WARNING:toil.leader:y/V/job0bQJlp           nmax = nrows, skip = 0, na.strings = na.strings, quiet = TRUE,
WARNING:toil.leader:y/V/job0bQJlp           fill = fill, strip.white = strip.white, blank.lines.skip = blank.lines.skip,
WARNING:toil.leader:y/V/job0bQJlp           multi.line = FALSE, comment.char = comment.char, allowEscapes = allowEscapes,
WARNING:toil.leader:y/V/job0bQJlp           flush = flush, encoding = encoding, skipNul = skipNul)
WARNING:toil.leader:y/V/job0bQJlp    2: read.table(title_file_path, sep = "\t", header = TRUE, colClasses = c(SEX = "character"))
WARNING:toil.leader:y/V/job0bQJlp    1: main()
WARNING:toil.leader:y/V/job0bQJlp    Warning message:
WARNING:toil.leader:y/V/job0bQJlp    In yaml.load(readLines(con), error.label = error.label,  ... :
WARNING:toil.leader:y/V/job0bQJlp      NAs introduced by coercion: . is not a real
WARNING:toil.leader:y/V/job0bQJlp    [job plots_module.cwl] Job error:
WARNING:toil.leader:y/V/job0bQJlp    Error collecting output for parameter 'align_rate':
WARNING:toil.leader:y/V/job0bQJlp    :1:1: Did not find output file with glob pattern: '[u'align_rate.pdf']'
WARNING:toil.leader:y/V/job0bQJlp    ERROR:cwltool:[job plots_module.cwl] Job error:
WARNING:toil.leader:y/V/job0bQJlp    Error collecting output for parameter 'align_rate':
WARNING:toil.leader:y/V/job0bQJlp    :1:1: Did not find output file with glob pattern: '[u'align_rate.pdf']'
WARNING:toil.leader:y/V/job0bQJlp    [job plots_module.cwl] completed permanentFail
WARNING:toil.leader:y/V/job0bQJlp    WARNING:cwltool:[job plots_module.cwl] completed permanentFail
WARNING:toil.leader:y/V/job0bQJlp    DEBUG:toil.fileStore:LOG-TO-MASTER: Job s/t/job1Pw5kF/g/tmpmejaCP-_serialiseJob-stream used 0.55% (11.3 MB [11822592B] used, 2.0 GB [2147483648B] requested) at the end of its run.
WARNING:toil.leader:y/V/job0bQJlp    Traceback (most recent call last):
WARNING:toil.leader:y/V/job0bQJlp      File "/home/johnsoni/virtualenvs/pipeline_1.1.13/lib/python2.7/site-packages/toil/worker.py", line 324, in workerScript
WARNING:toil.leader:y/V/job0bQJlp        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
WARNING:toil.leader:y/V/job0bQJlp      File "/home/johnsoni/virtualenvs/pipeline_1.1.13/lib/python2.7/site-packages/toil/job.py", line 1351, in _runner
WARNING:toil.leader:y/V/job0bQJlp        returnValues = self._run(jobGraph, fileStore)
WARNING:toil.leader:y/V/job0bQJlp      File "/home/johnsoni/virtualenvs/pipeline_1.1.13/lib/python2.7/site-packages/toil/job.py", line 1296, in _run
WARNING:toil.leader:y/V/job0bQJlp        return self.run(fileStore)
WARNING:toil.leader:y/V/job0bQJlp      File "/home/johnsoni/virtualenvs/pipeline_1.1.13/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 606, in run
WARNING:toil.leader:y/V/job0bQJlp        raise cwltool.errors.WorkflowException(status)
WARNING:toil.leader:y/V/job0bQJlp    WorkflowException: permanentFail
WARNING:toil.leader:y/V/job0bQJlp    ERROR:toil.worker:Exiting the worker because of a failed job on host jx05
WARNING:toil.leader:y/V/job0bQJlp    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/johnsoni/pipeline_1.1.14/ACCESS-Pipeline/cwl_tools/python/plots_module.cwl' plots_module.r y/V/job0bQJlp with ID y/V/job0bQJlp to 2