yuch7 / cwlexec

A new open source tool to run CWL workflows on LSF
Other
36 stars 8 forks source link

updated code to accept processors -n option in the LSF config file #49

Closed drkennetz closed 5 years ago

drkennetz commented 5 years ago

This will allow users to pass the "processors" : "4" option to an lsf.conf file. An example conf for bwa:

[dkennetz@nodem103 prod cwlexec_test]$ more bwa-backtrack-LSF.json
{
    "queue": "standard",
    "project": "CWL_test",
    "res_req": "span[hosts=1]",
    "steps": {
        "bwaAln1": {
            "processors": "4",
            "res_req": "rusage[mem=6000]"
            },
        "bwaAln2": {
            "processors": "4",
            "res_req": "rusage[mem=6000]"
            },
        "bwaSampe": {
            "res_req": "rusage[mem=25000]"
            },
        "samtoolsView": {
            "res_req": "rusage[mem=10000]"
            },
        "picardSortSam": {
            "res_req": "rusage[mem=50000]"
            },
        "picardMarkDuplicates": {
            "res_req": "rusage[mem=50000]"
            },
        "samtoolsFlagstat": {
            "res_req": "rusage[mem=1000]"
            }
    }
}

Which will distribute jobs to one host with a multiple processors:

JOBID      USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
11111111   dkennet RUN   standard   nodem          4*node           *tq.gz.sai Jun 24 14:17
11111111   dkennet RUN   standard   nodem          4*node           *tq.gz.sai Jun 24 14:17