search

cooperative-computing-lab / makeflow-examples

Example workflows for the Makeflow workflow system.
33 stars 18 forks source link

setting walltime for each command #35

Closed stemangiola closed 4 years ago

stemangiola commented 4 years ago

Hello, thanks for makeflow.

I have a script like the one below. What should I use insteal of TIME=xxx for custom walltimes (I'm using slurm)?

Thank a lot!

CATEGORY=create_input
MEMORY=10024
CORES=2
TIME=5
dev/armet_ACC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ ACC.tcga.harmonized.counts.allgenes.rds
dev/armet_BLCA_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ BLCA.tcga.harmonized.counts.allgenes.rds
dev/armet_BRCA_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ BRCA.tcga.harmonized.counts.allgenes.rds
dev/armet_CESC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ CESC.tcga.harmonized.counts.allgenes.rds
dev/armet_CHOL_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ CHOL.tcga.harmonized.counts.allgenes.rds
dev/armet_COAD_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ COAD.tcga.harmonized.counts.allgenes.rds
dev/armet_DLBC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ DLBC.tcga.harmonized.counts.allgenes.rds
dev/armet_ESCA_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ ESCA.tcga.harmonized.counts.allgenes.rds
dev/armet_GBM_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ GBM.tcga.harmonized.counts.allgenes.rds
dev/armet_HNSC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ HNSC.tcga.harmonized.counts.allgenes.rds
dev/armet_KICH_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ KICH.tcga.harmonized.counts.allgenes.rds
dev/armet_KIRC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ KIRC.tcga.harmonized.counts.allgenes.rds
dev/armet_KIRP_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ KIRP.tcga.harmonized.counts.allgenes.rds
dev/armet_LGG_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ LGG.tcga.harmonized.counts.allgenes.rds
dev/armet_LIHC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ LIHC.tcga.harmonized.counts.allgenes.rds
dev/armet_LUAD_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ LUAD.tcga.harmonized.counts.allgenes.rds
dev/armet_LUSC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ LUSC.tcga.harmonized.counts.allgenes.rds
dev/armet_MESO_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ MESO.tcga.harmonized.counts.allgenes.rds
dev/armet_OV_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ OV.tcga.harmonized.counts.allgenes.rds
dev/armet_PAAD_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ PAAD.tcga.harmonized.counts.allgenes.rds
dev/armet_PCPG_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ PCPG.tcga.harmonized.counts.allgenes.rds
dev/armet_PRAD_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ PRAD.tcga.harmonized.counts.allgenes.rds
dev/armet_READ_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ READ.tcga.harmonized.counts.allgenes.rds
dev/armet_SARC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ SARC.tcga.harmonized.counts.allgenes.rds
dev/armet_SKCM_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ SKCM.tcga.harmonized.counts.allgenes.rds
dev/armet_STAD_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ STAD.tcga.harmonized.counts.allgenes.rds
dev/armet_TGCT_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ TGCT.tcga.harmonized.counts.allgenes.rds
dev/armet_THCA_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ THCA.tcga.harmonized.counts.allgenes.rds
dev/armet_THYM_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ THYM.tcga.harmonized.counts.allgenes.rds
dev/armet_UCEC_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ UCEC.tcga.harmonized.counts.allgenes.rds
dev/armet_UCS_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ UCS.tcga.harmonized.counts.allgenes.rds
dev/armet_UVM_input.rds:
    Rscript dev/TCGA_makeflow_pipeline/create_input.R ~/unix3XX/PhD/deconvolution/ UVM.tcga.harmonized.counts.allgenes.rds
CATEGORY=lv1
MEMORY=30024
CORES=12
WALLTIME=8
dev/armet_ACC_lv_1.rds: dev/armet_ACC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_ACC_input.rds 0
dev/armet_BLCA_lv_1.rds: dev/armet_BLCA_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_BLCA_input.rds 0
dev/armet_BRCA_lv_1.rds: dev/armet_BRCA_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_BRCA_input.rds 0
dev/armet_CESC_lv_1.rds: dev/armet_CESC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_CESC_input.rds 0
dev/armet_CHOL_lv_1.rds: dev/armet_CHOL_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_CHOL_input.rds 0
dev/armet_COAD_lv_1.rds: dev/armet_COAD_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_COAD_input.rds 0
dev/armet_DLBC_lv_1.rds: dev/armet_DLBC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_DLBC_input.rds 0
dev/armet_ESCA_lv_1.rds: dev/armet_ESCA_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_ESCA_input.rds 0
dev/armet_GBM_lv_1.rds: dev/armet_GBM_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_GBM_input.rds 0
dev/armet_HNSC_lv_1.rds: dev/armet_HNSC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_HNSC_input.rds 0
dev/armet_KICH_lv_1.rds: dev/armet_KICH_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_KICH_input.rds 0
dev/armet_KIRC_lv_1.rds: dev/armet_KIRC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_KIRC_input.rds 0
dev/armet_KIRP_lv_1.rds: dev/armet_KIRP_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_KIRP_input.rds 0
dev/armet_LGG_lv_1.rds: dev/armet_LGG_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_LGG_input.rds 0
dev/armet_LIHC_lv_1.rds: dev/armet_LIHC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_LIHC_input.rds 0
dev/armet_LUAD_lv_1.rds: dev/armet_LUAD_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_LUAD_input.rds 0
dev/armet_LUSC_lv_1.rds: dev/armet_LUSC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_LUSC_input.rds 0
dev/armet_MESO_lv_1.rds: dev/armet_MESO_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_MESO_input.rds 0
dev/armet_OV_lv_1.rds: dev/armet_OV_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_OV_input.rds 0
dev/armet_PAAD_lv_1.rds: dev/armet_PAAD_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_PAAD_input.rds 0
dev/armet_PCPG_lv_1.rds: dev/armet_PCPG_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_PCPG_input.rds 0
dev/armet_PRAD_lv_1.rds: dev/armet_PRAD_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_PRAD_input.rds 0
dev/armet_READ_lv_1.rds: dev/armet_READ_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_READ_input.rds 0
dev/armet_SARC_lv_1.rds: dev/armet_SARC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_SARC_input.rds 0
dev/armet_SKCM_lv_1.rds: dev/armet_SKCM_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_SKCM_input.rds 0
dev/armet_STAD_lv_1.rds: dev/armet_STAD_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_STAD_input.rds 0
dev/armet_TGCT_lv_1.rds: dev/armet_TGCT_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_TGCT_input.rds 0
dev/armet_THCA_lv_1.rds: dev/armet_THCA_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_THCA_input.rds 0
dev/armet_THYM_lv_1.rds: dev/armet_THYM_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_THYM_input.rds 0
dev/armet_UCEC_lv_1.rds: dev/armet_UCEC_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_UCEC_input.rds 0
dev/armet_UCS_lv_1.rds: dev/armet_UCS_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_UCS_input.rds 0
dev/armet_UVM_lv_1.rds: dev/armet_UVM_input.rds
    Rscript dev/TCGA_makeflow_pipeline/infer_lv1.R dev/armet_UVM_input.rds 0
btovar commented 4 years ago

Currently Makeflow does not support that, but I have opened this issue:

https://github.com/cooperative-computing-lab/cctools/issues/2366

dthain commented 4 years ago

Hello @stemangiola - I want to make sure we clearly understand your use case before we write any code. Is your goal to put an upper limit on the run-time of each task, because some of them may hang indefinitely? Or is there something else going on?

stemangiola commented 4 years ago

Hello, often setting accurate walltime can favor backfill of jobs around bigger jobs. Different steps in the workflow might need different walltimes. Setting up a general walltime is not the best thing.

dthain commented 4 years ago

Ok, so the point here is that you want that walltime conveyed to the scheduler, and not that you have a desire to kill the job after a certain point, right?

(Although the scheduler might end up doing that if you exceed the walltime.)

On Thu, Aug 20, 2020 at 10:23 AM Stefano Mangiola notifications@github.com wrote:

Hello, often setting accurate walltime can favor backfill of jobs around bigger jobs. Different steps in the workflow might need different walltimes. Setting up a general walltime is not the best thing.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cooperative-computing-lab/makeflow-examples/issues/35#issuecomment-677696963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVADESNSQU6S3XYCBMVY63SBUWUVANCNFSM4QFTTIBQ .

stemangiola commented 4 years ago

Yes,

I would like setting wall-time in the same way I set CPU or MEMORY.

btovar commented 4 years ago

Sounds good, I should have a pr shortly.

btovar commented 4 years ago

Fixed with https://github.com/cooperative-computing-lab/cctools/pull/2372