search

FredHutch / batchman

Batchman - Nextflow on Fargate x AWS Batch
6 stars 1 forks source link

variable substitution in template? #45

Open nkrumm opened 4 years ago

nkrumm commented 4 years ago

In GitLab by @nhoffman on Dec 21, 2019, 11:21

Thinking about how to submit jobs for production pipelines for which there are file hierarchy conventions for input and output data... I just submitted a dada2-nf job with the following inputs:

s3://uwlm-personal/ngh2/dada2-nf/795/output
s3://uwlm-personal/ngh2/dada2-nf/795/work
s3://uwlm-personal/ngh2/dada2-nf/795/fastq-list.txt
s3://uwlm-personal/ngh2/dada2-nf/795/sample-information.csv

where "795" is a miseq run number. Would be awesome to be able to simply specify "795" - I'm not sure at which level parameter substitution would be best implemented.

nkrumm commented 4 years ago

In GitLab by @nhoffman on Jan 3, 2020, 11:22

Options:

  1. consider light parameter substitution in the json file
  2. open nextflow PR to have the params.json file be parsed before the config file is evaluated.
  3. optional "nextflow cmd line args" field (but this would not solve the param substitution case either)
nkrumm commented 4 years ago

In GitLab by @nkrumm on Jan 3, 2020, 23:51

After discussion, this will primarily be handled by a template.json with a selectable option for "simple" or "custom" parameter specification. The pipeline will then interpret this and correctly specify workflow inputs using either mode.

As the workDir is not a parameter (and it is challenging/impossible to specify workDir via params, we will provide light parameter/variable substitution within the batchman api to support this use case.