search

aws / amazon-genomics-cli

https://aws.github.io/amazon-genomics-cli/
Apache License 2.0
146 stars 82 forks source link

Setting environment variables in workflow engine container #396

Open mlin opened 2 years ago

mlin commented 2 years ago

miniwdl allows any of its configuration options to be overridden by environment variables (details). This is often a more-convenient way to tune them compared to editing a configuration file (also supported), especially in a cloud setting where the engine process will be running remotely. The existing engineOptions in the workflow manifest doesn't suffice because some of the options we'll be interested in tuning (e.g. AWS request rate limiters) are a little too obscure to be exposed through dedicated command-line options.

A cursory search of the Nextflow and Snakemake docs also suggest it may be useful to set environment variables for them.

Practically, the project YAML engine definition could have a sub-section with a map of environment variables to set in the engine container. For the engines using the head process model, these perhaps could be further overridden by some engineEnvironment map alongside engineOptions in the workflow manifest.

github-actions[bot] commented 1 year ago

Greetings! Sorry to say but this is a very old issue that is probably not getting as much attention as it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to open a new one.