New task: set variable dynamically in run

environmental-forecasting / model-ensembler

Model Ensemble tool for batch workflows on HPCs

https://pypi.org/project/model-ensembler/

MIT License

13 stars 1 forks source link

New task: set variable dynamically in run #21

Closed JimCircadian closed 2 years ago

JimCircadian commented 3 years ago

This relates superficially to #13 , but it'd be wonderful to have a _assignvar? task that can pick up files or values from various sources (filesystem or commands are the initial two that spring to mind given the coverage at the moment) and assign it to variables during the run.

This could make the context hierarchy slightly confusing as a developer, but actually really powerful as a user.

JimCircadian commented 2 years ago

The scope for this is fundamentally wrong, transit or communication of data needs to be defined through the workflow. As templating happens prior to pre_run, there's not a lot of point in doing this especially as runs themselves could pick up products from previous commands.