search

quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
41 stars 17 forks source link

Alter meltano run command in run time config #23

Closed jlloyd-widen closed 1 year ago

jlloyd-widen commented 1 year ago

I find myself, not infrequently, in need of altering the meltano run command when in the production environment in order to add the --log-level=debug flag or to just run the transform pipeline rather than the tap to target extract and load. Under the current feature set it is impossible to alter the meltano run command "on the fly" like this. I'd have to alter the actual python code to do so.

It would be nice to be able to add a run time config in the Launchpad that would overwrite the meltano run command for that specific run.

JulesHuisman commented 1 year ago

A agree a functionality like this would be a nice addition. For now you can use the MELTANO_CLI_LOG_LEVEL environment variable to set the log level if you want.