search

quantile-development / dagster-meltano

A Dagster plugin that allows you to run Meltano in Dagster
MIT License
42 stars 17 forks source link

overwriteable MELTANO_CLI_LOG_CONFIG environment variable #34

Closed resulyrt93 closed 1 year ago

resulyrt93 commented 1 year ago

We want to pass a new logging configuration file path to meltano resource through environment variable with that command export MELTANO_CLI_LOG_CONFIG=/<my-path>/logging.conf, not via run configuration. But when i try this, it doesn't pass to meltano environment since in package MELTANO_CLI_LOG_CONFIG envvar is overwritten. I think here should be like following to allow it.

return {
        "MELTANO_CLI_LOG_CONFIG": str(Path(__file__).parent / "logging.yaml"),
        "DBT_USE_COLORS": "false",
        "NO_COLOR": "1",
        **os.environ.copy()
}

Is there any reason to overwrite that i can't notice?

JulesHuisman commented 1 year ago

No that is totally fine!

The idea behind it was to give users the option to have a MELTANO_CLI_LOG_CONFIG when they run meltano using the cli, and then overwrite this in the package. But I think that is quite a rare occurrence, your use case is probably more applicable.

If you make a PR I can deploy the changes.

resulyrt93 commented 1 year ago

Thanks, I created a PR for it. It's waiting your approve. Also i can't attach my PR to this issue. Can you do that?

https://github.com/quantile-development/dagster-meltano/pull/35