The existing convention in dagster-meltano is to pass the meltano ENV vars to the op through the run config. This requires passing secrets either through the UI or from environment variables set on the host machine. While this pattern is fine for most situations, I pass the ENV vars to my ops through a different method to cut down on the number of unused secrets sitting in my docker container (security concern).
I make calls to AWS to get my secrets as part of the dagster job. My request is that a method be provided for passing ENV vars from a previous op to the meltano_run_op. My guess is that precedence would be best given to those variables coming from the run time config rather than from the in-job/op-to-op variables, but that could potentially be debated.
I suppose if someone can show me an alternative way of getting secrets from AWS and passing it to the run time configuration, I'll close this issue. The only way I can find to do that is to put it in the config of a job definition. When i do this, however, it exposes the secrets in the UI when I open the Launchpad which is also a security concern. So I want to avoid that as well.
The existing convention in
dagster-meltanois to pass the meltano ENV vars to the op through the run config. This requires passing secrets either through the UI or from environment variables set on the host machine. While this pattern is fine for most situations, I pass the ENV vars to my ops through a different method to cut down on the number of unused secrets sitting in my docker container (security concern).I make calls to AWS to get my secrets as part of the dagster job. My request is that a method be provided for passing ENV vars from a previous op to the
meltano_run_op. My guess is that precedence would be best given to those variables coming from the run time config rather than from the in-job/op-to-op variables, but that could potentially be debated.I suppose if someone can show me an alternative way of getting secrets from AWS and passing it to the run time configuration, I'll close this issue. The only way I can find to do that is to put it in the
configof a job definition. When i do this, however, it exposes the secrets in the UI when I open the Launchpad which is also a security concern. So I want to avoid that as well.