search

Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
142 stars 79 forks source link

Allow specifying credentials via parameters #38

Open MasterOdin opened 2 years ago

MasterOdin commented 2 years ago

Via the README#Credentials, this library only supports loading credentials through configured AWS CLI credentials. Would you support adding in the ability to configure the credentials (access key, secret key) as parameters to the connector? Looking at the underlying PyAthena library, it already supports it (see here), so it would just be amending this library to allow parsing it in / passing it along I think. Happy to look to put up a PR for this, but want to confirm if it would be merged before committing to it.

Dandandan commented 2 years ago

It not necessarily uses credential files, but I believe you can also use environment variables (I switch profiles this way).

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

You can bind those, like:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=....

That would be my suggested way, as otherwise you would need to put the secrets in a config file.

Would this work for you?

Dandandan commented 2 years ago

I think this could use some docs on how to use it

MasterOdin commented 2 years ago

While I think environment variables would work as we execute each dbt rpc instance in its own process, it's a bit annoying to use for our implementation. We would end up probably still sticking those variables into the profiles.yml file (as we want to store all DB specific stuff for all adapters in the file), and then parse it out into environment variables during runtime. Of course, would then be nicer to just use the profiles.yml only to avoid that.

Dandandan commented 2 years ago

While I think environment variables would work as we execute each dbt rpc instance in its own process, it's a bit annoying to use for our implementation. We would end up probably still sticking those variables into the profiles.yml file (as we want to store all DB specific stuff for all adapters in the file), and then parse it out into environment variables during runtime. Of course, would then be nicer to just use the profiles.yml only to avoid that.

Makes sense to me 👍 just wanted to know whether the alternatives were clear.