search

Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
140 stars 79 forks source link

How to pass IAM credentials? #61

Closed Antauri closed 2 years ago

Antauri commented 2 years ago

Is there any possibility for dbt/dbt_athena to run using IAM instead of the aws_profile_name settings (which requires access/secret key). The dbt/dbt_athena we are running is inside a dagster @op with proper kube2iam annotations. The invocation of dbt however using context.resources.dbt.run (vars = payload) from dagster doesn't seem to take the IAM role associated (we are sure the role works as we've tested it directly with PyAthena queries and simple s3 actions at the Dagster @op level). It just doesn't work when we invoke dbt/dbt_athena from the @op ... Ideas?

Antauri commented 2 years ago

By comparison, the Redshift dbt extension has the mode: iam setting in the dbt profile. There's nothing similar in dbt_athena.

Antauri commented 2 years ago

It was a temporary IAM issue on our side. When running dbt inside a dagster "run" container (runLauncher with K8sRunLauncher) that has an attached kube2iam role, the dbt/boto3 will automatically get the IAM role from the environment (we had to remove aws_profile_name from the profile definition of dbt). Hopefully this information helps others in the same context.