search

Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
140 stars 79 forks source link

`aws_profile_name` not used for glue, s3 calls #55

Open chronitis opened 2 years ago

chronitis commented 2 years ago

What I expected to happen:

With aws_profile_name in my DBT profile, all connections to AWS would use the specified SSO credentials (assuming I have the necessary permissions).

What happened

Everything works fine for seeds, tests, materializing tables and views.

When trying to materialize an incremental, partitioned table in insert_overwrite mode, the task fails part way through with the error Unable to locate credentials. This appears to happen after the temporary table for incremental update has happened, but before it has been inserted into the original table.

Cause

It looks like the profile is not used when making calls directly to Glue/S3 (rather than Athena). This happens in clean_up_table, clean_up_partitions (which construct boto3.Client directly), which expects to find static credentials in a config file or the environment.

This can probably be fixed by constructing a boto3.Session with profile and passing that to both pyathena and using it to construct S3/Glue clients.

I'll try and write a PR for this.

perlego-maares commented 1 year ago

Hi, is there any better work around to this?

What I ended up doing is setting the credentials to default profile

/.aws/config

[default]
region = eu-west-2
output = table
aws_access_key_id = xxx
aws_secret_access_key = xxx