Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
142 stars 79 forks source link

DBT run fails with assume role policy #151

Closed Ryno-Marx closed 1 year ago

Ryno-Marx commented 1 year ago

Hi all - I'm in need of some assistance - apologies if this is something that's been covered already.

I'm unable to run dbt-athena via an assume-role policy.
I've created a basic model that I'm running with the dbt-athena plugin.

I've created an AWS policy (dbt_policy) that allows all the permissions needed for the plugin to run and execute the models. For my testing I've assigned the policy to an IAM user dbt_poc_dev_user and it works perfectly.

Now I'd like to make use of an IAM role instead of a user. I've done the following:

With dbt run I get the following error:

13:58:57  Running with dbt=1.5.2
13:58:57  Registered adapter: athena=1.5.1
13:58:57  Unable to do partial parsing because config vars, config profile, or config target have changed
13:58:57  Unable to do partial parsing because profile has changed
13:58:58  Found 3 models, 0 tests, 0 snapshots, 0 analyses, 336 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
13:58:58
14:00:26
14:00:26  Finished running  in 0 hours 1 minutes and 27.70 seconds (87.70s).
14:00:26  Encountered an error:
Runtime Error
  Runtime Error
    [ErrorCategory:USER_ERROR, ErrorCode:PERMISSION_ERROR], Detail:Amazon Athena experienced a permission error. Please provide proper permission and submitting the query again. If the issue reoccurs, contact 
AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience., Message:Amazon Athena experienced a permission error. Please provide proper permission and submitting the query again. If the issue reoccurs, contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience.

If I switch the aws_profile_name back to the user it completes successfully

Another test that I've done is running a docker container with my code on EC2 via AWS Batch.

For this I don't need the aws_profile_name in the prrofiles.yml file. I just attached the dbt_policy to the EC2 execution role. Same result - I'm not sure if roles can be used with dbt-athena?

Ryno-Marx commented 1 year ago

Granted the role lake formation permissions and it sorted the problem