Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
140 stars 79 forks source link

[BUG] Error from boto3 API is not caught and create duplicated data #84

Open Gatsby-Lee opened 2 years ago

Gatsby-Lee commented 2 years ago

Hello all,

I have a dbt model with this config.

{{
  config(
    materialized='incremental',
    incremental_strategy='insert_overwrite',
    partitioned_by=('dummy_partition',),
  )
}}

With this config, the expected behaviors is like this.

The problem I noticed was that there can be duplicated data if the deleting S3 objects fails ( 'cuzed by permission issue )

This is the code and return value from Boto3 S3. I think this line should check if the the returned value's Error is empty or not. https://github.com/Tomme/dbt-athena/blob/master/dbt/adapters/athena/impl.py#L105

s3_bucket.objects.filter(Prefix=prefix).delete()
[{'ResponseMetadata': {'RequestId': 'S2M8Z423GKH25JZW', 'HostId': '+igjv+OblFmcETz1WWHkuhcaW0muPZef9bI=', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '+igjv+OblFmcETz1WWHkuhcaW0muPZef9bI=', 'x-amz-request-id': 'S2M8Z423GJZW', 'date': 'Sat, 02 Apr 2022 00:53:39 GMT', 'content-type': 'application/xml', 'transfer-encoding': 'chunked', 'server': 'AmazonS3', 'connection': 'close'}, 'RetryAttempts': 0}, 'Errors': [{'Key': 'dbt_v1/tables/3338-4afd-b918-ec7dbc7a2225/dummy_partition=default/20220402_003612_00013_t88vg_1685448d-3936-47ac-9a58-6640103ed4f3', 'Code': 'AccessDenied', 'Message': 'Access Denied'}, {'Key': 'dbt_v1/tables/3338-4afd-b918-ec7dbc7a2225/dummy_partition=default/20220402_003445_00029_9mcfm_a79200a7-f61e-476d-90ca-529139ec48bc', 'Code': 'AccessDenied', 'Message': 'Access Denied'}]}]