Support incremental_strategy and data cleaning up with boto3

Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)

Apache License 2.0

142 stars 79 forks source link

Support incremental_strategy and data cleaning up with boto3 #27

Closed tuan-seek closed 2 years ago

tuan-seek commented 2 years ago

Changes:

Add support for two incremental update strategy 'insert_overwrite' and 'append'
For 'insert_overwrite', overlapping partitions are cleaned up from S3 location
Work with tables partitioned by multiple keys. Only tested with keys of types 'string' and 'integer'.
Add support for 'external_location' by removing physical files on S3. This change overcomes HIVE_PATH_ALREADY_EXISTS error.

roslovets commented 2 years ago

Look forward to merging this PR. I can't use this adapter while data in S3 is not overwriting.

tuan-seek commented 2 years ago

Hi @Tomme Have you got time to review the latest change?

Tomme commented 2 years ago

Hi @Tomme Have you got time to review the latest change?

@tuan-seek Planned for this week. Apologies for my slow replies and reviews.

tuan-seek commented 2 years ago

Hi @Tomme Is there any blocker for this PR to be merged? I'm planning some new feature change and would prefer to do it in a new PR from master, in stead of adding to this one.

Btw, I have updated README to reflect the change in this PR. Feel free to update README as you like.