Tomme
commented
3 years ago Hey @mrshu 👋
I would agree that it is not impossible however to implement it would mean allowing the adapter destructive actions on your data within S3 which one could argue is out of the scope for Athena (and thus this adapter). I might ask on the #db-athena Slack channel for people thoughts and opinions.
Regardless a rough approach to how we could implement snapshots and upsert incremental models could be:
- Copy new and updated records to a temporary table
- Identify partition files which contain records which will be changed/updated, merge them into the temporary table
- Destroy old partition files
- Insert new data alongside relevant partition files
mrshu
Hey @Tomme ,
First of all, thanks a ton for
dbt-athena-- it works very nicely!I was wondering would would it take to implement snapshots with Athena. Since it doesn't really do updates, it will be quite a challenge but from the outset it does not seem completely impossible.
Would you see it that way as well? It seems the Presto adapter is also not implementing it, so I was wondering what would your thoughts on something like this be.
Thanks!