search

Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
140 stars 79 forks source link

Zero-downtime for table recalculation within dbt run #88

Open SOVALINUX opened 2 years ago

SOVALINUX commented 2 years ago

Hi I'm sorry if I'm missing something - I'm new in dbt and just trying to switch the project from custom data model management to dbt

Does dbt-athena supports table recalculation without downtime while it's being recalculated?

What workaround I used to do in my own script: to give every CTAS an unique name (i.e. ) and have a helper view with a fixed name () and simple query `select * from ` - so once new version of ctas is calculated I switch a view to the new version and drop the old one. All usages reference only view and the whole thing works without downtime

Will appreciate your thoughts and comments!

iasoon commented 2 years ago

I had a shot at implementing a PoC for this feature as well, maybe that could help! My code is at https://github.com/iasoon/dbt-athena/commit/cca66da328058e40ec5490cdb7a5a4589b042440. It seems to work in simple tests, but it's not battle-tested :)