search

fivetran / dbt_stripe

Data models for Stripe built using dbt.
https://fivetran.github.io/dbt_stripe/
Apache License 2.0
30 stars 31 forks source link

Add table rename at the end of each refresh #57

Closed nbenezri closed 1 year ago

nbenezri commented 1 year ago

Some tables take a long time to refresh, like:

09:12:05  119 of 175 START table model cdp_stripe.stripe__invoice_line_items ............. [RUN]
09:15:08  119 of 175 OK created table model cdp_stripe.stripe__invoice_line_items ........ [SELECT in 183.45s]

During this time - the table is not available to query in prod. One way to resolve this is to run in the end:

DROP TABLE IF EXISTS cdp_stripe.stripe__invoice_line_items_prod;
ALTER TALBE cdp_stripe.stripe__invoice_line_items rename to cdp_stripe.stripe__invoice_line_items_prod;

Is there such a way to do it when each table is ready?

fivetran-reneeli commented 1 year ago

Hi @nbenezri thanks for opening this issue!

Just to confirm I understand, are you saying that after the refreshed table gets created again, you're unable to query from it? But by renaming the table then you can? I'm having trouble understanding why it's unable to be queried in the first place. Or, can you query in dev, just not in prod?

nbenezri commented 1 year ago

When the refreshes going on it is unavailable. In the above example, during 09:12:05 - 09:15:08.

fivetran-reneeli commented 1 year ago

Hi @nbenezri sorry for the delay! I understand the issue you're experiencing, though this is the way dbt works as is it runs a create or replace statement. I think the issue may be better resolved looking at how your environment is set up. I would suggest working with your engineering team to make sure your dashboards are still available during table refreshes-- there should be some kind of best practices for when prod goes offline temporarily.

nbenezri commented 1 year ago

Yeah, I will rename the tables after each refresh. I thought there might be something simpler.

Thanks, Nir Ben Ezri DBA Team Lead DistroKid https://distrokid.com https://distrokid.com/

On Thu, Jan 26, 2023 at 11:24 PM Renee Li @.***> wrote:

Hi @nbenezri https://github.com/nbenezri sorry for the delay! I understand the issue you're experiencing, though this is the way dbt works as is it runs a create or replace statement. I think the issue may be better resolved looking at how your environment is set up. I would suggest working with your engineering team to make sure your dashboards are still available during table refreshes-- there should be some kind of best practices for when prod goes offline temporarily.

— Reply to this email directly, view it on GitHub https://github.com/fivetran/dbt_stripe/issues/57#issuecomment-1405672704, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATR4QSVYVLWF5KAAO2SJP7LWULTSXANCNFSM6AAAAAAUDZTMCM . You are receiving this because you were mentioned.Message ID: @.***>

fivetran-reneeli commented 1 year ago

Yeah, this is something more likely within dbt-core or your internal architecture! Will close this issue as this falls outside the realm of our packages