[ADAP-880] [Regression] BigQuery - Removal of "OR REPLACE" for CLONE (future)

[brianmori]

Is this a regression in a recent version of dbt-bigquery?

• [X] I believe this is a regression in dbt-bigquery functionality
• [X] I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Dear DBT team,

I would like to give you a heads-up because I am not sure you are aware of the (future) loss of this functionality.

We are using the BigQuery Clone capability with CREATE OR REPLACE to have atomicity when doing updates.

screenshot


We contacted the GCP Support because we did not understand why the clone time did not change in subsequent executions.

We were told Google Cloud is going to remove this capability in two or three weeks. We were not informed by anyone and when I spoke with other BigQuery users, it looks like no one is informed.

The motivation by Google Cloud is it was not meant to be supported and it is listed in their Limitations

One can interpret the OR REPLACE as an atomic execution of DROP and CREATE. The overwrite has not a meaning in SQL. In my understanding, overwrite of a table clone can be to overwrite some existing data with UPDATE and this use case is supported with CLONE.

screenshot


Google Cloud team answered they are not removing any feature without notifying the customers and the Limitations have always been very clear. Nothing will change for the REPLACE function. The enforcement will take place in 2-3 weeks to be aligned with the documented Limitations.

We would like to inform you DBT team has a macro for BigQuery which uses the CLONE with CREATE OR REPLACE.

It would be best if you can also verify with your BigQuery contacts what Google Cloud told us and if confirmed also on your part to inform your BigQuery DBT users to find an alternative solution to avoid an interruption of service on the data pipelines in production.

Same interruption will appear with this dbt project.

Best Regards, Brian

Expected/Previous Behavior

Tables are cloned with atomicity

Steps To Reproduce

It works at the moment. The functionality is going to be removed in two or three weeks.

Relevant log output

No response

Environment

- OS: all
- Python: all
- dbt-core (working version): all
- dbt-bigquery (working version): all
- dbt-core (regression version): all
- dbt-bigquery (regression version): all

Additional Context

I have added the references in the ticket description

[dataders]

@wazi55 could you help us confirm if this is the case? if so there's some extra engineering effort to maintain the functionality for end users of dbt-bigquery

Google Cloud is going to deprecate [support of the OR REPLACE'] in two or three weeks [for the CREATE TABLE ... CLONE] SQL command.

[wazi55]

@wazi55 could you help us confirm if this is the case? if so there's some extra engineering effort to maintain the functionality for end users of dbt-bigquery

Google Cloud is going to deprecate [support of the OR REPLACE'] in two or three weeks [for the CREATE TABLE ... CLONE] SQL command.

Thanks, I will ask around

[wazi55]

@brianmori Do you mind sharing a contact you are talking to in GCP? I talked to our PM, (there are multiple of them), he's not aware of it, would be better if I can talk to someone directly who worked on your ticket.

Regards Z

[brianmori]

Here is the ticket number for the PM to check : 46639521

[brianmori]

Hi DBT Team,

I received a feedback this morning from Google Cloud. They are going to keep the functionality 🎆

They are going to remove the relative documentation for the time being to further improve this functionality.

I do suggest your PM to stay in the loop in any case.

Regards,

[dataders]

good news. this is certainly the lowest effort outcome on our part.

They are going to remove the relative documentation for the time being to further improve this functionality.

can you tell me more about this? that they'll remove the Limitations that you mentioned in your original issue?

[brianmori]

Absolutely. I understood Google is going to remove from the documentation the "CREATE OR REPLACE" is supported with CLONE for the time being. The functionality will continue to work though.

[wazi55]

Good to hear!

[dataders]

I'm going to close this issue for now then. Here's hoping that we won't be coming back to this in the future