dbt-labs / dbt-snowflake

dbt-snowflake contains all of the code enabling dbt to work with Snowflake
https://getdbt.com
Apache License 2.0
276 stars 167 forks source link

[SPIKE] Investigate treating dynamic tables as `RelationType.Table` #1040

Open dataders opened 4 months ago

dataders commented 4 months ago

Is this your first time submitting a feature request?

Describe the feature

spin off of @jtcohen6's #1038

Investigate treating dynamic tables as RelationType.Table instead of SnowflakeRelationType.DynamicTable with the understanding that we need to run an additional describe query at the start of the dynamic_table materialization to figure out if it's actually a dynamic or a static table (among other configs).

Similar to the workaround described here: https://github.com/dbt-labs/dbt-snowflake/issues/1016#issuecomment-2104295670

This should only be done in the event that #1038 does not pan out, performance-wise

Pros

Describe alternatives you've considered

if #1038 works, we don't need to do this

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

dataders commented 4 months ago

the framing here makes me think of a hypothetical third option based on work done to abstract away relation-specific details from CRUD DDL statements. Of course with SQL, instead of CRUD, we have CRADD: CREATE, REPLACE, ALTER, DROP,DESCRIBE. dbt-snowflake's macros/relations/ directory shows this clearly.

This work was done to support Dynamic Tables, but rolling this out globally for the other materializations might serve addressing option 2.

By that I mean we might have options available immediately to remove some of the cases, without removing the SnowflakeRelationType.DynamicTable entirely.

for example, the lowest-hanging fruit here is that purportedly, a Dynamic Table can be dropped with a DROP TABLE and DYNAMIC is not required.

We can either:

  1. retain the snowflake__get_drop_dynamic_table_sql() macro, but drop DYNAMIC from it, or
  2. introduce a snowflake__ override of default__get_create_sql() that executes snowflake__get_drop_table_sql() if relation.is_dynamic

the describe hitch

While we can certainly alter dynamic table materialization directly, to call the additionally required DESCRIBE, it would be cleaner if it could be incorporated into the existing relation-agnostic "CRADD" macros -- especially given that we'll likely need to take this path for future relation types on the roadmap (EXTERNAL ICEBERG HYBRID)

perhaps the existing snowflake__describe_dynamic_table() could be incorporated somehow into either: