Open dataders opened 4 months ago
the framing here makes me think of a hypothetical third option based on work done to abstract away relation-specific details from CRUD
DDL statements. Of course with SQL, instead of CRUD
, we have CRADD
: CREATE
, REPLACE
, ALTER
, DROP
,DESCRIBE
. dbt-snowflake's macros/relations/
directory shows this clearly.
This work was done to support Dynamic Tables, but rolling this out globally for the other materializations might serve addressing option 2.
By that I mean we might have options available immediately to remove some of the cases, without removing the SnowflakeRelationType.DynamicTable
entirely.
for example, the lowest-hanging fruit here is that purportedly, a Dynamic Table can be dropped with a DROP TABLE
and DYNAMIC
is not required.
We can either:
snowflake__get_drop_dynamic_table_sql()
macro, but drop DYNAMIC
from it, orsnowflake__
override of default__get_create_sql()
that executes snowflake__get_drop_table_sql()
if relation.is_dynamic
describe
hitchWhile we can certainly alter dynamic table materialization directly, to call the additionally required DESCRIBE
, it would be cleaner if it could be incorporated into the existing relation-agnostic "CRADD"
macros -- especially given that we'll likely need to take this path for future relation types on the roadmap (EXTERNAL
ICEBERG
HYBRID
)
perhaps the existing snowflake__describe_dynamic_table()
could be incorporated somehow into either:
get_describe_sql()
, orload_cached_relation()
/list_relations_without_caching()
that can dispatch to relation-specific shims?
Is this your first time submitting a feature request?
Describe the feature
spin off of @jtcohen6's #1038
Investigate treating dynamic tables as
RelationType.Table
instead ofSnowflakeRelationType.DynamicTable
with the understanding that we need to run an additionaldescribe
query at the start of thedynamic_table
materialization to figure out if it's actually a dynamic or a static table (among other configs).Similar to the workaround described here: https://github.com/dbt-labs/dbt-snowflake/issues/1016#issuecomment-2104295670
This should only be done in the event that #1038 does not pan out, performance-wise
Pros
create or replace
when switching between tables and dynamics tables. (The team is looking into this, it may be feasible but it may not be.)Cons
describe
statement per DT, until we can re-plumb the materialization logic (or start caching more relation attributes) to avoid rerunning.create or replace table
for DT → table, this will have a known edge case where switchingmaterialized: 'dynamic_table'
tomaterialized: table
does not work.Describe alternatives you've considered
if #1038 works, we don't need to do this
Who will this benefit?
No response
Are you interested in contributing this feature?
No response
Anything else?
No response