dbt-msft / dbt-sqlserver

dbt adapter for SQL Server and Azure SQL
MIT License
216 stars 101 forks source link

support Ephemeral models #166

Open dataders opened 3 years ago

dataders commented 3 years ago

from #137, here's the proposed solution

Ephemeral models

The general approach outlined in https://github.com/dbt-msft/dbt-synapse/commit/9180da5a6d092c6b4b22d5388aa6592d2b9d77c4 is still the same, though I was doing it on easy mode, and didn't actually try to solve for ephemeral model compilation :)

The mechanism is: reimplement the compiler, define a new one (like in dbt/adapters/sqlserver/compilation.py), and reimplement methods like _inject_ctes_into_sql and _recursively_prepend_ctes.

Taking a step back: In order to support ephemeral models with CTEs on SQLServer and Synapse, given that T-SQL doesn't support CTEs nested inside either CTEs or subqueries, would we have to take the same approach as the one outlined above?

Namely, instead of recursively prepending ephemeral models as CTEs to the start of the query:

with  __dbt__cte__ephemeral_model as (

with my_cte as (

    select 1 as id

)

select * from my_cte
),ephemeral_model as (

select * from __dbt__cte__ephemeral_model

),

another_cte as (

    select 2 as id

)

select * from ephemeral_model
union all
select * from another_cte

Recursively prepend them as temp tables, to be executed in-transaction with the body of the query:

create table #ephemeral_model as (

with my_cte as (

    select 1 as id

)

select * from my_cte
);

with ephemeral_model as (

select * from #ephemeral_model

),

another_cte as (

    select 2 as id

)

select * from ephemeral_model
union all
select * from another_cte
jtcohen6 commented 12 months ago

I don't think we should do this :)

Newer MSFT databases should support nested CTEs (https://github.com/dbt-msft/dbt-sqlserver/issues/457). We shouldn't expose the full Compiler interface to adapters (https://github.com/dbt-labs/dbt-core/pull/9134).