microsoft / dbt-fabric

MIT License
79 stars 28 forks source link

Incremental update fails if the schema the the __dbt_temp differs from the base table #182

Closed M4Al closed 5 months ago

M4Al commented 6 months ago

I have a dbt model like this

WITH serviceEnteries AS (
    SELECT  
    case [PostingType] when 1 then 'Budget' when 2 then 'Actual' when 0 then 'Precalculation' end as PostingType
     ,OrderNo
    , SomeField

Now if in the incremental part are only records that result in 'Budget' lines, the schema of both tables will differ as can be seen here: image

image

This is due to the fact how the Fabric Warehouse engine guesses data types.

dbt will then continue and try to sync these schema's by updating the PostTypefield from varchar(7) to varchar(14) While it's technically possible it currently fails for some reason:

14:08:44  Changing col type from varchar(7) to varchar(14) in table database: "DWH"
schema: "silver_fin"
identifier: "ServiceEntry"

14:08:44  fabric adapter: Error running SQL: macro alter_column_type
14:08:44  fabric adapter: Rolling back transaction.
14:08:44  On model.mpl_dp.ServiceEntry: ROLLBACK
14:08:44  On model.mpl_dp.ServiceEntry: Close
14:08:44  Compilation Error in model ServiceEntry (models\silver\ServiceEntry.sql)
  'tmp_relation' is undefined

  > in macro alter_column_type (macros\adapters\columns.sql)
  > called by macro materialization_incremental_fabric (macros\materializations\models\incremental\incremental.sql)
  > called by model ServiceEntry (models\silver\ServiceEntry.sql)

I'm not sure why it fails to update the type.

prdpsvs commented 5 months ago

@M4Al , There is currently a bug in the adlter_column_type macro. I fixed this in 1.8.6 PR #178. I should release this soon.

M4Al commented 5 months ago

Fixed in 1.8.6