dremio / dbt-dremio

dbt (data build tool) adapter for the Dremio
Apache License 2.0
44 stars 21 forks source link

[Bug]: Incremental merge strategy not working (ambiguous column error) #242

Closed PilotKresse closed 1 day ago

PilotKresse commented 3 weeks ago

Is there an existing issue for this?

Current Behavior

When using incremental strategies with the incremental model, building the model fails on an incremental run with an error (column ... is ambiguous)

The reason is probably that in the generated MERGE INTO statment, the columns are not prefixed with DBT_INTERNAL_SOURCE in the statements following "when not matched then insert"

` merge into [TABLE] as DBT_INTERNAL_DEST using [TABLE] as DBT_INTERNAL_SOURCE on ( DBT_INTERNAL_SOURCE.id = DBT_INTERNAL_DEST.id ) when matched then update set "id" = DBT_INTERNAL_SOURCE."id","col" = DBT_INTERNAL_SOURCE."col"

when not matched then insert
    ("id", "col")
values
    ("id", "col")`

Expected Behavior

Incremental models with merge strategy should work

Steps To Reproduce

Run this example Model:

{{ config( materialized="incremental", incremental_strategy="merge", unique_key="id" ) }}

select 1 AS id, 1 AS col UNION select 2 AS id, 2 AS col UNION select 3 AS id, 2 AS col {% if is_incremental() %} UNION select 4 as id, 4 as col {% endif %}

the first time (non-incremental run) it works fine. On the second run, it will fail with error Column 'id' is ambiguous

Environment

- OS: macOS Sonoma
- dbt-dremio: 1.7.0
- Dremio Software: N/A
- Dremio Cloud: Yes

Relevant log output

09:06:25  Running with dbt=1.7.0
09:06:25  Registered adapter: dremio=1.7.0
09:06:25  Found 10 models, 1 test, 14 sources, 0 exposures, 0 metrics, 570 macros, 0 groups, 0 semantic models
09:06:25  
09:06:31  Concurrency: 2 threads (target='prod')
09:06:31  
09:06:31  1 of 1 START sql incremental model refined.address_changes ..................... [RUN]
09:06:41  1 of 1 ERROR creating sql incremental model refined.address_changes ............ [ERROR in 9.50s]
09:06:41  
09:06:41  Finished running 1 incremental model in 0 hours 0 minutes and 15.89 seconds (15.89s).
09:06:41  
09:06:41  Completed with 1 error and 0 warnings:
09:06:41  
09:06:41    Runtime Error in model address_changes (models/refined/address_changes.sql)
  ERROR: Column 'id' is ambiguous
09:06:41  
09:06:41  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
howareyouman commented 1 day ago

Fixed in master, will be shipped with 1.8 dremio-dbt version.