Open edwrand opened 1 month ago
Thanks for reaching out @edwrand !
Did you confirm that the unique_key
has unique values for both of the following?
yes both places are unique but I still see this issue
@edwrand Unfortunately, I am not able to reproduce this error without seeing your data and the details of your incremental model.
If you're able to share a simple model and example data that produces this error, that would be ideal. But barring that, I'd suggest that you find the SQL that is causing the "Duplicate row detected during DML action" and trying to troubleshoot it in the Snowflake UI.
Here's how you can find that SQL:
Search within your target directory for a file named like this:
target/run/YOUR_PROJECT_NAME/models/.../fact_loan_histories.sql
You'll see some code that generally looks like this:
merge into analytics_dev.dbt_dbeatty.fact_loan_histories as DBT_INTERNAL_DEST
using analytics_dev.dbt_dbeatty.fact_loan_histories__dbt_tmp as DBT_INTERNAL_SOURCE
on (
DBT_INTERNAL_SOURCE.id = DBT_INTERNAL_DEST.id
)
when matched then update set
"ID" = DBT_INTERNAL_SOURCE."ID","UPDATED_AT" = DBT_INTERNAL_SOURCE."UPDATED_AT"
when not matched then insert
("ID", "UPDATED_AT")
values
("ID", "UPDATED_AT")
;
commit;
Then you can try running that SQL in the Snowflake UI. If it gives the "Duplicate row detected during DML action", then you can try to determine the source of the error. There should only be two possibilities:
Let us know what you find out!
Is this a new bug in dbt-core?
Current Behavior
I have an incremental model pulling from a collection of ephemeral models. I have defined a unique key (surrogate key of several columns). Using the MERGE strategy on the unique key I have defined (I have confirmed this key is unique), I get the error Duplicate row detected during DML action.
This confuses me because I have confirmed my key is unique and this how are these duplicate rows when MERGing?
Expected Behavior
I expected my MERGE strategy to incrementally merge new records based on my unique surrogate key. Instead this model fails with the error "Duplicate row detected during DML action"
Steps To Reproduce
Relevant log output
Environment
Which database adapter are you using with dbt?
snowflake
Additional Context
dbt-error-logs.odt