search

fleetio / dbt-segment

Data models for Segment built using dbt (getdbt.com).
https://hub.getdbt.com/dbt-labs/segment/latest/
Apache License 2.0
10 stars 7 forks source link

incremental stitched model produces duplicate records #8

Closed efernandez1-2u closed 1 year ago

efernandez1-2u commented 1 year ago

Describe the bug

Steps to reproduce

Expected results

Actual results

Screenshots and log output

System information

The contents of your packages.yml file:

Which database are you using dbt with?

The output of dbt --version:

dbt 1.5

The operating system you're using:

The output of python --version:

Additional context

Are you interested in contributing the fix?

efernandez1-2u commented 1 year ago

The issue is actually not the code in the models. I had this in my dbt_project.yml file which ended up being the problem however Im not sure why it caused an issue: `dbt_segment: enabled: true +tags: [dbt_segment]

base:
  +tags: []
  +schema: ngwt_cmbd

sessionization:
  materialization: incremental
  +tags: []
  +schema: ngwt_cmbd`

the materialization: incremental line seemed to be the problem because when I removed it, no duplicates occurred in either full refreshes or regular incremental runs. I just fail to understand how that had the ability to cause an issue. The materialization is already in the model definition so would it matter if its defined in either place or both, doesnt only one superceded the other? And in this case it was the same materialization so how would it have changed things. The issue I originally submitted is resolved in that my incremental runs are not producing duplicates anymore but could there be a different issue with dbt_project configuration or is this expected?