Data Warehouse Duplicated Data

[Gilbert09]

Bug Description

Bug description

• There's currently an issue with how we're ingesting incremental data from users sources (such as Zendesk, or a Postgres database).
• Updated rows aren't being merged with existing data as we expect, and so new rows are being inserted instead - causing duplicated data to exist
• We're working on a fix for this right now, but in the meantime, we've updated our documentation to indicate that duplicate data can exist for updated rows, and to only use incremental syncs for newly added data with a fallback to full refresh syncs for any table that updates existing rows
  • Updated docs for this can be found here

Resolution

• We're working on updating how we store users sourced data - you can follow along here https://github.com/PostHog/posthog/pull/23584
• A downstream library we use, DLT, are helping by implementing row merging on this new data format. You can follow along their progress here too https://github.com/dlt-hub/dlt/issues/1453
• We'll update this issue as we go along

[Gilbert09]

DLT have implemented merging of existing data for delta tables, and released an alpha package for us to test out. We've got an open PR here https://github.com/PostHog/posthog/pull/23850 to release this - we're doing some more internal testing and then releasing this. If everything goes according to plan, I'd expect this to be out on Monday.

[Gilbert09]

Everything has been merged here, and so we now support incremental upserts again!