Add deduplication on query level

MarconLP commented 10 months ago

We've received several reports on duplicated events. We will tackle this problem in three places (Duplicate removal at ingestion, Duplicate removal at CH, Duplicate removal at query time)

Add duplicate removal at query time:

Whatever wasn't caught by the other two, is caught at query time.
Ideally, we should be querying from a historical de-duped table, thus incurring in 0 de-duplication costs, and only running DISTINCT ON/GROUP BY on the freshest data that hasn't been de-duplicated yet.
This way, we keep performance impact minimum

MarconLP commented 10 months ago

how many duplicates can we process before performance is taking a big hit

jetaggart commented 10 months ago

Any updates on this? We have some fairly serious data issues due to duplication making many of our reports unusable. We can create custom HogQL to do this deduping on the report level but obviously that has downside. Our team that creates self-service reports are unable to trust the numbers right now.

MarconLP commented 10 months ago

Hey @jetaggart, could you open a ticket through the PostHog app? https://app.posthog.com/home#supportModal=support%3Adata_integrity

jetaggart commented 10 months ago

I have an open ticket that is being worked on however I'm wondering what the underlying issue is and what resolution will be (ticket 5270)

PostHog / posthog

Add deduplication on query level #17136