feat: Add a debug table for clickhouse_events_json - Githubissues

PostHog / posthog

🦔 PostHog provides open-source product analytics, session recording, feature flagging and A/B testing that you can self-host.

https://posthog.com

Other

21.2k stars 1.26k forks source link

feat: Add a debug table for clickhouse_events_json #23377

Closed fuziontech closed 3 months ago

fuziontech commented 3 months ago

Problem

We currently don't have any metrics to compare the number of events coming into kafka and the number of metrics that are at rest in sharded_events tables.

Changes

This adds on a kafka debug table that: 1) Does not deduplicate (important for keeping tabs on duplicates and where they are coming from, the topic or the CH Kafka consumer) 2) Stores the raw event payload as a string. As long as we are JSON encoding events we run the risk of having malformed JSON payloads. This let's us introspect and replay if needed straight from CH.

Does this work well for both Cloud and self-hosted?

How did you test this code?

posthog-bot commented 3 months ago

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot commented 3 months ago

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

github-actions[bot] commented 3 months ago

Size Change: 0 B

Total Size: 1.06 MB

ℹ️ View Unchanged

| Filename | Size | | :--- | :---: | | `frontend/dist/toolbar.js` | 1.06 MB |

_{compressed-size-action}

posthog-bot commented 3 months ago

📸 UI snapshots have been updated

3 snapshot changes in total. 0 added, 3 modified, 0 deleted:

chromium: 0 added, 3 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot commented 3 months ago

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot commented 3 months ago

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

chromium: 0 added, 4 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot commented 3 months ago

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.