Sprint May 15 to May 26

[mariusandra]

Global Sprint Planning

3 things that might take us down

1. Integers?
   2. Postgres
2. Sporadic clickhouse queries spiking
   3. Some stuff would be improved by upgrading clickhouse

Retro: What can we do better next sprint?

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

### High priority

-

### Low priority / side quests

-

[mariusandra]

Team Product Analytics

Captain Toads next 2 weeks:

• @thmsobrmlr first week
• @mariusandra second week

Retro

• "Data Exploration" aka "JSON insights" aka "Name Pending" public launch @thmsobrmlr
  • ✅ Calling it "View Source"
  • Rolled out to 100% of users.
  • Can use another sprint of cleanup (removing flag and non-data exploration stuff, moving query conversion to the backend)
• Get PoEv2 out @yakkomajuri
  • Support one of the weeks.
  • Insights should be ready.
  • Squash is approved and in testing.
• HogQL public beta @mariusandra
  • Not much progress
  • Support hangovers (any event, funnel aggregation)
  • Pentest issues

Hang over items from previous sprint

• All of them in some capacity

OKR

• 🟡🔴 Objective 1: PostHog 3000 Hacker News launch is a wild success
• 🔵🟡 Objective 2: Make PostHog performance frustration free for our 10 largest customers
• 🟢🟡 Objective 3: HogQL

High priority

• Fixing reported issues by consultants
• HogQL release and CMD+Enter @mariusandra (non-support work only 2 days this sprint)
• PoEv2 and query support @yakkomajuri
• PostHog 3000 new navigation @Twixes

Low priority / side quests

• Ongoing "fixing things" work.
• Removing all legacy insight components @thmsobrmlr (support 1 week + support hangovers 1 week)

[danielxnj]

Team Infrastructure

Infra hero next two weeks @danielxnj

Retro

• Could not switch Postgres because the replication broke due to schema changes
• Enabled CloudFront on the US

Hang over items from the previous sprint

• US Production Postgres switchover

OKR

1. AWS Cost 🔴 increase in about 50%,
2. AWS US prod migration from root account to dedicated account 🟡

High priority

• Postgres switch to Aurora RDS, currently blocked by a slow VACUUM process running on the posthog_person table.
• Work on AWS costs, as they are going up by about 50%; waiting for approval to make AWS reservations to reduce the compute costs.
• Kafka US Production AWS account switch, tests were done on dev.

Low priority/side quests

• Add Clickhouse data source to the internal Grafana

[raquelmsmith]

Team Growth

Retro

• [x] Make platform features more understandable/visible in billing-by-products UI
• [x] Run experiment with new frontend for subscribe-by-products to make sure it doesn't kill conversion
  • Conversion was fine, live for all new customers
• [ ] Improvements for CS
  • Free trials - is usage still tracked when they are on a free trial? What happens when they go over the usage limits for the plan they are on?
  • Annual flat fees with overage fees - cumbersome to administer, need multiple subscriptions
• [x] Billing limits are applied before discounts but should be applied after

Hang over items from previous sprint

• Improvements for CS
  • Free trials - is usage still tracked when they are on a free trial? What happens when they go over the usage limits for the plan they are on?
  • Annual flat fees with overage fees - cumbersome to administer, need multiple subscriptions

OKR

1. We can easily run experiments on pricing 🟡
2. Each paid product is positioned as a first-class citizen within our onboarding and upgrade flows 🟡

High priority

• [ ] Test out new product + group analytics pricing
• [ ] Improvements for CS
  • Free trials - is usage still tracked when they are on a free trial? What happens when they go over the usage limits for the plan they are on?
  • Cancelling (or not) subs for unpaid invoices https://github.com/PostHog/company-internal/issues/1019
• [ ] Move all existing customers to new billing-by-products UI https://github.com/PostHog/billing/issues/207
• [ ] Run experiment for 10M event allotment on free plan?
• [ ] Create modal for prompting to upgrade to addons when sub-ing to a main product

Low priority / side quests

• [ ] Improve modal for upgrading to paid features
  • Let them subscribe in the modal
  • Tell them which product they need

[tiina303]

🚰 Team Pipeline 🚰

Support heros next 2 weeks: Xavier, then Tiina Vacation: Tomas for week 2

Retro

• @hazzadous - happy about stuff we got out, sad about that some we didn't. Zendesk stuff seems OK, we're actually able to solve people's problems which is nice.
• @tiina303 - feel sad that our alerts especially the ones going to all team oncall are in going to get paged for E2E lag it's not easy to know what to do. Happy to work on simplifying code & improving data trust.
• @tomasfarias - happy with progress on Squash side, finally running it in DEV. Shout out to Xavier to for the worker stuff. No-one has reviewed the batch job huge PG :(
• @xvello - prod has been ok'ish, we're not hands off yet. Lots of changes happening has caused things to be not great ... feeling overwhelmed with how much I need to keep track of.

Last Sprint

• 🟡 making sure we don't drop person properties when there are updates across different distinct IDs @tiina303
  • Code clean-up is on the way, will continue to next sprint
• 🟡 Librdkafka migration for analytics @xvello
  • Contribute nodes worker support to librdkafka
  • PR opened, not reviewed by maintainers yet. Not blocking as we removed Piscina
  • Analytics workload uses librdkafa for consumer & producer
  • Producer has been stable since last week
  • Consumer rolled-back today, need to carry over and investigate throughput
• 🟡 batch exports to S3 enabled for 3 teams @tomasfarias

Plan

• making sure we don't drop person properties when there are updates across different distinct IDs @tiina303
• Librdkafka migration for analytics consumer @xvello
• batch exports to S3 enabled for 3 teams @hazzadous (EDIT)

OKRs

• Objective: CDP - @fuziontech
  • 🟡Key Result: Rock solid webhooks - @hazzadous
  • 🟢Key Result: Rock solid batch processing (exports) - @tomasfarias
• Objective: Pipeline unblocks query performance and ensures data quality - @tiina303
  • 🟡Key Result: Persons on Events write path and squash shipped with monitoring and alerting
  • 🟡Key Result: Backfills are complete for all teams
  • 🟡Key Result: Guaranteed deduplication of UUIDs in events table within 7 days (with docs explaining it)
  • 🟡Key Result: Query time deduplication of events to cover until ClickHouse deduplication is complete
• Objective: Pipeline Robustness and Reliability - @xvello
  • 🟡 Key Result: Errors cause kafka lag instead of DLQ production e.g. person creation failures cause kafka lag not go to DLQ
  • 🟢 Key Result: Plugin server scales elastically and deploys do not create more than 2 minutes of lag

[neilkakkar]

Team Feature Success

Support hero: Li

Eric is off entire sprint.

Retro

Managing rollout & gathering feedback & doing user interviews: Beta management Eric Duong

• except deletions (blocked on the infra work), all released by Friday

Dashboard templates on flags Eric Duong

• didn't get to it

Flag autocapture Neil Kakkar

• got deprioritised for person on events v2 + decide rate limiting. Halfway there.

MVP for productising interviews Li Yi Yu

• research + focusing on feedback first, over interviews.

Hang over items from previous sprint

Flag autocapture MVP for productising interviews Dashboard templates on flags

Next sprint

1. Flag autocapture + beta management follow up Neil Kakkar Why? Core component of feature success. Automates flag analytics & linking to recordings.
2. Productising feedback Li Yi Yu Why? Core component of feature success. Allows getting qualitative user feedback quickly.

[benjackwhite]

Team ~Notebooks~ Monitoring

Support hero: @pauldambra

Retro

• todo...

Hang over items from previous sprint

• Notebooks MVP - Got a real long way here but there was a lot of learning work to get into a half decent state
• S3 Recordings - we put a little more effort in so we can now list and read from S3 🥳 but there are some really tricky race condition bugs still to be dealt with.

Goals

• Goal 1 : Finish the 80:20 of session recording 🟢
  • S3 work is on track it feels
• Goal 2 : Notebooks MVP 🟢
  • Starting to shape up - Insights are the most painful part
• Goal 3 : Release MVP of Error tracking product 🟡
  • Not started - delayed another sprint to make sure we don't end up too split

High priority

• Notebooks MVP - for real this time... @benjackwhite @pauldambra why? We didn't get the energy to really do this so now we double down and get it out before taking on anything new

Low priority / side quests

• Fix final issues with the recording ingester

[fuziontech]

💾 Team Data Warehouse 💾

Support hero: James

Retro

• Accidentally deleted 1.7Bn events. 🙈
• S3 restore is not a silver bullet 🚅
• Replicating to EBS now (scope for big booms is much smaller)
• Helm chart is in a really great state.
• Cluster controller (need to roll out properly) but is working
  • Adds/removes tags
  • Configures ZK for CH
• Fewer CH fires on US Prod but not quite out of the woods yet. Still a few spikes to deal with.

High priority

• Get vector setup for Dev and maybe Prod Ingestion @frankh
  • Start with low risk ones
  • Partition statistics
  • Performance events
  • Meta columns in vector
• Clickhouse Copier - RACE (whoever has bandwidth first)
  • Use to get a few billion events in the S3 cluster
  • For query benchmarking purposes
• Backups @fuziontech
  • Testing CH7 and CH8
  • Look into disk backups
• Upgrade CH Cluster versions for cluster @fuziontech