Sprint - Oct 14 to Oct 25, 2024

[mariusandra]

Global Sprint Planning

3 things that might take us down

1. Unpredictable ingestion lag
   2. Clickhouse

Urgent incident follow-ups

https://github.com/orgs/PostHog/projects/103/views/2

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

### High priority

-

### Low priority / side quests

-

[mariusandra]

Team Customers Drive Profits

Support: Oliver

Retro

• 🟢 Communicate action and destination upgrades @mariusandra
  • We did a lot of communication. Destinations are upgraded. CDP is shipped.
  • Stuck behind new pricing before we can ask everyone to move over, so old action webhooks still exist.
• 🟢 Bring destinations 3000 out of feature preview @benjackwhite
  • It's out, it's done, it's live, all teams run on cyclotron
• ⚪ Documenting and updating runbooks for realtime destinations @benjackwhite
  • Ben is on holiday, not sure what got done if anything
• 🔴 Pricing? @annaszell
  • Focus this sprint

OKRs

• Widespread usage of the new CDP
• Messaging V1
• Ingestion is rock solid

Sprint focus

High priority

• CDP pricing @annaszell @mariusandra
• Figure out what to build with messaging (and build v0.0.1-alpha) @mariusandra
• Error tracking & support. Cross team support, not a lot of feature work (on CDP). @oliverb123
• Google & Meta ads destinations, help with messaging UI @MarconLP

[pauldambra]

Team Last Sprint in Daylight Savings Time

Support hero: @pauldambra

Almost all in the full sprint

Retro

• a bunch of tiny fixes and UI improvements this sprint @pauldambra
• are we doing too much support ticket triage? @marandaneto

---

• ✅ start implementing the flutter session replay plugin
  • draft is already working
• ✅ experiment with SwiftUI masking
  • experiment worked
• 🌀 start recording on a trigger (e.g. error captured) @richard-better
  • core is there just the final push to get shipped now - should fit within the sprint
• ✅ track impact of replay capture improvements @pauldambra
  • there is more we can do but this seems largely resolved for posthog cloud
• ✅ onboard Yiannis
• 🤞 nobody gets lost or arrested in Vienna
• 🤞 uncaptured network requests @pauldambra
  • no reports of this since last improvements
• ◐ error assignment into activity log @pauldambra
  • draft PR should be sorted this sprint

OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

🟡📱Goal 1: People think of PostHog as a mobile solution 🟡 📹 Goal 2: Replay playback 3000

• delayed-ish till Nov and Cory time in Valencia 🟡 🎤 Goal3: Ship the things users keep asking for

High priority

Mobile

• get mobile recordings into billing_usagereport @pauldambra
• track size of mobile recordings so we understand better how expensive mobile ingestion / storage is @pauldambra
• get first alpha of flutter replay out @marandaneto
• implement the swift ui masking from last sprint's experiment @ioannisj
• auto capture for element interactions in iOS https://github.com/PostHog/posthog-ios/issues/179 @ioannisj

Replay

• roll out the URL trigger and adding a URL denylist @richard-better

Low priority / side quests

• trigger to start recordings on event @richard-better
• lazy loading replay in the SDK @pauldambra

[daibhin]

Team ~GPS~ Error Tracking

Time off: @oliverb123 @neilkakkar maybe off 1-2 days @daibhin @neilkakkar EU on-call week 1 each

Retro

✅ Symbolify a single stacktrace @oliverb123

• New Rust service created and reading from Plugin Server
• Local environments configured and demangled our first stack frame!
• End-to-end test to run through entire service
• Source map upload endpoint & S3 bucket ready to go

✅ Better filtering @neilkakkar

• Universal search added to exception query
• Added super property support to Python to support exception filtering by repository
• Searching by url, file, or task name

🟢 Start planning issues overrides table schema @daibhin

• RFC in-progress for the approach we plan on taking to handle issue merging

✅ Enable autocapture on billing repo @daibhin ✅ Reached out to Python SDK customers. 2 people going through setup @daibhin ⏩ Rename stack trace properties on JS SDK (beware: Sentry integration relies on some of this) @neilkakkar

OKR

🟢 Human readable stack traces for Javascript 🟢 Ensure scalability of query time issue grouping ⚪️ Alerting via CDP 🟢 Launch

High priority

Storing resolved source maps @oliverb123

• Create Postgres model for source map uploads (symbol set reference -> S3 URL) @neilkakkar
• Writing / Reading source maps via S3
• Local caching / invalidation of symbol set results

Overrides table for issue merging @daibhin

• Resolve open questions for alerting support
• Create ClickHouse table
• Re-write exceptions query to join on overrides table

Advanced fingerprinting in Rust @neilkakkar

• Port fingerprinting from plugin server
• Compute fingerprint representation for unresolvable frames
• Add boolean to track resolution of a frame
• Fingerprinting logic (e.g. exclude unresolvable stack frames if trace includes resolvable frames)

Low priority / side quests

• Add language tag to frames on the JS & Python client @neilkakkar @daibhin
• Choose persistent key / value store for stack frame resolution (Postgres or Dynamo or something else) @oliverb123

[jurajmajerik]

Team Experiments

Q4 2024 Objectives

1: Migrate to HogQL queries

We will migrate Experiments to HogQL, making result calculations more reliable and performant. This will also enable the addition of new features listed below.

2: Multiple experiment goals supported + visualized

We will add support for multiple goal metrics in a single experiment, allowing them to be visualized together and making it easier to interpret results across all metrics simultaneously.

3: Reusable experiment metrics

We will add the ability to create metric sets that users can save and reuse. This will reduce friction, improve maintainability, and decrease the likelihood of errors.

4: Review and adjust methodology with a statistician

We will review our current methodology to ensure it is both accurate and easy for our users to understand.

Retro

🟢 Feature Success offsite

• worked on Experiment UI improvements during hackathon
• Phani walked us through No-code experiments, still some todos before full handover

🟡 Continued on HogQL for Experiments (https://github.com/PostHog/posthog/issues/24914) - about 60-ish % done - query endpoints are now callable

Sprint goals

• Support hero for the Feature Success queue
• Continue HogQL rewrite
  • plug statistical calculations
  • migrate UI behind a feature flag
• Review the methodology with a statistician - starting this Friday!

Secondary

• Investigate default exposure for trends showing 1

[Phanatic]

Team Surveys

Retro

• Customer support :
  • Overall really good.
  • Met a lot of SLAs

✅ Ship & Beta testing no-code experiments with customers - @Phanatic ✅ Fixed some issues with surveys and cohort filters from support hero coverage - @dmarticus ✅ Various small UI fixes for Surveys and Flags - @jurajmajerik

Side Quests

• Show a banner on Surveys page with posthog-js version in use not supporting survey features.

---

OKRs

1. Implement Enterprise-ready features for Surveys
2. Increase adoption of Surveys.

High priority

• Beta test no-code experiments with customers.

• 17512

• Investigate support for mobile surveys in posthog-react-native library

[fuziontech]

Team Where Do We Put Our Data?

OKR Q2 2024

Objective

Goals for Q4:

• Improve elasticity and flexibility of our data store by putting all our data in Iceberg
  • Work with Altinity effectively to ship read path for Iceberg on ClickHouse - Brett Hoerner
  • Setup infrastructure to ship all of our data to Iceberg on S3 - James Greenhill
  • Shipping Query logs to Iceberg - James Greenhill
• Continue improving CH operational expertise
  • Upgrade to a later version of Clickhouse - James Greenhill
  • Capacity planning - James Greenhill
  • Automation - Daniel Escribano
  • Put some of the basic mitigation operations in runbooks - Daniel Escribano
• Schema management
  • Tool for schema migration (coordinator schemas) - Daniel Escribano
  • Tool for long running mutations - Daniel Escribano
• Continued investment in performance
  • Tooling for other teams to understand which queries are slow and why - Ted Kaemming
  • Investigate variability of queries - Ted Kaemming
  • Per-team limits on queries/query complexity (needs product work) - Ted Kaemming

Board

https://github.com/orgs/PostHog/projects/85/views/2

Retro

🔥 Outages 🔥

• EU Batch Exports using large amounts of memory + earlyoom chaos monkey
• CH node OOM due to UDF on US causing widespread issues in the cluster

Last Sprint's Goals

• Write path for Iceberg setup to iterate ✅
• Backup cluster is moved to i4i.metal (more disk more io) ✅
• Upgraded to ClickHouse 24.4 ⚠️
  • Progress has been made, but the new query parser is much more strict with our queries
• We have a plan with Altinity ✅

This Sprint's Goals

TL;DR

• Reformat 🧊 berge schema to accommodate Batch Export work @bretthoerner @Daesgar
• Prod writing events to Iceberg to support ^ @Daesgar
• Have a plan for Materialized Columns on 🧊 @fuziontech @tkaemming
• Schedule compaction job in DW/Temporal somewhere @bretthoerner & DW
• Setup TTL for events for 48 months to gp3 disk @fuziontech
• Complete upgrade to CH 24.4 @fuziontech
• First PRs with Altinity @bretthoerner (most likely)
• Automated TF for CH @Daesgar

Side Quest

• Setup Altinity CH Operator for BE use (ephemeral, elastic, and decoupled version testing)

Board Snapshot

[zlwaterfield]

Team Growth

Retro

Retro items

- @raquelmsmith - [ ] Implement Personless default switch - [ ] Keep working with new onboarding templates flow - [x] Move activation events to the backend - [x] Help Patricio get up and running - [x] Work with data science contractor - @zlwaterfield - [x] next round of emails from mailgun to c.io - annual credits - [x] release to more users - [x] make the banner closable but still show on the page - [x] https://posthog.slack.com/archives/C01MGUHFH6G/p1726790198011829 - credits ui for annual customer and existing customers with a discount - RBAC - [x] Finish reviewing Ben's existing PR and RFC - [ ] Finish v1 of the RFC for billing - [ ] Start working on it - decide if we should continue on Ben's PR or start from scratch and do it piece by piece - get at least 1 PR merged for it - [ ] (from previous) ICP analysis - review completed work and improve charts/dashboard for new data flowing in from SalesForce - @patricio-posthog - [x] get onboarded - [x] submit first PRs on optimizing syncinc invoices

Q4 Goals

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

1. 🟡 Get to a 4/10 in Enterprise platform support (Zach)
   • 🟡 Release Enterprise addon and iterate with sales to find a pricing model that makes sense
   • 🟡 Build RBAC
2. 🟡 Make revenue & forecasting more reliable (Patricio)
   • 🟡 Rejig how we do sync_invoices so we don’t see big drops - respond to webhooks instead of large syncs
   • ⚪ Evaluate temporal vs celery for usage reports and billing tasks so we can adhere to rate limits better
   • ⚪ Side hustle, if time permits: Rejig how anonymous / identified events are billed, as the current implementation is confusing.
3. 🟡 Activation: reverse trials, analytics onboarding flow, understand CC activation (Raquel)
   • ⚪ Dig into the data, watch replays to figure out why CC activation rate has been falling
   • ⚪ Add reverse trial to onboarding
   • 🟡 Add reverse trial to billing page for teams

This sprint

Time off: Zach 1 day, Raquel 2 days

• @raquelmsmith
  • [x] Fix forecasting bug that says peoples' forecast is much higher than what they see on billing page
  • [ ] Fix bugs with product analytics onboarding flow
  • [ ] Personless default change
  • [x] Make sure metrics move didn't break wnything
  • [x] Re-up activation metrics for product analytics - try using insight saved instead of insight created
  • [x] Use new product intent table in activation metrics
• @patricio-posthog
  • [ ] Continue working with the sync invoices changes to use webhooks instead of a daily sync using Stripe API
  • [ ] Check billing limit bug raised by jdoodle (support ticket)
  • [ ] May investigate slow load times in billing page and try to optimize them (confirm first if it only affects ElevenLabs)
• @zlwaterfield
  • RBAC
    • [ ] Finish v1 of the RFC for billing
    • [ ] Start working on it - decide if we should continue on Ben's PR or start from scratch and do it piece by piece - get at least 1 PR merged for it
  • [x] Finish up the Technical Exercise RFC - resolve comments and get merged, share out
  • [ ] Get the CC entry PR shipped
  • [ ] Adding user id to billing token for events
  • [x] Adding Hogfetti component
  • Growth review (if time permits)
    • [ ] We had a number of “accidental” signups that got refunded in sept, which might show up as churn
    • [ ] Teams plan survey after they subscribe
    • [ ] Use new ICP data to compare with CC activations - are high ICP people still subscribing at the same rate?
    • [x] Add is subscribed bool and mrr to org group

[danielxnj]

Team Infra

Retro / hangover

• ArgoCD all the way @danielxnj @bciaraldi https://github.com/PostHog/charts/issues/1836
  • 🟢 Argocd Rolling syncs
  • 🟢 Argocd diff workflow
  • 🟡 We want all internal k8s stuff deployed with ArgoCD (12 internal helm charts were migrated, 25% done) status
  • 🟡 We want a PostHog service deployed with ArgoCD (Billing was migrated for dev)
• 🔴 PoC'ing out what a great secret management workflow would look like @bciaraldi

OKR

1. 🏃 Continuous Delivery
2. 💰 Cost visualization
3. 🧪 Continuous Integration

High Priority

• ArgoCD all the way @danielxnj @bciaraldi
  • Finish moving all internal helm charts to ArgoCD
  • Fully deploy a PostHog service deployed with ArgoCD

Side quest

• Drift detection

[Twixes]

Team AI Product Manager

Support hero: final support for Product Analytics @skoob13, then @Twixes

Retro

• ✅ New queries for retrieving relevant events/properties: event taxonomy query and property values query @skoob13
• ✅ Productizing the ReAct approach to generation @skoob13
• ✅ Showing a set of suggested questions in the UI @Twixes

Objective

LLM-based product manager as a service

High priority

• Router between AI agent's skills (trends, funnels, SQL, etc.) @skoob13
• Support for all insight types @skoob13 @Twixes
• Basic SQL support @Twixes
• Answer feedback button @Twixes

Low priority / side quests

• Environments out as a feature preview @Twixes

[aspicer]

Team Product Analytics

Support hero: final support for Product Analytics @skoob13, then @Twixes

Retro

• ✅ Both members have one week of support
• ✅ Anirudh to roll out alerts and make improvements based on user feedback
  • Well received, no major issues.
  • From early user feedback, adding relative alerting and making changes to teh dynamics of the snoozing mechanism
• ✅ Sandy to get UDFs rolled out and work on cohort migration in free time
  • UDFs deployed and rolled out to team.
  • OOM issues discovered and resolved
  • Code framework for cohort migration built, will be increasing scope by adding property group types in one by one

Sprint

Anirudh

• Create snoozing mechanic for firing alerts
• Launch relative alerts
• Roll out to more users (talk to comms about finding beta test customers)

Sandy

• UDF rollout for steps + moving funnel trends to UDF
• Get a subset of cohorts calculating on HogQL

[EDsCODE]

Team Data House

OKR Q4 2024

Objective

Query 3000

• Key Results:
  • Autocomplete
  • Increase general BI experience/product https://github.com/PostHog/meta/issues/157
  • Declutter the data warehouse UI and make the features intuitive to find

Data Modeling MVP

• Key Results:
  • Infrastructure decided and implemented
  • Integrating external data with feature flags
  • External data everywhere in insights/persons/cohorts
  • Get billing team to use modeling in posthog for their invoices_with_annual table

Retro

• [ ] dashboard insight variables @Gilbert09 Started, finishing off dashboard rollups of insights to then release to team 2 for testing
• [x] data modeling Beta ui refactoring (making materialized vs regular views really clear when querying, QA canvas view) @EDsCODE
• [X] salesforce custom fields (internal request) @EDsCODE
• [x] Batch Exports investigation around replication lag solution @tomasfarias
• [x] Batch Exports bug fixing @tomasfarias

Priority

eric out second week

• [ ] Ross onboarding task @rossgray
• [ ] release dashboard insight variables @Gilbert09
• [ ] Fix bigquery source @Gilbert09
• [ ] Start new SQL editor @Gilbert09 @EDsCODE
• [ ] Release data modeling @EDsCODE
• [ ] Backfill PR @tomasfarias
• [ ] RFC + starting investigation for batch exports lag @tomasfarias

[dmarticus]

Team Flags

Support hero: @dmarticus, then @jurajmajerik (old feature success team is sharing the same support queue until our newly split teams scale up)

Retro

• Offsite
• Team split
• Support heroics
  • One bad CSAT (see context here, TL;DR customer was grouchy about a perceived outage when we didn't have one, didn't take kindly to my explanation.
• SLAs are all good

Things shipped

• ✅ Added metrics to the new feature flags service to track how well it's working in production without hitting DB, redis, etc
• ✅ Code references hackathon/deep dive on code references feasibility
  • Writeup is here, TL;DR this is valuable but will require heavier lifting than simply using Github's search API. Maybe be worth taking on as a stretch OKR once Haven joins and ramps up.

---

OKRs

1. Ship new feature flags in production and accomplish the 10x scale + 2x speed boost performance targets (ideally decreases infra spend by N% as well, will try to figure out a more specific target going forward)

Sprint

High priority

• Finish cohort matching for new feature flag service
  • support full subset of cohort match operations (not_in, etc) at the service/DB level

Low Priority/Stretch Goal

• Port over existing feature flag integration tests from the django to rust repo (lots of valuable alpha in our existing /decide tests, but this doesn't need to happen this sprint)