Sprint - Jul 22 to Aug 2, 2024

[pauldambra]

Global Sprint Planning

3 things that might take us down

1. MSK Kafka -> someone shipping something that writes a lot to kafka, b/c we use shared kafka. Could we monitor/alert? Notes: we have alerts on capture latency as a whole, so if it becomes too high we have a runbook. See infra planning (looking at kafka alternatives which might make it easy to scale up "many kafkas")
2. 1-based indexes - need to decide w/ Hog, please check the RFC as this will influence a lot of how functions are written in future.

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

### High priority

-

### Low priority / side quests

-

Sprint name game score

6 out of 10. Barely average. C'mon people.

[pauldambra]

Team Something

Support hero: @pauldambra

• @pauldambra - out a day and a bit
• @marandaneto - out a few days
• @daibhin - out last two days

retro

• 3 and a bit days of incidents and confusing bugs on top made for "lots of fun" - @pauldambra
• basically turned into a one week spring for me 😫 - @pauldambra
• feels good that filters have landed without kerfuffle - @daibhin
• very excited to delete the old filtering code - @daibhin
• still feel like we don't have a handle on how to investigate complex replay bugs @daibhin
• exciting that we've started on pricing changes - @pauldambra and @annikaschmid
• feels hard to coordinate pricing changes, there are a lot of pricing changes changes happening - @annikaschmid
• we're still doing a lot of things at the same time which means we risk half-assing things @pauldambra @daibhin

items from previous sprint

High priority

• 🟡 pricing changes @pauldambra
  • https://github.com/PostHog/product-internal/pull/578
  • chugging away with this, could definitely do this... i think slow thing now is estimating impact and running compare plan tool
• ✅ universal filters
• error tracking
  • ingestion fangling @pauldambra
    • done replay side heatmap and error changes from https://github.com/PostHog/posthog/issues/23395
    • have an example of how we could make ingestion cheaper in https://github.com/PostHog/posthog/pull/23620
  • 🌀 local stack trace demangling
  • ✅ 3 more customer interviews @daibhin
  • ✅ sparklines @daibhin
  • 🌀 start adding assignee/resolve issue states in postgres @daibhin
    • still aiming for friday
    • might delay merging grouped errors though

Low priority / side quests

• we are recruiting beta testers for Android and iOS - (we should have screenshots now) @marandaneto @annikaschmid ✅ mobile support @pauldambra
  • holding the fort while @marandaneto relaxes in Brazil ✅ continue investigating recordings snapshots capture @pauldambra 🟡 finish the rrweb alpha-16 upgrade! @daibhin and @pauldambra
  • maybe waiting for a type fix from rrweb
  • David' CSS parsing changes have been merged, so we might end up waiting for alpha 17

OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

• 🟡 📱Goal 1: People think of PostHog as a mobile solution
• 🟡 🪲 Goal 2: Error tracking in people's hands
• 🟡 ⁉️ Goal 3: Hiring

High priority

• make sure pricing changes have landed @pauldambra
• error tracking
  • if/when new ingestion pipeline
    • add s3 mounted volume @pauldambra
    • add source maps
      • what's the state of the art here
    • demangle posthog stack traces
  • alerting/notifications? @daibhin
    • only if we can CDP this?
      • e.g. slack message when error first happens / an archived error re-occurs
• heatmap ingestion separation? @pauldambra

Low priority / side quests

• replay capture "large messages" @pauldambra
• feels like there is a person filtering bug where we don't get consistent results in replay @daibhin / @pauldambra
• replay bug investigation tooling @pauldambra / @daibhin
  • playback snapshot by snapshot
  • chrome extension avoidance
• session replays & survey following up on the new filtering experience @daibhin

[benjackwhite]

Team Crocodile Dundee Programming

Team Availability

• Marius: 50% (second week off)
• Ben 100%

Retro

• Marius
  • Starting to feel like a great programming exp (Hog)
  • Great to build the Kinesis app and implement missing features
  • Happy about the progress
• Ben
  • Working with Brett is super nice and seems that Webhooks and App Metrics should be ready and working soon
  • Generally lots of good polishing progress
  • Good to get some actual users and feedback (generally feedback was positive and mostly UX based)
  • I made 6 integrations in one day!

Hang over items from previous sprint

• 🟢 Launch the private beta
• 🟡 Get 5 happy users
• 🟢 Ensure the system stays up and running
• 🟡 Hook up the rusty webhook service
• 🟡 Port over all existing destinations
  • not all ported, but Kinesis was complex which meant we ended up with lots of nice STL functions
  • Experience feels pretty good
  • Implement missing language features
• 🟢 Secret management (actual encryption will follow)
• 🟢 Add calculation and limits for resource usage (memory)

OKR

• 🟢 Goal 1: Widespread usage goal
  • Generally available to all customers
  • 5 happy customers (tight feedback loop with them)
  • Get all post-ingestion plugins migrated to Hog Functions
  • Idea: Template gallery (publish your own template for others to use)
  • Scaling work
• 🟡 Goal 2: Messaging V1
  • Build on top of hog functions to have “HogWorkflows”
  • Requirements gathering - what do we need to build here
  • We should be able to replace some (or all) of our customer.io workflows with our product
• 🟢 Goal 3: Hog Functions as a building block
  • Work with other teams to spread understanding of the power of Hog (functions)
  • Generate various use cases for embeddable functions
  • Multiple sources for functions (ActivityLog, InternalEvents, Alerts)
  • More destinations for functions (tracking events, updating person properties)

Sprint plan

Megaissues: CDP & Hog

• Focus on "big Hog problems" @mariusandra
  • [x] Array access (1-based?)
  • [ ] Exceptions
  • [ ] Port over more plugins/apps:
  • [ ] GCS pubsub (& blob?)
  • [ ] Stretch: inline functions/lambdas (map, reduce, each, etc)
• Final work to enable HogFunctions as feature preview @benjackwhite
  • [ ] Hook up to an actual webhook service
  • [ ] Load testing of Hog functions in our own account
  • [ ] Get AppMetrics connected
  • [ ] Native oauth integration (needed for a lot of destinations, e.g. salesforce)
  • [ ] Show function status in the list view and alert when disabled
  • [ ] Test autocapture work (likely works but lets just triple check)

[jurajmajerik]

Team Feature Success

Support hero: @jurajmajerik Days off: Juraj: 0 days Phani: 2-3 days Dylan: 2 days Neil: 0 days

Retro

• event & action based triggers for surveys - actions - @Phanatic ✅
• no-code experiments RFC - @Phanatic ⌛
• Some experiments UX fixes (banners, display MDE next to the progress bar + explain), docs refresh - @jurajmajerik ✅
• Setup instrumentation for flip-flopping problem of experiment significance - @jurajmajerik
• https://github.com/PostHog/posthog/issues/22131 continue on rewrite - @dmarticus ⌛

Hang over items from previous sprint

• no-code experiments RFC - @Phanatic

---

OKRs

1. Make sure feature flags can handle 10x current scale
2. No-code experiments
3. Split out experiments into its own product

High priority

• Experiment confidence intervals @jurajmajerik
• verification framework/service to ensure that rust flags service works the same as decide. @dmarticus
• no-code experiments RFC - @Phanatic

Low priority / side quests / maybe Neil will get to this next year

• Temporal queues for feature success - @neilkakkar

[benjackwhite]

Team Infra

Retro / hangover

• 🟢 Reverse proxy sharding approach + plan @frankh
  • Plan in place and we know high level what to do, just need to do it.
• 🟢 Billing alerts and follow up from incident issues @danielxnj
• 🟡 Postgres issue investigation in EU - bigint etc @danielxnj
  • Generally good but one ID field still needs changing (working with Jams)
  • Worked well on dev but locked up on EU replica (coming up with follow up plan
• 🟡 Start planning out security group changes @danielxnj
  • Testing on dev to lock down DBs as starting point
• 🟢 K8s rbac improvements
  • Rolled out with admin/developer/readonly roles
• 🟢 VPA grafana chart changes in all regions @benjackwhite
• 🟢 Autoscaling based on celery queue depth @benjackwhite
  • Just need to roll it out in EU as well
• 🥳 Testing out Warpstream
  • Brought this into prio due to last sprint MSK issues
  • By end of sprint will have a duplicate replay ingestion stack running on it
  • Will run for some time to validate perf and costs estimates
• 🟡 Infiscal testing

OKR

1. 🦹 Zero-trust security 🟢
2. 🤓 10x Developer Experience 🟡
3. 💪 Every service lives and dies alone
4. 💰 Save big on cost 🟡

High priority

• Actually implement reverse proxy sharding idea @frankh
• Get PoC of Warpstream for both Replay and events in place @frankh
• Postgres EU schema maintenance @danielxnj
• Helmfile is starting to hit its scaling limits and has random failures @danielxnj
• K8s upgrades maintenance @danielxnj
• RDS maintenance @danielxnj
• Roll out celery keda to EU @benjackwhite
• Infiscal decision and plan @benjackwhite @danielxnj

[EDsCODE]

Team Data <->, collecting of Hogs and more

OKR Q2 2024

Objective

Query 3000

• Key Results:
  • Autocomplete
  • Increase general BI experience/product https://github.com/PostHog/meta/issues/157
  • Declutter the data warehouse UI and make the features intuitive to find

Data Modeling MVP

• Key Results:
  • Infrastructure decided and implemented
  • Integrating external data with feature flags
  • External data everywhere in insights/persons/cohorts
  • Get billing team to use modeling in posthog for their invoices_with_annual table

Retro

• [x] finish launching pricing
• [x] finish work for showing errors from syncs @EDsCODE
• [x] Improve error visibility from querying @Gilbert09
• [x] data modeling (TBD after planning meeting) @tomasfarias
• [x] add historical exports to pipeline 3000

High Priority

• [ ] Data Modeling MVP/experimentation. We have some ideas that need to be tested @tomasfarias
• [ ] Data Warehouse Scene UI refactor/improvements @EDsCODE @Gilbert09
  • [ ] https://docs.google.com/document/d/1Qy9e8BLMJKY7mmcrr1tv54mNuDc4cgVctwj_8kAhyec/edit
• [ ] Get a better view of adoption both absolute and relative to broad posthog sign ups @EDsCODE

[zlwaterfield]

Team Growth

Retro

Retro items

- @raquelmsmith (support first week, on-call second week) - Personless events launch - [x] Oversee pricing calc changes, keep iterating until sales feels like it's good and we feel like it works for us as well - [ ] If above is completed, make sure comms are sent out - [ ] Figure out if we an roll default out to everyone - Data warehouse pricing - [x] Launch it for non-beta-users - Dashboard templates in onboarding - [ ] Fix error that happens after creation, merge https://github.com/PostHog/posthog/pull/23069 - [ ] Add flow to launch toolbar from in-app - Hiring - Project-access-on-invites - [x] Do some digging to see what this entails (I don't think it will be involved or difficult) - @zlwaterfield (on call first first week - support second week) - subscribe to all products - [ ] Run backfill for subscribe to all products and notify users - [x] Remove feature flag code for subscribe to all products and cleanup code - stripe startup metadata - [ ] Finish stripe metadata clean - 20-30 left to manual fixes + a few hours of manual checks - [x] Build a startup plan dashboard in dashboard in PostHog - misc - [x] Think through "Free / paid - same feature-set" - [ ] Add at least one E2E SAML test - [ ] Run backfill for starting/ending backfill bug - [ ] Re-run the plans map compare - [x] Deprecate billing v2 (PR done just need to merge)

Q3 Goals

✅=finished 🟡=in progress 🔴=won't finish ⚪=not started

1. 🟡 Make onboarding awesome for Product analytics and Data warehouse (Raquel)
2. ⚪ Support self-serve annual commitments (Zach)
3. 🟡 Dive into the data to understand our billing metrics and customers better (Zach)
4. ✅ Launch pricing for data warehouse (Raquel)
5. 🟡 Hire 2 people (one for billing, one for auth/permissions focus)

This sprint

Time off: @raquelmsmith (July 15-19 and July 26)

• @zlwaterfield
  • [ ] Complete the subscribe to all products backfill (left over - ran into data issues that have been resolved)
  • [ ] Complete the startup plan metadata clean and dashboard (left over)
  • [ ] Add at least one E2E SAML test (left over)
  • [ ] Improve activation error redirects in billing
  • [x] Block customers from resubscribing if they've previously had their sub canceled from failed payments
  • [x] Update support response time copy to use "target response time"
  • [ ] Improve billing limits - store as number, improve validation / error handling in client and server
  • [ ] Startup customer events for customer.io emails when rolling off
  • [ ] Misc plan issues - users on enterprise when they shouldn't be, mis matched tiers, free session replay plans, etc.
• @raquelmsmith
  • [ ] Launch / announce personless
  • [ ] Help w/ session replay pricing update
  • [ ] Fix error that happens after creation, merge https://github.com/PostHog/posthog/pull/23069
  • [ ] Add flow to launch toolbar from in-app
  • [ ] rest tbd ...

[fuziontech]

Team Click Haus, Haus of the Hogs

OKR Q2 2024

Objective

James as a Service -> Clickhouse as a Service

• P0 tasks such as
  • 🟡 Deletes
  • 🟢 Keeping clusters happy
  • 🟢 Provisioning more disks
  • 🟢 Schema Reviews
  • 🟢 Debugging
  • 🟡 Performance
  • 🟢 Backups/Restores
• Decide whether ByConity is the way forward
  • 🟢 Load it with data, set up
  • 🟢 Test performance, test the functionality/compatibility gaps
• IF ByConity works, migrate over to it
  • 🟢 Enumerate all functionality that doesn’t work and update the functions/contribute to ByConity
  • 🟢 Syntax
  • 🟡 If it works on metal, put it in k8s with Karpenter
  • 🟡 Evaluate which nodes we should use
• IF ByConity doesn’t work, reshard US to look like EU cluster
  • 🟡 All clusters (Dev, US, EU) should be consistent in shape and topology. This will make it easier to manage and maintain the clusters and apply learnings from one cluster to another.
  • 🟢 We want all cluster operations to be automated and managed through some form of infra as code that is available in source control.
  • 🟡 Schema management on ClickHouse should be entirely automated and managed through source control with no exceptions. This includes Coordinator schemas.
  • 🟢 We should be able to spin up and down replicas of any cluster with no manual intervention.
  • 🟢 We should be able to upgrade ClickHouse versions with no manual intervention.
  • 🟡 We should have tooling / runbooks for resharding (if we continue down the current coordinator path)

Board

https://github.com/orgs/PostHog/projects/85/views/2

Retro

@Daesgar The good: We have accomplished quite a bit. US cluster is pretty much done. 4 new replicas are available. Just about to the point where we can retire the old nodes. We can start with deletes and test ByConity. Very happy that backups are working - huge peace of mind. We can retire the snapshot nodes.

The bad: Waiting a lot for PRs to get approved (in general). Feels annoying to bug people in dev channel. Developer role in AWS does not have privileges to read IAM roles (we should grant this)

@fuziontech - It feels like we got a lot done and that we've hit our stride. We are both working effectively together getting things done at the same time. It's really nice having follow the sun coverage for CH. Overall 11/10 right now.

Board Snapshot

[Twixes]

Team ___ (placeholder left intentionally)

Support hero: @anirudhpillai (one week by @Twixes, one week by @aspicer)

Time off: @webjunkie (full sprint) + @aspicer (first week)

Retro

• @webjunkie Insight caching state investigation (do we pre-warming? -> inclined to remove). 🟢 New pre-warming rolled out to the largest customers (for when Julian's back: cleaning up after the older InsightCachingState state)
• @skoob13 Probably start experimenting with LLMs on insights (natural text -> query nodes). Prioritized rolling out multiple breakdowns.
• @aspicer Driving along tickets. 🟢 Also, funnels as a user-defined function.
• @Twixes tbd 🟢 Support. Project environments still WIP.
• @thmsobrmlr Support.

OKR

Q3 2024 objectives

1. Rock-solid analytics (@thmsobrmlr + @webjunkie + @aspicer + @anirudhpillai)
   1. 🟢 Legacy Minus – removing legacy insights code so that we can move fast
   2. 🔴 Tests Plus – shipping fewer bugs in the first place.
   3. 🔴 Metrics Plus – catching issues before before users report
   4. 🟡 Performance Plus - eliminating UX pain via maximum query performance/reliability, based on Metrics Plus data
   5. 🟢 Support Plus – sparking joy for users when they’re led to report a bug
2. Answering more product questions, deeper (@thmsobrmlr + @webjunkie + @aspicer + @anirudhpillai)
   1. 🔴 Growth Plus - increasing ease of onboarding, and subsequent retention
   2. 🟡 Analysis Plus - answering more product questions, more deeply
3. 🟡 ArtificialHog (@Twixes + @skoob13) – an LLM-based chat-like interface for answering product questions.

High priority

• @thmsobrmlr Remove filters from the frontend, allowing us to save new insights with query only.
• @skoob13 "First time this event was done by user" filer in Trends. Then, LLM-based insight generation MVP.
• @Twixes 1 week of support. Project environments rolled out internally. Onboarding @annaszell.
• @aspicer Just 1 week of support.

[robbie-c]

Team web analytics

Support hero: @robbie-c

Retro

Sessions v2 backfill finished on EU, I've started migrating some users, planning to change the default later this week.

Sessions attribution explorer has been pretty handy for one customer, it's enabled some customers to give me super detailed feedback on how our attribution works.

I've also made some other improvements to attribution, for example we now support much more traffic referred from mobile apps (e.g. the native android search widget). I wrote a scraper that uses the list of domains we already have, and grabs their .well-known files to see what mobile apps they have.

Tasks

• 🟢 Figure out difference between queries with session v1 and v2
• 🟡 Clear out the support queue
• 🟢 Finish the attribution debug tool
• 🟡 Improve the refresh logic

OKR

1. Make querying fast enough for large customers
2. Heavily requested features
3. Work better with other products
4. Product and growth

High priority

• Figure out and fix #23690 Person profiles not being merged, using recent versions of the JS sdk (maybe 2 different issues, customer facing one is more urgent than posthog facing one)
• Get everyone in EU onto sessions v2, iron out any problems
• Come up with a solution for a few linked issues: people want a default set of filters, and people want it to be easier to separate their sites. There's been a lot of dicussion on this in the past, see https://github.com/PostHog/posthog/issues/18863 https://github.com/PostHog/posthog/issues/12181 https://github.com/PostHog/posthog/issues/20314 <-- I'm leaning towards the first link here but figuring this out is part of the work

Stretch goals

• Add some visual regression tests for web analytics
• Docs changes for the improvements to attribution

Ongoing

• In the background, continue to backfill the sessions table

[tiina303]

Team Pipeline

Off: Tiina 7 days, Xavier 5 days Support: Brett

Retro

High priority

• [x] Hog support for Rusty-Hook (Brett)
• [x] Separate pipeline for $$heatmap events (Xavier, Tiina, Paul)
  • iwarnings, heatmaps and exceptions will all be deployed to prod by eow
• [x] Inline a processEvent plugin (Oliver)
  • likely will ship to by eow

Low priority / side quests

• [x] Collect rdkafka metrics (broker response latency, error rates) for all node producers & consumer (Xavier)
  • likely will ship by eow

OKR

✅=finished 🟢=on track to finish this quarter 🟡=might not finish 🔴=won't finish ✔️=progressed last sprint ; ➡️=planned work for this sprint

🟢➡️ Test Warpstream as PoC and decide whether to do it or not 🟢✔️➡️ Pipeline scalability Improving pipeline throughput 🟢✔️➡️ Help other teams ship fast 🟢➡️ Stretch: better e2e monitoring

High priority

• [ ] Phase out final uses for jobs and scheduler deployments (Xavier)
• [ ] Migrate all semver flattener users to use the inlined plugin and inline some more plugins (Oliver)
• [ ] Rusty hook to use app metrics v2 for debuggability (Brett)
• [ ] https://github.com/PostHog/posthog/issues/23153 (Brett)
• [ ] Improvements to separate pipelines, e.g. dashboard (Tiina)

Low priority / side quests

• [ ] mvp of event and property definitions as a separate service (Oliver)
• [ ] Kafka topic in front of Rusty Hook (Brett)