PostHog / posthog

🦔 PostHog provides open-source product analytics, session recording, feature flagging and A/B testing that you can self-host.

https://posthog.com

Other

20.76k stars 1.24k forks source link

Sprint March 18 to March 29, 2024 #20890

Closed mariusandra closed 5 months ago

mariusandra commented 6 months ago

Global Sprint Planning

3 things that might take us down

(recency bias says)

zookeeper
hot partitions in replay
security around data ingestion
moving $CUSTOMER from us to eu, moving 900m events. live traffic will also switch
- Watch MSK/Clickhouse < we can put them on coordinator if we need to

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

# Team ___

**Support hero:** ___

## Retro

<!-- Grab the high and low priority items from last time and add whether that item was completed or not -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## OKR

1. OKR, status (red/yellow/green) and action points if yellow/red

### High priority

-

### Low priority / side quests

-

mariusandra commented 6 months ago

The Pro A.-Team 🚌 🚀 ⛓️🎖️

Support hero:

Michael

Other

Julian working 1 week

Retro

All in progress

@Gilbert09
- Data Warehouse tables into Insights.
- General BI/DW 🫡
@thmsobrmlr
- Funnels
@mariusandra
- Continue the HogQL conversion (testing funnels? more trends?)
@Twixes
- Environments
- Dashboard loading bugs, filters vs queries.
@webjunkie
- support

OKR Q1 2024

🟢 = done 🟡 = working on, expect to finish 🔴 = might not make it

HogQL & Data Exploration (Julian, Marius)
- 🟡 --> 🟢 Convert ALL our insights to use HogQL as their base.
- 🟡 Write great docs for HogQL and data exploration nodes.
- 🔴 Improve the type system and get rid of assumeNotNull.
- 🔴 Proactively monitor query performance.
Product Analytics frontend (Thomas, Michael)
- 🔴 Clean things up now that we have PostHog 3000 and HogQL everywhere.
- 🔴 Work on the universal "Explore" view.
BI (Tom)
- 🟡 Support non-event data sources on HogQL insights.
- 🟡 Build the new querying experience, and/or integrate it into the "Explore" view.

Next sprint

Support (Michael)
HogQL trends (Marius)
HogQL funnels (Thomas)
HogQL paths -> funnels (Julian)

pauldambra commented 6 months ago

Team repl

OMG last sprint of the quarter!

Support hero: Paul

a few days leave for Paul
~ 1 week leave for David
2 days leave Manoel (28, 29)

Retro

support has been super busy @pauldambra @marandaneto
fun incident with blobby when paul and ben (old JS client misbehaving) @annikaschmid
iOS session recording not always working with swift ui, similar to jetpack compose problem in Android, will make product less useful for customers @marandaneto
does retro feel rushed @pauldambra

Hang over items from previous sprint

Main Story

🟡 iOS session replay have first captured recording @marandaneto
- we have recordings, can start to PR, but not close enough to putting this for alpha
- screenshots (or images converted to video/gif) still might be a fallback - but way more data and performance impact on app because has to be on main thread
- see complaints about performance in competitors github
✅ 5 Open Source Android apps tested with android replay @pauldambra
- need 1 more by end of the week
- provided lots to work on so definitely a good approach
🟡 validating embeddings usefulness @daibhin @pauldambra
- error clustering might be useful...
- need to get UI out to test internally

Low priority / side quests

✅ RN release a new major version for node/web/RN SDKs @marandaneto
✅ ordering for replay listing @daibhin
capture reporting quota limiting back to replay clients @pauldambra
hoqgl filtering for replay @daibhin

OKR

OKR, status (red/yellow/green) and action points if yellow/red

🟡📱 happy android replay users
- about 1.3k android recordings captured a day
- current approach is:
  - Manoel and Paul do some rounds of testing
  - then we go looking for more user testing
  - once we think their testing won't tell us about things we already know about
🟡 people love replay even more
- still more bug reports than we'd like
- canvas replay is out
- some rrweb bugs patched directly
- some folk are using replay as the main product - even wanting no event capture 🙈
🟡 first class mobile support
- shout outs on twitter for strength of mobile SDKs
- not focussed here, focus is on replay
- but making good progress
- continuing to crank out improvements & features to mobile SDKs

High priority

iOS session recording - release with or without SwiftUI support @marandaneto
5 more Open Source Android apps tested with android replay @pauldambra
part sprint for @daibhin
- test error clustering with PostHog and ideally with some customers

Low priority / side quests

remove user agent parsing from posthog-js @pauldambra
stop running replay clients when over quota @pauldambra
css-in-js lots of support requests here @daibhin

raquelmsmith commented 6 months ago

Team Growth

Retro

Retro items

### High priority - @raquelmsmith - [x] Teams plan - in progress - [ ] 21M events free plan - in progress - need to make RFC for billing, but can start work - [ ] Rethink some activation metrics (remove team member invited, double check rates with diff activities) - in progress - [x] Prep for Zach to start 🙌 - [x] https://github.com/PostHog/billing/pull/471 - @xrdt - [x] Don't drop data for 7 (or maybe 3) days until after it's done - in progress, almost done - Customer trust scores - [x] Experiment - put people directly on insight creation page - [ ] Experiment on wording for "upgrade" / "subscribe" / "add a credit card" on billing page - ready to go - [ ] Let people self-serve to new plans, like the new replay plan so they can get access to the controls

OKR

✅=finished 🟢=on track to finish this quarter 🟡=might not finish 🔴=won't finish

🟡 Improve activation rates for product analytics and session replay
- ✅ Polish onboarding flow with Cory
- 🟢 Run 1 experiment per week
- 🟡 Do an average of 2 customer interviews per week
- 🔴 Redefine and improve activation rates for PA & SR
🟡 Launch pricing for:
- ✅ Teams
- 🟡 Product analytics 21M free

This sprint

High priority

@raquelmsmith
- [ ] Dashboard creation from onboarding - start
- [ ] Fix proration for teams plan customers
- [ ] Support panel - allow submitting of bug report
- [ ] Rethink some activation metrics (remove team member invited, double check rates with diff activities) - in progress
- [ ] Schedule call for evaluating how the Teams plan is doing
  - Negative feedback about projects limits
  - Negative feedback (including internal) about limiting history views
  - Create dashboard in metabase to see how this plan specifically is performing
@xrdt
- [x] Pricing calculator - cap slider at 250M, under final tier say contact us
  - How many people send us over 250M events per month but aren't on special plan
  - Customer with potential price differences here: https://docs.google.com/document/d/1fb6XzDNSOK0G4ICuEQhFtkfDczCIeK6ZNh1dtTA-vU4/edit?disco=AAABJGMkSRM
- [x] Change stripe price for product analytics, highest tier 0.00001 - migrate people who aren't over that volume?
  - New price: https://dashboard.stripe.com/prices/price_1OtaGHEuIatRXSdzIzbyvCym
  - [ ] Run script compare_prices to see who would pay more
- [ ] 21M events free plan - in progress - need to make RFC for billing, but can start work
  - Set up limiting for insights
  - Do we need a plan transition for this?
  - How are we evaluating this?
  - Related: https://github.com/PostHog/posthog/pull/20245
@zlwaterfield
- [x] Reverse proxy panel in onboarding
- [x] GeoIP config option for product analytics
- [ ] Reskin Onboarding product feature tables - https://github.com/PostHog/posthog/pull/20471 - maybe

benjackwhite commented 6 months ago

Team Infra

Support hero: @daniel

Hang over items from previous sprint

🟡 Rollout redirect approach for static assets @danielxnj
- Tested on dev and works, just needs more validation and slow rollouts
🟢 More follow up on SOC2 and compliance with customers @danielxnj
- Followed up with Simon and follow up in Drata around some missing bits around security networking etc.
- Docker image updates
🟢 Get everything onto contour ingress so we can offer Pipeline a way of gradually rolling out capture
- This has taken forever but we are very nearly there (EU is fully on contour)
🟢 Job spec out for SRE hire @benjackwhite
🟢 Attempt Mr Blobby v3 with EBS shared mount volume @benjackwhite @danielxnj
- Attempted but not an option - we're basically backing out of work here as Mr Blobby seems okay for now
- Postgres would be the next potential test scratch DB
🟢 Get docs and JS fully ready for rollout of .i.posthog.com domains
- @benjackwhite to ping marketing and see if we want to announce anything

OKR

💪 Deploy with confidence 🟡
- 🟢 Manually controlling traffic between two versions of a service
- 🟢 Automatic canary deploying with rollbacks based on HTTP
- 🔴 Manual canary deploys for testing scary changes
💰 Keep costs under control ✅
👀 More Observability 🔴
🎯 Soc2 ✅

High priority

Purchase reserved RDS instances @frankh
- why? Risk vs cost works out - to be fully confirmed with pipeline team
OpenVPN cert updates (only going to worry about CI/CD and infra peeps) @danielxnj
- why? Current security risk as they have no expiry
Get app onto Canary deploys now that we are on contour @danielxnj
- why? This is about having safer automatic deployments (based on HTTP errors)
Test out capture-rs traffic with contour @frankh
Get flagged PoC for proper permissions out @benjackwhite

Low priority / side quests

Add alerts for ECR vulnerabilities @danielxnj

robbie-c commented 6 months ago

Team Web Analytics

Off: none

Last sprint

Sessions table merged
Sessions table wired up to HogQL
1/2 queries completed
Backfill script written (not successfully run yet)

Hang over items from previous sprint

Run sessions backfilll script
Write stats table query using sessions table

OKR

Finish the beta to-do list
Find 5 high-ICP reference customers

High priority

Run sessions backfilll script
Write stats table query using sessions table
HogQL sessions table optimisation
Add session filters

Low priority

polish and be responsive to support

EDsCODE commented 6 months ago

Team Data Shack

OKR Q1 2024

Objective

Get EU moved over to new cluster

Key Results:
- Terraform, ansible, data migrated over, backups, housewatch
- Deprecate old cluster

ByConity evaluation

Key Results:
- Load up with all clickhouse data and benchmark

5 reference customers with qualities:

Key Results:
- Paying a reasonable amount
- Tell us they’re delighted
- Not a big edge case
- Using our in house data import pipeline
- Weekly queries on a data warehouse connected table
- Saving queried tables to insights?

Data Warehouse queries are compatible with insights hogQL

Retro

Product

[ ] @EDsCODE data warehouse data in trends breakdown frontend
[ ] @EDsCODE Restore row limit tracking for all integrations
[ ] @EDsCODE Reach out and do user research on current usage (the goal is to figure out what's needed to increase usage)
[x] @Gilbert09 @EDsCODE both prioritized: Ship MVP: View/table linking precursor to linking data warehouse data to person models. Able to filter these like person properties in trends
[x] Person properties filtering
[x] @Gilbert09 better typing from pipeline import jobs
[x] @EDsCODE resolve duplicate table sync bug

Infra

[x] SOC2 support (mostly)
[ ] Adapt current Ansible playbooks to new cluster configs starting with Coordinator (in progress)
[x] TF new US node to test migrating cluster to

High priority

Product

[ ] @EDsCODE Implement data warehouse series logic at hogql layer (vs query by query)
[ ] @Gilbert09 property typings for data warehouse tables (for autocomplete)
[ ] Zendesk integration (helping @annikaschmid dogfood data warehouse product) -> should lead to adding row limits

Infra

[ ] wire up ansible to pull inventory from TF or AWS using tags
[ ] have US instance up with ansible playbook from galaxy
[ ] setup CI for both of the above

xvello commented 6 months ago

Team Pipeline

Off: nil Support: Tiina

Retro

We fell behind on support and migrations and diverted resources to return to a better place.

High priority

[ ] Launch Person IDs on Events for everyone - Ted:
- [x] Manual run of the squash process for two teams - Tomas -> hitting disk space limits on old data, need help here
- [ ] Backfill & enable read path -> delayed by support work, carry over
[ ] Replay overflow implemented (blobby detects, updates redis zset, capture reroutes) - Xavier
- [x] Capture side (rerouting) and infra done
- [ ] Working on plugin-server side (detection) next sprint
[x] Batch Export for cloud to cloud migrations - Brett
[ ] Improve visibility into pipeline (what's in the queue, what's slow) - Brett -> carry over, downgraded after migration work

Low priority / side quests

[ ] Look at how large a squash table can be - Ted
[ ] Distinct ID tombstoning in PG to fix person reuse - Ted -> back to backlog
[ ] Incident follow-ups

OKR

✅=finished 🟢=on track to finish this quarter 🟡=might not finish 🔴=won't finish

✅ Wrapped up Webhooks Delivery system - Brett
✅ Wrapped up capture-rs posthog-js rollout - Xavier
🟢 Wrapped up New Pipeline UI - Tiina
🟡 Wrapped up PoE (with backfills and squashes) - Ted
🟢 Volume spikes on session replay do not hurt other users - Xavier
🟡 Person property updates don't slow down ingestion - Brett
🟢 Requested data deletion is complete and reliably executed going forward - Tiina

High priority

[ ] Person property updates don't slow down ingestion - Brett
[ ] Launch Person IDs on Events for everyone - Ted & Tomás
- [ ] The squash job runs weekly on Sundays (gated on team IDs because irreversible) -> blocked on disk space issues
- [ ] We have metrics on override table size (total size, highest total per team) to correlate with query performance
- [ ] v3 read path with Analytics: overrides are loaded in a CH dictionary available for query, we monitor its resource usage (memory, cpu) as we backfill values
- [ ] Overrides for all teams are backfilled to CH and available for query
- [ ] Read path feature flag rolled out to all teams
[ ] Replay overflow implemented, plugin-server detection part - Xavier

Low priority / side quests

[ ] Pipeline UI available for beta testers - Tiina
[ ] Confirm data deletions run OK now - Tiina
[ ] Incident follow-ups

neilkakkar commented 6 months ago

Team Feat 🚶‍♂️

Support hero: @neilkakkar Days off: Juraj: 0 days Neil: 2 days

Retro

Experiments UI create & view overhaul - @jurajmajerik -> Create done, view in progress
Fix common support issues with surveys, so support is minimal here - @jurajmajerik -> One big issue down, feedback button issues pending -> we'll hide this for now, too half-baked.
Address any incident remediation issues - @neilkakkar -> Most SDKs updated, Ruby/Go in progress
Filter out internal users bug in experiments - @neilkakkar -> Done
Local evaluation of cohorts in PHP SDK - @neilkakkar -> Done

Hang over items from previous sprint

OKRs

🔴 Doubling down on surveys
🟡 Doubling down on experiments

High priority

Experiments UI draft + results view - basic version with all pre-existing functionality we can release - @jurajmajerik
Investigate and fix (or find alternatives) for cohort OOM errors for $bigClients - @neilkakkar
Add property filters to event matching in cohorts - @neilkakkar
Find a good flow for restricting users to exposed users in secondary metrics for experiments - @neilkakkar

Low priority / side quests

Explore experiment metrics easily in insights - @jurajmajerik
Temporal queues for feature success - @neilkakkar