Sprint 1.37.0 2/2 - June 13 to June 24

[marcushyett-ph]

Global Sprint Planning

Retro: Status of Outcomes from Previous Sprint

1. Fix the main known issues regarding reliability and data integrity (@yakkomajuri please update status: good progress, but will need more work for another sprint.) Why: To make the plugin server more reliable and prevent churn from data integrity issues Why now: To solve pressing customer issues and make us comfortable deploying large changes to ingestion
2. Complete the final ingestion-related tasks for persons/groups on events (@yakkomajuri please update status: essentially no progress, deprioritized in favor of the move to MSK, debugging Kafka consumers, and deploying the plugin server split.) Why: To unlock high scale querying Why now: To ensure our ingestion pipeline guarantees hold true in its new proposed state
3. Subscriptions - Scheduling of email reports for any Insight (using export functionality to generate images) (@benjackwhite please update status: UI mostly finished, email management (unsubscribing etc.) still togo. 80% done) (50% of team is on holiday during this sprint)
4. Switch over to using EKS for the web (events and for app) @hazzadous & @guidoiaquinti (please update status: ❓)
5. Move all of session recording parts to ST1 @fuziontech (please update status: ❓)
6. Simplifying events + actions @alexkim205 — Customers struggle to differentiate or group similar events, causing inconsistency in reporting and inaccurate analyses
   • Mockups finalized
   • New definition view/edit detail pages are implemented and released
   • (In progress) API work to return actions alongside event definitions
   • (In progress) Deprecating actions tab in UI
7. Nail FOSS @EDsCODE — This has been stale for several months and people can’t properly use FOSS until the refactor is done. (please update status: ❓)

Retro: What can we do better next sprint?

1. 2. 3. 4. 5.

Support hero this sprint

Week 1: @neilkakkar Week 2: @fuziontech

Plan: Proposed Goals for Next Sprint

Each goal should have a single owner. Owner can only be an engineer.

1. Nail Exports - why? a lot of customer demand for various types of exports (better CSV), subscriptions and image exports @mariusandra
2. Nail FOSS - why? because we're open source (incuding some kind of celebration 🎉) @EDsCODE
3. Simplifying events + actions @alexkim205 — Why? Customers struggle to differentiate or group similar events, causing inconsistency in reporting and inaccurate analyses
4. Nail Session Analysis - why? Bring back session analysis, a ton of customer demand for using recordings and understand how time is spent @rcmarron
5. Gitlab installation (setup snowplow ingestion plugin and hand holding) @fuziontech - Why: Support a valuable customer and collect feedback
6. Blob storage for session recording setup for a team on cloud @hazzadous - Why: Reclaim storage on ClickHouse and reduce load on ClickHouse. Reduce costs
7. Specifically production traffic will be consumed by new infrastructure in parallel (incuding some kind of celebration 🎉) @hazzadous @guidoiaquinti - Why: Move towards a more repeatable infrastructure defined in code. 6 month goal
8. Ensure persons drift between Postgres and ClickHouse is less than 1% @tiina303 - Why: To make the plugin server more reliable and prevent churn from data integrity issues
9. Complete the final ingestion-related tasks for persons/groups on events to get ready for async migration @Twixes - Why: To unlock high scale querying

Team sprint planning

For your team sprint planning copy this template into a comment below for each team.

Team ___

## Retro

<!-- Talk about what went well, what didn't go well and any actions to improve next time -->

- 

## Hang over items from previous sprint

<!-- For each item, decide to re-prioritise (and add below) or deprioritise -->

- Item 1. prioritised/deprioritise

## Planning

<!-- Each item should have a single owner. Owner can only be an engineer. -->

### High priority

-

### Bugs

-

### Low priority / side quests

-

[mariusandra]

Team App East

Retro

• @kappa90 need to improve on some things but was good
• @pauldambra First week great: beach. Second week wasn't part of the sprint due to recording on s3, but it took a lot of time. Didn't make a lot of progress. Should be more honest/aware regarding estimations. Add a "pessimistic estimate" to the planning template.
• @benjackwhite 3 weeks of person-power on subscriptions --> 1 week. Paul and Marius got other things. We didn't treat the sprint goal as the main priority. Should actually try to make progress on the goals for morale. 24h delay with design on subscriptions.
• @mariusandra jumped on a customer question, spent many days, didn't work on sprint goal at all.
• @lottiecoxon didn't feel massively part of the sprint. Got a lot of stuff done (loading animations). Rope in an engineer when working on a product earlier.
• @marcushyett-ph +1 to a bunch of stuff. Support hero is fun.

Hang over items from previous sprint

• Subscriptions. MVP will be behind a flag. Some polish might be needed.

Planning

Nail exports

• via mail (subscriptions)
• different sizes
• export large csvs (object storage)
• via png link? (object storage)
• via slack
  • user must auth somehow... how?
• via iframe
• via react component
  • import { Insight } from '@posthog/insight'
  • <Insight data={responseFromApi} />

High priority

• Nail exports @mariusandra

Bugs

-

Low priority / side quests

-

[yakkomajuri]

Team Ingestion

1. Ensure persons drift between Postgres and ClickHouse is less than 1% @tiina303

Why: To make the plugin server more reliable and prevent churn from data integrity issues

Best estimate at the moment is 6% - we will ship the metric and move the goal post if necessary.

2. Complete the final ingestion-related tasks for persons/groups on events @Twixes

Why: To unlock high scale querying

[fuziontech]

Team Platform Infrastructure

Retro

• @fuziontech - Really great sprint. Kafka in AWS and out of :heroku:!!! Made progress on reclaiming disk space on cluster. Gitlab is setup. Still have loads to do, but feeling optimistic about direction. I'm bummed about Bogdan ghosting us, but I'm very happy with the current state of the DE and SRE pipelines. Just have to keep cranking!
• @hazzadous - It was short - Kafka stuff. We got a reasonable amount done. S3 stuff for session recording (Distraction)
• @guidoiaquinti - Pretty hard thb - Starting to struggle juggling all of the workstreams at once. Kafka was a bit thrashy. Support hero is not sustainable. 6 support hero requests just today. Most of these support hero requests are super cursory. We really need to protect engineering time (since it's the most $). Very appreciative of @hazzadous for being a backstop. Very reactive bitnami breaking

Hang over items from previous sprint

• Having a dev / staging / prod environments setup and running in new AWS account using EKS in parallel with existing ECS cluster. All stateful resources will be shared between the two environments (Kafka, PG, CH, Redis) but traffic will be able to be consumed from new environment @guidoiaquinti & @hazzadous (Status: see below)

why: to move towards our 6 month goals of having infrastructure as code

• @fuziontech - Tool to reclaim storage on cluster

Why: to unblock person on events migration

Planning

High priority

• Gitlab installation (setup snowplow ingestion plugin and hand holding) @fuziontech

Why: Support a valuable customer and collect feedback

• Blob storage for session recording setup for a team on cloud @hazzadous

Why: Reclaim storage on ClickHouse and reduce load on ClickHouse. Reduce costs

• Specifically production traffic will be consumed by new infrastructure in parallel @hazzadous @guidoiaquinti

Why: Move towards a more repeatable infrastructure defined in code. 6 month goal