[EPIC] build a data warehouse product

[jamesefhawkins]

Is your feature request related to a problem?

When users scale, data accuracy and reporting flexibility get more important. As a result, they eventually want to export their customer and product data to a warehouse. This happens for two reasons (1) it means they can query a wider range of data together (2) they can debug data issues.

This means they will still use PostHog as their data pipeline.

However, their business metrics and logic all move to the warehouse too, where they can trust them. We've seen multiple customers in this situation - always higher value ones.

This means, however, their value of PostHog is decreased and the customer has a ton of data engineering to do.

2 options:

• not care - the market is big enough, so if we lose some revenue because of this, and remain "just a pipeline", we'll be fine
• build a warehouse

We aren't here to just be fine...

Describe the solution you'd like

We offer customers a data warehouse. It's simpler for them, it's more valuable, and it's all in one.

The problems this will solve:

• trust in data
• keeps product analytics/feature flags/experiments relevant and easy to use forever - in fact it'd make them more powerful
• no need to set up complex data stack
• no need to hire data engineers but is compatible with them
• gradual on ramp to a warehouse versus rip 'n replace - we've a ton of potential distribution and can actually win

Customers will need:

• Data in Need Revenue/production database/CRM data
  • Stripe / postgres / mysql / hubspot / whateverDB data
  • 80/20 of tools that target companies use
  • there are plenty of OS connectors
• Need to model entities
  • dbt style product in PostHog
  • every entity type (cohorts/groups/events/persons) are a default SQL file that customers can override
• Need to query / visualize data
  • hogql or credentials
• Monitoring / Alerting (?)

We should host ourselves instead of using clickhouse cloud. This reduces vendor risk (them raising prices or going out of business), represents data to just us, and forces us to build clickhouse as a core competency.

Sequencing

• Querying/visualization we’re already working on
• Infrastructure
• ETL
• Modeling entity types (DBT)

How does this go wrong

There's no reason we can't start now, but we will probably need 2-3 clickhouse experts in our org. We've the money to hire them.

Describe alternatives you've considered

we've had a previous suggestion to build translation layer - feels abstracted / error prone / hacky and requires customers to setup the entire stack and has a complex story

Thank you for your feature request – we love each and every one!

[mariusandra]

Describe alternatives you've considered

we've had a previous suggestion to build translation layer - feels abstracted / error prone / hacky and requires customers to setup the entire stack and has a complex story

Are you referring to HogQL -> BigQuery, etc here... or just HogQL in general (aka HogQL -> ClickHouse)?

[lharries]

@mariusandra he's referring to "HogQL -> BigQuery"

For "we've had a previous suggestion to build translation layer - feels abstracted / error prone / hacky and requires customers to setup the entire stack and has a complex story"

I don't think they'd need to setup the entire stack. We'd still provide them clickhouse out the box, but they could move to another backend if needed.

For "feels abstracted / error prone / hacky" - I think it would be feasible to do this well based on a discussion with Marius. Although it's not trivial.

However, overall I agree that the focus should be on making our current system backed by clickhouse the best possible so they don't need to switch warehouse. But I think it's worth keeping the translation layer as a potential future avenue