Clickhouse sqlmesh pipeline

opensource-observer / oso

Measuring the impact of open source software

https://opensource.observer

Apache License 2.0

69 stars 16 forks source link

Clickhouse sqlmesh pipeline #1165

Closed ryscheng closed 4 weeks ago

ryscheng commented 6 months ago

What is it?

dbt pipelines do not span warehouses.

Since we are already looking to do more stuff in Clickhouse, we can use this as a testing ground for sqlmesh. This is very P2 for the time being.

I filed a separate issue to move data in between the 2 pipelines: https://github.com/opensource-observer/oso/issues/1164

ryscheng commented 6 months ago

Deprioritizing this

davidgasquez commented 6 months ago

Probably doesn't cover 100% of the models as Clickhouse needs a somewhat different approach than BigQuery but SQLGlot might be useful for the base models (can run at the Dagster level).

ryscheng commented 2 months ago

column lineage - seeing schema changes
test-deploy / playground in duckdb
generated models per network

ryscheng commented 1 month ago

Going to backlog this for now, sqlmesh is working on Clickhouse integrations, but its not ready yet. There's only BigQuery, duckdb, and postgres

ryscheng commented 1 month ago

Updated scope: @ravenac95 got this working off a sqlmesh branch. Remaining work is to have it orchestrated by Dagster