opensource-observer / oso

Measuring the impact of open source software
https://opensource.observer
Apache License 2.0
71 stars 16 forks source link

Dagster + SQLMesh Metrics: Run sqlmesh python models on trino #2444

Open ravenac95 opened 1 week ago

ravenac95 commented 1 week ago

What is it?

The second part of a set of issues for getting Dagster + SQLMesh Metrics running. This is to get sqlmesh running the python models on trino. If, somehow, this is performant enough without any additional work to run the metrics with the special pre-warmed duckdb as a cache + rolling query runner, then we can still accomplish all the metrics work without as much time spent developing a more complex option.

ravenac95 commented 1 week ago

Sadly, after having spent quite a few hours on this (there were some setup things to fix and then bugs in the python models). running this with just the python models as is on trino is still not enough. Some observations when running

Places we will go from here to explore (still not using duckdb):