gouline / dbt-metabase

dbt + Metabase integration
https://pypi.org/project/dbt-metabase/
MIT License
442 stars 63 forks source link

Add dbt-metabase explore command #78

Closed z3z1ma closed 9 months ago

z3z1ma commented 2 years ago

Lets consider our usual flow of

dbt compile -m my_model navigate to compiled model copy paste sql find somewhere to run it to test it OR dbt run -m my_model check in datagrip, dbeaver, dbvisualizer, vscode sql ext, etc. repeat x?... which the latter implies potentially multiple CTAS, tables being dropped and rewritten, etc. or the inability to test in production vs compile and pasting somewhere you can run it.

I think we can easily offer an ability to invoke something like:

dbt-metabase explore my_model

And on our end conceptually, we use a subprocess to invoke dbt compile -m my_model Read the compiled sql from the manifest.json or file. Construct and pass it into a json object that looks something like {dataset_query: "SELECT * FROM ..."} with your compiled model query inputted Base64 encode the json dumped string And construct a URL like this as many Metabase aficionados may be familiar with (its how metabase caches your in progress queries in the editor).

https://my.metabase/question#eyJkYXRhc2V0X3F1ZXJ5Ijp7InR5cGUiO...c2V0dGluZ3MiOnt9fQ==

This will allow, with a simple invocation, using metabase as a front end graphical dbt model explorer without the above workflow being an impedance. I have a similar goal in my project dbt-osmosis which uses a dbt-osmosis workbench invocation with streamlit as a backend for rapidly developing/prototyping and diffing the data output of model changes. I think for Metabase power users, this is a simple and effective alternative too.

We empower users to explore models, all without actually ever asking them for db creds. Interesting right. Opening this now to gauge feedback prior to implementation.

z3z1ma commented 2 years ago

@gouline

I dont think this is too ambitious for a 0.9.0 release along with the other big feat, metrics. These two feats are in complete isolation from the other parts the community is contributing too which is also nice.

Personally, we on my team would use it quite a bit since we often "workbench" queries in Metabase as a convenience over dbeaver, etc. Its very nice to explore results, distributions, etc.

If I take a stab at it, I'd imagine the actual code footprint to be fairly small.

z3z1ma commented 2 years ago

We have liftoff

Screenshot from 2021-11-24 17-24-52

Screenshot from 2021-11-24 17-26-51