Make dbt docs and Apache Superset talk to one another
Odds are rather high that you use dbt together with a visualisation tool. If so, these questions might have popped into your head time to time:
In case your visualisation tool of choice is Supserset, you are in luck!
Using dbt-superset-lineage
, you can:
This will help you:
The package was presented during Coalesce, the annual dbt conference, as a part of the talk From 100 spreadsheets to 100 data analysts: the story of dbt at Slido. Watch a demo in the video below.
pip install dbt-superset-lineage
dbt-superset-lineage
comes with two basic commands: pull-dashboards
and push-descriptions
.
The documentation for the individual commands can be shown by using the --help
option.
It includes a wrapper for Superset API, one only needs to provide
SUPERSET_ACCESS_TOKEN
/SUPERSET_REFRESH_TOKEN
(obtained via /security/login
)
as environment variable or through --superset-access-token
/superset-refresh-token
option.
N.B.
dbt compile
(or dbt run
) against the production profile, not your development profile schema + table
) across
them, specify the database through --dbt-db-name
and/or --superset-db-id
optionsPUT
requests are only supported if CSRF tokens are disabled in Superset (WTF_CSRF_ENABLED=False
).Pull dashboards from Superset and add them as exposures to dbt docs with references to dbt sources and models, making them visible both separately and as dependencies.
N.B.
$ cd jaffle_shop
$ dbt compile # Compile project to create manifest.json
$ export SUPERSET_ACCESS_TOKEN=<TOKEN>
$ dbt-superset-lineage pull-dashboards https://mysuperset.mycompany.com # Pull dashboards from Superset to /models/exposures/superset_dashboards.yml
$ dbt docs generate # Generate dbt docs
$ dbt docs serve # Serve dbt docs
Push model and column descriptions from your dbt docs to Superset as plain text so that they could be viewed in Superset when creating charts.
N.B.:
--superset-refresh-columns
overrides columns.filterable
and columns.groupby
to true
,
because of this issue.→
or <null>
.$ cd jaffle_shop
$ dbt compile # Compile project to create manifest.json
$ export SUPERSET_ACCESS_TOKEN=<TOKEN>
$ dbt-superset-lineage push-descriptions https://mysuperset.mycompany.com # Push descrptions from dbt docs to Superset
Licensed under the MIT license (see LICENSE.md file for more details).