search

opendatahub-io-contrib / datamesh-platform

Apache License 2.0
5 stars 5 forks source link

DBT metadata ingestion through OpenMetadata / Airflow #41

Open caldeirav opened 3 months ago

caldeirav commented 3 months ago

Setup OpenMetadata in order to integrate with our DBT pipelines (as code, contained in the data product). List of integrations to be tested: https://docs.open-metadata.org/v1.4.x/connectors/ingestion/workflows/dbt

The DBT ingestion can be configured manually from the OpenMetadata UI as per the following instructions: https://docs.open-metadata.org/v1.4.x/connectors/ingestion/workflows/dbt/configure-dbt-workflow-from-ui

In our case, the manifest.json will be retrived from the GitHub repository for the completed data pipeline, via the File Server method.

caldeirav commented 2 weeks ago

We can successfully configure services and ingestion now in OpenMetadata and the airflow integration is working as expected. See example below for sample data product.

Image

However the dbt ingestion is still not working as per #48