DBT Removed Downstream Lineage Edges aren't removed in Openmetadata - Githubissues

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.

https://open-metadata.org

Apache License 2.0

5.43k stars 1.03k forks source link

DBT Removed Downstream Lineage Edges aren't removed in Openmetadata #18411

Open wobu opened 13 hours ago

wobu commented 13 hours ago

Affected module

Ingestion Framework
Lineage UI Module

Describe the bug When ingesting lineages over dbt externally, downstream lineage edges are only added but never removed.

To Reproduce

run model c defined by joins of tables a an b.
ingest dbt lineage into OMD
modify model c to use only table a.
ingest dbt lineage into OMD

Result: we observe both lineages: a -> b, a -> c

Expected behavior

lineage from a -> b
no lineagea -> c

Version:

OS: Docker Container, Kubernetes, Helm
Python version:
OpenMetadata version: 1.5.8
OpenMetadata Ingestion package version: openmetadata-ingestion[dbt]==1.5.8

Additional context

Slack Community Thread: https://openmetadata.slack.com/archives/C02B6955S4S/p1729854317181499

wobu commented 13 hours ago

Draft PR for reproducing the issue: https://github.com/open-metadata/OpenMetadata/pull/18410