search

open-metadata / OpenMetadata

OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
https://open-metadata.org
Apache License 2.0
5.13k stars 975 forks source link

Improve Dagster integration. Support SDA #9216

Open geoHeil opened 1 year ago

geoHeil commented 1 year ago

https://openmetadata.slack.com/archives/C02B6955S4S/p1670503431014219

The dagster integration is only using op and not SDA lineage.

Therefore, DBT and ingestion tools like Airbyte are not represented correctly.

Please can you ensure that proper SDA (software-defined assets) support is present in OM to ingest the lineage in the same way as dagster has:

geoHeil commented 1 year ago

similar is: https://github.com/open-metadata/OpenMetadata/issues/8495 however, dagster does not seem to be implemented by that change

geoHeil commented 1 year ago

The current integration has some problems:

  1. The lineage view is too small
  2. The assets are not integrated with OM
  3. The assets of dagster (outside of i.e. DBT) but also their lineage from dagster is not mapped to the one in OM
  4. There is a catch_all __ASSET_JOB_0 pipeline which lumps together (including assets) from several (default UI triggered) runs in dagster
PascalStehling commented 8 months ago

Hi,

are there any Plans to include dagster support for higher Versions, as the currently supported Version is Version 1.0.13, which was released in October 2022 and since then there were alot of changes and improvements.

geoHeil commented 8 months ago

I think it is not breaking at least until 1.4

however the support is still limited i.e. the dagster lineage external to DBT / the database is not yet merged with what is retrieved by other ingestors