apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.11k stars 916 forks source link

Does it support metadata lineage to OpenMetadata #6241

Open smileyboy2019 opened 7 months ago

smileyboy2019 commented 7 months ago

Code of Conduct

Search before asking

Describe the feature

Discovering blood relationships through OpenMetada, but not supporting Spark, Flink, and other blood relationships. If using kyuubi to automatically generate blood relationships by executing ETL on SQL, submitting them to OpenMetada, or providing blood API integration

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

smileyboy2019 commented 7 months ago

Also, I would like to know if the engines supported by kyuubi, such as JDBC, Impala, Presto, etc., all support lineage analysis

pan3793 commented 7 months ago

Lineage extraction is computing engine-specific functionalities, currently, Kyuubi provides a Spark lineage plugin which supports extracting lineage from query plan and sending it to the Spark event bus, and you can consume such lineage events by implementing a Spark listener