Closed nephtyws closed 2 months ago
I found below comment in make_lineage_mce
:
Note: this function only supports lineage for dataset aspects. It will not update lineage for any other aspect types.
Is it intended or needs improvement to support other aspects?
For datajobs, lineage is always assigned to the datajob, which points at the upstream/downstream datasets. We have some code samples listed here https://datahubproject.io/docs/api/tutorials/lineage/#add-lineage-to-non-dataset-entities
It will still show up in the UI as you'd expect.
Aha. Thanks for your answer, @hsheth2 !
Describe the bug When I try to create a lineage between Airflow Task (
urn:li:dataJob:(urn:li:dataFlow:()
) and any dataset (e.g.urn:li:dataset:(urn:li:dataPlatform:mysql,database,table,prod
) code gives error as below.I can verify it can be directly created when I try on console, since console using GraphQL call to create lineage, as below. Note that I changed original DAG and task name to dummy.
GMS logs said:
Is it intended or bug, or what is wrong with me? I don't see any clue on API document.
To Reproduce Steps to reproduce the behavior:
Expected behavior Lineage create should be done between Airflow Task and any dataset using Python SDK, without error.
Additional context