redshift unload lineage cannot output mcp - Githubissues

datahub-project / datahub

The Metadata Platform for your Data and AI Stack

https://datahubproject.io

Apache License 2.0

9.95k stars 2.95k forks source link

redshift unload lineage cannot output mcp #11584

Closed yingyingqiqi closed 3 weeks ago

yingyingqiqi commented 1 month ago

Describe the bug There is an issue with Redshift ingestion; when S3 is used as the downstream (include_unload_lineage), it cannot produce MCP outputs.

Screenshots There is a bug with self.aggregator._is_temp_table ;

s3://xxx is not in all_tables
Incorrect platform so we cannot output the unload lineage mcp.

https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/ingestion/source/redshift/lineage_v2.py#L97-L116

https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/sql_parsing/sql_parsing_aggregator.py#L514-L524

https://github.com/datahub-project/datahub/blob/f147b51fc8113864d3d59268381882dd7ea5d7e4/metadata-ingestion/src/datahub/sql_parsing/sql_parsing_aggregator.py#L1112-L1117

Desktop (please complete the following information):

Version [0.14.1]

jjoyce0510 commented 3 weeks ago

Pending submission of this PR