Description:
When creating a new Impala table using Spark with existing Impala tables, the data lineage is not being added. It seems that the toEntity in the code doesn’t retrieve information for the new table.
Expected Behavior:
in general, if it's a new table, I thought it would be appropriate to add a table to the data service (impala here) inside open metadata and data lineage should be automatically added for the new Impala table created in Spark, ensuring complete metadata lineage.
Actual Behavior:
No data lineage is added for the new table, resulting in incomplete metadata lineage.
Description: When creating a new Impala table using Spark with existing Impala tables, the data lineage is not being added. It seems that the
toEntityin the code doesn’t retrieve information for the new table.Expected Behavior: in general, if it's a new table, I thought it would be appropriate to add a table to the data service (impala here) inside open metadata and data lineage should be automatically added for the new Impala table created in Spark, ensuring complete metadata lineage.
Actual Behavior: No data lineage is added for the new table, resulting in incomplete metadata lineage.
Slack Link
Additional Information: