search

CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Track lineage in Waimak #76

Open alexjbush opened 5 years ago

alexjbush commented 5 years ago

It might be nice to track label lineage in Waimak and attach it to some kind of label metadata structure. It would contain the input labels and dependencies that were used to produce this label.

This could then be written out as metadata (table comment) when producing Hive tables. This would produce a simple data lineage for each table.

We could also think of an approach for tracking source files that records in labels came from. This would be tough though as "'input_file_name' does not support more than one sources": https://issues.apache.org/jira/browse/SPARK-18667 https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala#L611

timpharo commented 3 years ago

I would like to second this. It would be nice to be able to track linage through the labels to allow better understanding of lineage and also to allow graphs to be generated showing this. An example:

Graph