search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
185 stars 95 forks source link

Whether to consider adding column levels #742

Open javaht opened 1 year ago

javaht commented 1 year ago

Whether to consider adding column levels ？

wajda commented 1 year ago

What do you mean?

javaht commented 1 year ago

Column level data lineage

wajda commented 1 year ago

Yes, but how exactly would you like it to be represented? What output is missing from the agent?

wajda commented 1 year ago

The output JSON contains attributes definitions. Each attribute has a childRef property that refers to either another attribute (the one it depends on and derives the value from), or an expression that was used to calculate the values for the given attribute. This forms the attribute (or column) level lineage.