search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
176 stars 90 forks source link

Whether to consider adding column levels #742

Open javaht opened 10 months ago

javaht commented 10 months ago

Whether to consider adding column levels ？

wajda commented 10 months ago

What do you mean?

javaht commented 10 months ago

Column level data lineage

wajda commented 10 months ago

Yes, but how exactly would you like it to be represented? What output is missing from the agent?

wajda commented 10 months ago

The output JSON contains attributes definitions. Each attribute has a childRef property that refers to either another attribute (the one it depends on and derives the value from), or an expression that was used to calculate the values for the given attribute. This forms the attribute (or column) level lineage.