Attribute dependency not captured for complex plans

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark

https://absaoss.github.io/spline/

Apache License 2.0

185 stars 95 forks source link

Attribute dependency not captured for complex plans #671

Open uday1409 opened 1 year ago

uday1409 commented 1 year ago

We observed that when there are multiple nested queries along with window functions involved, attribute dependency not properly tracked. For example, avg_monthly_purhcase dependency should point to customer_id and purchase_amount but the link is completely broken. Attaching test queries and generated plan..

SampleQueryWithNestedWindowFunction.txt customerprocess_plan.txt

uday1409 commented 1 year ago

I will check if I can fix this when I get time. Able to reproduce and have a test query as well

wajda commented 1 year ago

thank you @uday1409, your contribution is highly appreciated!