I have a requirement where I would need to scan pyspark notebook but within Azure synapse environment so whenever we run a synapse notebook the lineage for it gets tracked by spline, I am not sure how to achieve this, I have tested spline on local pyspark scripts which is good to go but what and where to deploy on synapse environment is confusing.
Your assistance required and your time is appreciated.
wajda
commented
6 months ago
First, thank you for a wonderful library,
I have a requirement where I would need to scan pyspark notebook but within Azure synapse environment so whenever we run a synapse notebook the lineage for it gets tracked by spline, I am not sure how to achieve this, I have tested spline on local pyspark scripts which is good to go but what and where to deploy on synapse environment is confusing.
Your assistance required and your time is appreciated.
Thanks, Talha