I have a case of using delta live tables that runs SQL notebooks.
I cannot add cell of python
sc._jvm.za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking(spark._jsparkSession)
since DLT doesn't support magic like %python
So I created a separate notebook of python - but still fails - although the spline Jar is installed on the cluster
I think that because DLT manages dynamically the clusters - I think that all the installation of jars on the cluster take no effect.
Question
Does someone has experience using spline in Databricks in DLT?
I have a case of using delta live tables that runs SQL notebooks.
I cannot add cell of python
sc._jvm.za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking(spark._jsparkSession)since DLT doesn't support magic like%pythonSo I created a separate notebook of python - but still fails - although the spline Jar is installed on the cluster I think that because DLT manages dynamically the clusters - I think that all the installation of jars on the cluster take no effect.Question
Does someone has experience using spline in Databricks in DLT?