search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
183 stars 93 forks source link

CobrixPlugin failing with newer Cobrix versions #632

Closed cerveada closed 1 year ago

cerveada commented 1 year ago 
23/03/27 07:55:13 ERROR SplineAgent: Unexpected error occurred during lineage processing for application: Pramen Spark App #spark-application-1679903595321
java.lang.NoSuchFieldException: za.co.absa.cobrix.spark.cobol.source.CobolRelation.sourceDir
        at za.co.absa.commons.reflect.ValueExtractor.$anonfun$extract$2(ValueExtractor.scala:39) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at scala.Option.getOrElse(Option.scala:121) ~[scala-library.jar:?]
        at za.co.absa.commons.reflect.ValueExtractor.extract(ValueExtractor.scala:39) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at za.co.absa.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:140) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at za.co.absa.commons.reflect.ReflectionUtils$.extractFieldValue(ReflectionUtils.scala:116) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at za.co.absa.commons.reflect.ReflectionUtils$.extractValue(ReflectionUtils.scala:146) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at za.co.absa.spline.harvester.plugin.embedded.CobrixPlugin$$anonfun$baseRelationProcessor$1.applyOrElse(CobrixPlugin.scala:36) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at za.co.absa.spline.harvester.plugin.embedded.CobrixPlugin$$anonfun$baseRelationProcessor$1.applyOrElse(CobrixPlugin.scala:34) ~[spark-3.3-spline-agent-bundle_2.12-1.0.6.jar:?]
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:168) ~[scala-library.jar:?]

Additional context: Spline Agent: spark-3.3-spline-agent-bundle_2.12-1.0.6.jar Glue 4.0 Spark 3.1.1 Scala 2.12 Cobrix 2.6.4

It seems that the harvester is relying on a field that is no longer there. - sourceDir new version has sourceDirs