search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
183 stars 93 forks source link

An error occurred when using HarvesterJsonSerDe.impl.JsonToEntity #624

Closed thbzh closed 1 year ago

thbzh commented 1 year ago

I have a code like the following:

override def send(plan: ExecutionPlan): Unit = {
    import za.co.absa.spline.harvester.json.HarvesterJsonSerDe.impl._
    logInfo("================ fromJson :" + plan.toJson.fromJson[ExecutionPlan].name)
  }

this function fromJson throwed a exception like this:

Exception in thread "main" org.json4s.package$MappingException: No usable value for attributes
No usable value for childRefs
No constructor for type AttrOrExprRef, JObject(List((id,JString(expr-9))))
    at org.json4s.reflect.package$.fail(package.scala:95)
    at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:526)
    at org.json4s.Extraction$ClassInstanceBuilder.$anonfun$instantiate$4(Extraction.scala:546)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
    at org.json4s.Extraction$ClassInstanceBuilder.instantiate(Extraction.scala:546)
    at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:597)
    at org.json4s.Extraction$.$anonfun$extract$9(Extraction.scala:400)
    at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
    at org.json4s.Extraction$.extract(Extraction.scala:392)
    at org.json4s.Extraction$.extract(Extraction.scala:39)
    at org.json4s.ExtractableJsonAstNode.extract(ExtractableJsonAstNode.scala:21)
    at za.co.absa.commons.json.AbstractJsonSerDe$JsonToEntity.fromJson(AbstractJsonSerDe.scala:57)
    at com.thb.spark.spline.udf.Lineage$.main(Lineage.scala:11)
    at com.thb.spark.spline.udf.Lineage.main(Lineage.scala)
Caused by: org.json4s.package$MappingException: No usable value for childRefs
No constructor for type AttrOrExprRef, JObject(List((id,JString(expr-9))))
    at org.json4s.reflect.package$.fail(package.scala:95)
    at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:526)
    at org.json4s.Extraction$ClassInstanceBuilder.$anonfun$instantiate$4(Extraction.scala:546)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
    at org.json4s.Extraction$ClassInstanceBuilder.instantiate(Extraction.scala:546)
    at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:597)
    at org.json4s.Extraction$.$anonfun$extract$9(Extraction.scala:400)
    at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
    at org.json4s.Extraction$.extract(Extraction.scala:392)
    at org.json4s.Extraction$CollectionBuilder.$anonfun$mkCollection$1(Extraction.scala:410)
    at scala.collection.immutable.List.map(List.scala:297)
    at org.json4s.Extraction$CollectionBuilder.mkCollection(Extraction.scala:410)
    at org.json4s.Extraction$CollectionBuilder.result(Extraction.scala:443)
    at org.json4s.Extraction$.$anonfun$extract$8(Extraction.scala:382)
    at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
    at org.json4s.Extraction$.extract(Extraction.scala:382)
    at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:514)
    ... 18 more
Caused by: org.json4s.package$MappingException: No constructor for type AttrOrExprRef, JObject(List((id,JString(expr-9))))
    at org.json4s.reflect.package$.fail(package.scala:95)
    at org.json4s.Extraction$ClassInstanceBuilder.$anonfun$constructor$2(Extraction.scala:477)
    at scala.Option.getOrElse(Option.scala:189)
    at org.json4s.Extraction$ClassInstanceBuilder.constructor(Extraction.scala:477)
    at org.json4s.Extraction$ClassInstanceBuilder.instantiate(Extraction.scala:532)
    at org.json4s.Extraction$ClassInstanceBuilder.result(Extraction.scala:597)
    at org.json4s.Extraction$.$anonfun$extract$9(Extraction.scala:400)
    at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
    at org.json4s.Extraction$.extract(Extraction.scala:392)
    at org.json4s.Extraction$CollectionBuilder.$anonfun$mkCollection$1(Extraction.scala:410)
    at scala.collection.immutable.List.map(List.scala:293)
    at org.json4s.Extraction$CollectionBuilder.mkCollection(Extraction.scala:410)
    at org.json4s.Extraction$CollectionBuilder.result(Extraction.scala:443)
    at org.json4s.Extraction$.$anonfun$extract$8(Extraction.scala:382)
    at org.json4s.Extraction$.customOrElse(Extraction.scala:606)
    at org.json4s.Extraction$.extract(Extraction.scala:382)
    at org.json4s.Extraction$ClassInstanceBuilder.buildCtorArg(Extraction.scala:514)
    ... 39 more

How can I call serialization/deserialization?

thbzh commented 1 year ago

my maven dependencies:

<dependency>
            <groupId>za.co.absa.spline.agent.spark</groupId>
            <artifactId>spark-3.0-spline-agent-bundle_2.12</artifactId>
            <version>1.0.5</version>
            <exclusions>
                <exclusion>
                    <artifactId>scala-library</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-xml_2.12</artifactId>
                    <groupId>org.scala-lang.modules</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>scala-reflect</artifactId>
                    <groupId>org.scala-lang</groupId>
                </exclusion>
            </exclusions>
        </dependency>

my scala version is 2.12.17

cerveada commented 1 year ago

The bundle is not supposed to be used as a maven dependency. You should use agent-core instead.

See https://github.com/AbsaOSS/spline-spark-agent#selecting-artifact

cerveada commented 1 year ago

Why are you converting toJson.fromJson ???

wajda commented 1 year ago

looks like it's a classpath conflict between different json4s versions. Try to analyse your Maven dependency graph and see if there is a conflict.

cerveada commented 1 year ago

@thbzh We removed the dependencies on json libraries, now they are all shaded. So your issue should be solved by using the latest agent version.

Feel free to reopen or create a new issue if you still have a problem.