search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
183 stars 93 forks source link

AWS Glue 4.0 - NoSuchMethodError #602

Closed Griesbacher closed 1 year ago

Griesbacher commented 1 year ago

Hi,

thanks for the great work!

I'm trying to run the latest agent (spark-3.3-spline-agent-bundle_2.12-1.0.4) with Glue 4.0 on AWS and a pyspark script, which fails in an exception. I was hoping maybe you could give me a hint what to look out for.

I've tested my code with the agent against "vanilla" Spark 3.3.* which works perfectly fine - obviously it cannot be exactly the same code because it is not running as an AWS managed service. Glue 4.0 should use Spark 3.3.0 as stated here: https://docs.aws.amazon.com/glue/latest/dg/release-notes.html

The same code is also working fine with Glue 3 (spark-3.1-spline-agent-bundle_2.12-1.0.4) and Glue 2 (spark-2.2-spline-agent-bundle_2.11-1.0.4).

I tried to create a minimal example for the problem:

Pyspark script:

import dataclasses
import sys
from dataclasses import dataclass

from awsglue.context import GlueContext
from awsglue.utils import getResolvedOptions
from pyspark import SparkConf
from pyspark.context import SparkContext

@dataclass(frozen=True)
class Arguments:
    """Stores and defines the program arguments."""

    source_database: str
    source_table: str
    target_database: str
    target_table: str
    s3_output_path: str
    kms_key_arn: str

def parse_arguments() -> Arguments:
    """Validate and parse the script arguments."""
    field_names = [field.name for field in dataclasses.fields(Arguments)]
    all_arguments = getResolvedOptions(sys.argv, field_names)
    filtered_arguments = {key: value for key, value in all_arguments.items() if key in field_names}
    return Arguments(**filtered_arguments)

def main() -> None:
    """Contain the primary logic of the job."""
    arguments = parse_arguments()
    spark_settings = {
        "spark.spline.mode": "ENABLED",
        "spark.sql.queryExecutionListeners": "za.co.absa.spline.harvester.listener.SplineQueryExecutionListener",
        "spark.spline.lineageDispatcher": "console",
        "spark.spline.lineageDispatcher.console.className": "za.co.absa.spline.harvester.dispatcher.ConsoleLineageDispatcher",
    }
    spark_config = SparkConf().setAll([(key, value) for key, value in spark_settings.items()])
    spark_context = SparkContext(conf=spark_config)
    spark_context._jsc.hadoopConfiguration().set("fs.s3.enableServerSideEncryption", "true")
    spark_context._jsc.hadoopConfiguration().set("fs.s3.serverSideEncryption.kms.keyId", arguments.kms_key_arn)
    spark_context._jsc.hadoopConfiguration().set("fs.s3.canned.acl", "BucketOwnerFullControl")
    glue_context = GlueContext(spark_context)
    glue_context.setConf("spark.sql.parquet.writeLegacyFormat", "true")
    glue_context.setConf("spark.hadoop.fs.s3.maxRetries", "50")

    spark_session = glue_context.spark_session
    query = f"SELECT * FROM {arguments.source_database}.{arguments.source_table}"
    print(query)  # noqa: T201

    data_frame = spark_session.sql(query)
    data_frame = data_frame.drop("Foo").withColumnRenamed("Bar", "Baz")

    print(f"count: {data_frame.count()}")  # noqa: T201

    data_frame.write.option("path", arguments.s3_output_path).mode("overwrite").saveAsTable(
        f"{arguments.target_database}.{arguments.target_table}", format="parquet"
    )
    print("finished")  # noqa: T201

if __name__ == "__main__":
    main()

Glue job config:

{
  "jobConfig": {
    "name": "test_spline_agent_4",
    "description": "",
    "role": "arn:aws:iam::***",
    "command": "glueetl",
    "version": "4.0",
    "workerType": "Standard",
    "numberOfWorkers": 2,
    "maxCapacity": 2,
    "maxRetries": 0,
    "timeout": 2880,
    "maxConcurrentRuns": 10,
    "security": "none",
    "scriptName": "test_spline_agent_4.py",
    "scriptLocation": "s3://***/",
    "language": "python-3",
    "jobParameters": [
      {
        "key": "--kms_key_arn",
        "value": "arn:aws:kms:***",
        "existing": false
      },
     {
        "key": "--s3_output_path",
        "value": "s3://***",
        "existing": false
      },
      {
        "key": "--source_database",
        "value": "***",
        "existing": false
      },
      {
        "key": "--source_table",
        "value": "***",
        "existing": false
      },
      {
        "key": "--target_database",
        "value": "***",
        "existing": false
      },
      {
        "key": "--target_table",
        "value": "***",
        "existing": false
      }
   ],
    "tags": [],
    "jobMode": "DEVELOPER_MODE",
    "developerMode": false,
    "connectionsList": [],
    "temporaryDirectory": "s3://***/tmp",
    "logging": true,
    "glueHiveMetastore": true,
    "metrics": false,
    "dependentPath": "s3://***/spark-3.3-spline-agent-bundle_2.12-1.0.4.jar,s3://***/snakeyaml_1.33.jar",
    "bookmark": "job-bookmark-disable",
    "flexExecution": false,
    "minFlexWorkers": null,
    "pythonPath": null
  },
  "hasBeenSaved": false,
  "script": "***"
}

Log / Error:

INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Initializing Spline Agent...
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Spline Version: 1.0.4 (rev. 4266fca)
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Init Type: AUTO (codeless)
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Init Mode: ENABLED
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Lineage Dispatcher: Console
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Post-Processing Filter: Password replace
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Ignore-Write Detection Strategy: Write metrics
...
INFO [Thread-7] harvester.SparkLineageInitializer (Logging.scala:logInfo(61)): Spline successfully initialized. Spark Lineage tracking is ENABLED.
...
ERROR [spark-listener-group-shared] util.Utils (Logging.scala:logError(98)): throw uncaught fatal error in thread spark-listener-group-shared
java.lang.ExceptionInInitializerError: null
   at za.co.absa.spline.harvester.HashBasedUUIDGenerator.nextId(idGenerators.scala:57) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.DataTypeIdGenerator.nextId(idGenerators.scala:83) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.converter.DataTypeConverter.convert(DataTypeConverter.scala:39) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.agent.SplineAgent$$anon$1$$anon$2.za$co$absa$commons$lang$CachingConverter$$super$convert(SplineAgent.scala:76) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.commons.lang.CachingConverter.$anonfun$convert$1(converters.scala:47) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at scala.collection.mutable.MapLike.getOrElseUpdate(MapLike.scala:206) ~[scala-library.jar:?]
   at scala.collection.mutable.MapLike.getOrElseUpdate$(MapLike.scala:203) ~[scala-library.jar:?]
   at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80) ~[scala-library.jar:?]
   at za.co.absa.commons.lang.CachingConverter.convert(converters.scala:47) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.commons.lang.CachingConverter.convert$(converters.scala:44) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.agent.SplineAgent$$anon$1$$anon$2.convert(SplineAgent.scala:76) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.agent.SplineAgent$$anon$1$$anon$2.convert(SplineAgent.scala:76) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.converter.DataTypeConverter.convert(DataTypeConverter.scala:43) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.converter.AttributeConverter.convert(AttributeConverter.scala:42) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.PlanOperationNodeBuilder$$anon$1.za$co$absa$commons$lang$CachingConverter$$super$convert(PlanOperationNodeBuilder.scala:36) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.commons.lang.CachingConverter.$anonfun$convert$1(converters.scala:47) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at scala.collection.mutable.MapLike.getOrElseUpdate(MapLike.scala:206) ~[scala-library.jar:?]
   at scala.collection.mutable.MapLike.getOrElseUpdate$(MapLike.scala:203) ~[scala-library.jar:?]
   at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80) ~[scala-library.jar:?]
   at za.co.absa.commons.lang.CachingConverter.convert(converters.scala:47) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.commons.lang.CachingConverter.convert$(converters.scala:44) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.PlanOperationNodeBuilder$$anon$1.convert(PlanOperationNodeBuilder.scala:36) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.PlanOperationNodeBuilder.$anonfun$outputAttributes$1(PlanOperationNodeBuilder.scala:66) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) ~[scala-library.jar:?]
   at scala.collection.immutable.List.foreach(List.scala:388) ~[scala-library.jar:?]
   at scala.collection.TraversableLike.map(TraversableLike.scala:233) ~[scala-library.jar:?]
   at scala.collection.TraversableLike.map$(TraversableLike.scala:226) ~[scala-library.jar:?]
   at scala.collection.immutable.List.map(List.scala:294) ~[scala-library.jar:?]
   at za.co.absa.spline.harvester.builder.plan.PlanOperationNodeBuilder.outputAttributes(PlanOperationNodeBuilder.scala:66) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.PlanOperationNodeBuilder.outputAttributes$(PlanOperationNodeBuilder.scala:65) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.read.ReadNodeBuilder.outputAttributes$lzycompute(ReadNodeBuilder.scala:28) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.read.ReadNodeBuilder.outputAttributes(ReadNodeBuilder.scala:28) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.read.ReadNodeBuilder.build(ReadNodeBuilder.scala:42) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.builder.plan.read.ReadNodeBuilder.build(ReadNodeBuilder.scala:28) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.LineageHarvester.$anonfun$harvest$6(LineageHarvester.scala:72) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) ~[scala-library.jar:?]
   at scala.collection.immutable.List.foreach(List.scala:388) ~[scala-library.jar:?]
   at scala.collection.TraversableLike.map(TraversableLike.scala:233) ~[scala-library.jar:?]
   at scala.collection.TraversableLike.map$(TraversableLike.scala:226) ~[scala-library.jar:?]
   at scala.collection.immutable.List.map(List.scala:294) ~[scala-library.jar:?]
   at za.co.absa.spline.harvester.LineageHarvester.$anonfun$harvest$4(LineageHarvester.scala:72) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at scala.Option.flatMap(Option.scala:171) ~[scala-library.jar:?]
   at za.co.absa.spline.harvester.LineageHarvester.harvest(LineageHarvester.scala:65) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.agent.SplineAgent$$anon$1.$anonfun$handle$1(SplineAgent.scala:91) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.agent.SplineAgent$$anon$1.withErrorHandling(SplineAgent.scala:100) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.agent.SplineAgent$$anon$1.handle(SplineAgent.scala:72) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.listener.QueryExecutionListenerDelegate.onSuccess(QueryExecutionListenerDelegate.scala:28) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.listener.SplineQueryExecutionListener.$anonfun$onSuccess$1(SplineQueryExecutionListener.scala:41) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.listener.SplineQueryExecutionListener.$anonfun$onSuccess$1$adapted(SplineQueryExecutionListener.scala:41) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at scala.Option.foreach(Option.scala:257) ~[scala-library.jar:?]
   at za.co.absa.spline.harvester.listener.SplineQueryExecutionListener.onSuccess(SplineQueryExecutionListener.scala:41) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at org.apache.spark.sql.util.ExecutionListenerBus.doPostEvent(QueryExecutionListener.scala:165) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.sql.util.ExecutionListenerBus.doPostEvent(QueryExecutionListener.scala:135) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.sql.util.ExecutionListenerBus.postToAll(QueryExecutionListener.scala:135) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.sql.util.ExecutionListenerBus.onOtherEvent(QueryExecutionListener.scala:147) ~[spark-sql_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:100) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:12) ~[scala-library.jar:?]
   at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) ~[scala-library.jar:?]
   at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1447) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
   at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96) ~[spark-core_2.12-3.3.0-amzn-1.jar:3.3.0-amzn-1]
Caused by: scala.tools.reflect.ToolBoxError: reflective compilation has failed: cannot initialize the compiler due to java.lang.NoSuchMethodError: scala.tools.reflect.package$$anon$4.INFO()Lscala/reflect/internal/Reporter$Severity;
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$api$.liftedTree1$1(ToolBoxFactory.scala:360) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$api$.compiler$lzycompute(ToolBoxFactory.scala:346) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$api$.compiler(ToolBoxFactory.scala:345) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.apply(ToolBoxFactory.scala:372) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.parse(ToolBoxFactory.scala:429) ~[scala-compiler-2.12.15.jar:?]
   at za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.<init>(HarvesterJsonSerDe.scala:49) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.<clinit>(HarvesterJsonSerDe.scala) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   ... 71 more
Caused by: java.lang.NoSuchMethodError: scala.tools.reflect.package$$anon$4.INFO()Lscala/reflect/internal/Reporter$Severity;
   at scala.tools.reflect.package$$anon$4.<init>(package.scala:85) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.package$.frontEndToReporter(package.scala:77) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$api$.liftedTree1$1(ToolBoxFactory.scala:350) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$api$.compiler$lzycompute(ToolBoxFactory.scala:346) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$api$.compiler(ToolBoxFactory.scala:345) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$withCompilerApi$.apply(ToolBoxFactory.scala:372) ~[scala-compiler-2.12.15.jar:?]
   at scala.tools.reflect.ToolBoxFactory$ToolBoxImpl.parse(ToolBoxFactory.scala:429) ~[scala-compiler-2.12.15.jar:?]
   at za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.<init>(HarvesterJsonSerDe.scala:49) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   at za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.<clinit>(HarvesterJsonSerDe.scala) ~[spark-3.3-spline-agent-bundle_2.12-1.0.4.jar:?]
   ... 71 more

The Glue job finishes successfully only the agent fails.

I've noticed that Glue 4 is missing the snake yaml library (https://mvnrepository.com/artifact/org.yaml/snakeyaml/1.33) that's why I added it manually to the path.

I thought the error might be related to https://github.com/AbsaOSS/spline-spark-agent/issues/382. Therefore I checked the Scala version Glue 4 is using. To do so I've added a plugin as descibed here: https://github.com/AbsaOSS/spline-spark-agent#plugin-api, which prints scala.util.Properties.versionNumberString-> 2.12.7, which should be okay?! Just to double check I also recompiled the agent with 2.12.7, but the result was the same error.

I also tested the versions spark-3.3-spline-agent-bundle_2.12-0.7.10 and spark-3.3-spline-agent-bundle_2.12-0.7.13, but it resulted in the same error.

But now I'm out of ideas... Could you point me towards something I could check out?

Best regards, Philip

wajda commented 1 year ago

I've got no idea why is that happening exactly but it seems to have something to do with the scala-compiler minor version mismatch (2.12.7 vs 2.12.15 vs 2.12.17) Theoretically it shouldn't matter, but we aren't living in the ideal world.

We use runtime compilation to overcome API discrepancy between different Json4s and Jackson versions used in different Spark distributions. I guess I'll try to cut the dependency on JSON serde library used in Spark and with that get rid of runtime scala compilation and dependency on scala compiler.

wajda commented 1 year ago

@Griesbacher can you test this build please and see if it fixes the issue? https://teamcity.jetbrains.com/repository/download/OpenSourceProjects_AbsaOSS_SplineAgentSpark_AutoBuildArtifactsSparkScala212/4080044:id/bundle-3.3/target/spark-3.3-spline-agent-bundle_2.12-1.1.0-SNAPSHOT.jar

Griesbacher commented 1 year ago

@wajda this version looks good! The execution plan/event are printed in the logs. I just have to adapt my plugin to the new version, because I use za.co.absa.spline.harvester.json.HarvesterJsonSerDe.impl.EntityToJson which seems to be missing now: java.lang.NoSuchMethodError: za.co.absa.spline.harvester.json.HarvesterJsonSerDe$.impl()Lza/co/absa/commons/json/AbstractJsonSerDe;. But I'll adapt the plugin once version 1.1.0 is available. Thank you very much!

wajda commented 1 year ago

Yes, removing it was the fix :) If in your plugin you have just import HarversterJsonSerDe.impl._ then it's enough to simply recompile your code. The change was done inside the impl itself, it's now object, not a field, but the usage should remain the same.

imjaleel commented 1 year ago

@Griesbacher Hey there. Can you be so kind as to provide me the JAR that helped fixed this issue? I'm facing a similar issue, and would like to build around Spline until the future release with this fix is available.

wajda commented 1 year ago

It's fixed in version 1.1.0 that will be released in the next day or two.