search

awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
635 stars 299 forks source link

Error is thrown while creating dynamic frame using Glue 4.0 #180

Closed gunjanshah80 closed 1 year ago

gunjanshah80 commented 1 year ago

Hi Team,

We plan to upgrade our jupyter notebook with Glue 4.0 which is currently perfectly running fine with Glue 1.0. As instructed we built AWSGlueETLPython-4.0.0-jar-with-dependencies.jar and deployed in our /opt/spark/jars folder..

NB: Spark is downloaded from https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-4.0/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0.tgz To setup glue libraries for 4.0, we downloaded it from 'https://github.com/awslabs/aws-glue-libs/archive/refs/tags/v4.0.zip'

The jupyter notebook is launched successfully, however it ran into issue while creating a dynamic frame using below command:

datasource_sef = glueContext.create_dynamic_frame.from_catalog(database = ,table_name = ,transformation_ctx = "glue_sef_table" ,push_down_predicate = "as_of_date='20230209'")

ANTLR Tool version 4.3 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.7.2 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.3 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.7.2 used for parser compilation does not match the current runtime version 4.8java.lang.NoSuchMethodError: org.json4s.CustomSerializer.(Lscala/Function1;Lscala/reflect/Manifest;)V at com.amazonaws.services.glue.util.StringToBoolean$.(JsonOptions.scala:124)

Attaching the entire error for your reference.

glue-err.txt

jessieweiyi commented 1 year ago

Do we have any update on this issue? I am upgrading glue code from 1.0 to 4.0. And I am getting 

E                   py4j.protocol.Py4JJavaError: An error occurred while calling o436.parquet.
E                   : java.lang.NoSuchMethodError: org.json4s.JsonDSL$.pair2Assoc(Lscala/Tuple2;Lscala/Function1;)Lscala/Tuple2;

Wondering whether something is wrong?

gunjanshah80 commented 1 year ago

The issue has now been resolved by copying version of AWS Glue jars. Thanks for reaching out to me. Marking this as Resolved now.

Regards, Gunjan