Closed gunjanshah80 closed 1 year ago
Do we have any update on this issue? I am upgrading glue code from 1.0 to 4.0. And I am getting
E py4j.protocol.Py4JJavaError: An error occurred while calling o436.parquet.
E : java.lang.NoSuchMethodError: org.json4s.JsonDSL$.pair2Assoc(Lscala/Tuple2;Lscala/Function1;)Lscala/Tuple2;
Wondering whether something is wrong?
The issue has now been resolved by copying version of AWS Glue jars. Thanks for reaching out to me. Marking this as Resolved now.
Regards, Gunjan
Hi Team,
We plan to upgrade our jupyter notebook with Glue 4.0 which is currently perfectly running fine with Glue 1.0. As instructed we built AWSGlueETLPython-4.0.0-jar-with-dependencies.jar and deployed in our /opt/spark/jars folder..
NB: Spark is downloaded from https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-4.0/spark-3.3.0-amzn-1-bin-3.3.3-amzn-0.tgz To setup glue libraries for 4.0, we downloaded it from 'https://github.com/awslabs/aws-glue-libs/archive/refs/tags/v4.0.zip'
The jupyter notebook is launched successfully, however it ran into issue while creating a dynamic frame using below command:
datasource_sef = glueContext.create_dynamic_frame.from_catalog(database =
,table_name =
,transformation_ctx = "glue_sef_table"
,push_down_predicate = "as_of_date='20230209'")
ANTLR Tool version 4.3 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.7.2 used for parser compilation does not match the current runtime version 4.8ANTLR Tool version 4.3 used for code generation does not match the current runtime version 4.8ANTLR Runtime version 4.7.2 used for parser compilation does not match the current runtime version 4.8java.lang.NoSuchMethodError: org.json4s.CustomSerializer.(Lscala/Function1;Lscala/reflect/Manifest;)V
at com.amazonaws.services.glue.util.StringToBoolean$.(JsonOptions.scala:124)
Attaching the entire error for your reference.
glue-err.txt