JDK for Java 17 support in Glue 4.0 dev containter - amazon/aws-glue-libs:glue_libs_4.0.0_image_01

[Des135]

Hi there, I'm trying to run a Glue job locally using glue_libs_4.0.0_image_01 dev container. This Scala script written in Scala 2.12 and references some Java code/classes from an external *.jar provided to the Glue job, written and compiled with Java 17. Glue 4.0 according to documentation runs Spark 3.3.0, which has support of Java 17 https://issues.apache.org/jira/browse/SPARK-33772

When I submit the script+jar to the Docker container, I get the following error in logs:

23/08/29 12:59:53 INFO AnalyticsExportScalaJob$: Executing Scala script..
java.home system property = 1.8.0_372
Exception in thread "main" java.lang.UnsupportedClassVersionError: com/mygluejob/spark/helper/PropertyLoader has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at com.mygluejob.spark.AnalyticsScalaJob$.main(AnalyticsScalaJob.scala:21)
        at com.mygluejob.spark.AnalyticsScalaJob.main(AnalyticsScalaJob.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1006)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1095)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1104)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/08/29 12:59:53 INFO ShutdownHookManager: Shutdown hook called
23/08/29 12:59:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-c0e3cba1-b3b4-47a7-887b-7d66b4fb02d6

Question 1: Does AWS Glue 4.0 runtime support external jars written & compiled in Java 17? (cause underlying Spark 3.3.0 should support it) Question 2: If yes, is it possible to update the glue_libs_4.0.0_image_01 dev container to provide Java 17 support?

Thanks

[aajisaka]

Q1: Does AWS Glue 4.0 runtime support external jars written & compiled in Java 17? (cause underlying Spark 3.3.0 should support it)

No. Glue 4.0 runtime is Java 8 and it doesn't support external jars written & compiled in Java 17.

[Des135]

Thanks for the answer @aajisaka. Also would be grateful if you could point me to any official AWS docs explicitly mentioning Java 8 as the only supported runtime, cause I wasn't able to find it here https://docs.aws.amazon.com/glue/latest/dg/release-notes.html or in any other related docs