Update supported Spark and Java versions in installation guide

andygrove commented 3 months ago

What is the problem the feature request solves?

Installation guide currently says:

Requirements:

Apache Spark 3.3, or 3.4
JDK 8 and up
GLIBC 2.17 (Centos 7) and up

We now support Spark 3.5 and have experimental support for 4.0

We do not support JDK 8 with Spark 4 ... we should state which Java versions are supported for each Spark version to avoid confusion

Describe the potential solution

No response

Additional context

No response

adi-kmt commented 2 months ago

Hey @andygrove, could take this up

justahuman1 commented 1 month ago

Hi @adi-kmt are you still working on this? I can take if not. I already have the changes ready, let me know, thanks

https://github.com/justahuman1/datafusion-comet/commit/111a6ba35d83eab2f542085aee160fbf1b1f5297

zemin-piao commented 1 month ago

Hey folks,

I installed the jar https://mvnrepository.com/artifact/org.apache.datafusion/comet-parent-spark3.5_2.12/0.3.0 on my cluster where we run spark 3.5

When running spark with apache datafusion comet jar included, the driver throws this exception:

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.UnsupportedClassVersionError: org/apache/comet/CometRuntimeException has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.CometDriverPlugin.init(Plugins.scala:63)
    at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
    at org.apache.spark.internal.plugin.DriverPluginContainer.<init>(PluginContainer.scala:46)
    at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:210)
    at org.apache.spark.internal.plugin.PluginContainer$.apply(PluginContainer.scala:193)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:579)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:238)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

Seems some classes are compiled by java 11 also for comet jars for spark 3.5 version... May I ask whether this is expected or maybe something wrong with my setup also?

apache / datafusion-comet