search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 761 forks source link

Attempt to run HiBench on JDK 17 or higher versions #742

Open Panxuefeng-loongson opened 5 months ago

Panxuefeng-loongson commented 5 months ago

Hi,

I attempted to evaluate the performance of JDK 17 and even more recent versions, such as JDK 21, within a big data context but faced challenges. Upon reviewing the code, I noticed that the highest specified Spark version is 3.1.1[1]. According to the Spark documentation, the highest JDK version supported by Spark 3.1.1 is JDK 11[2]. Does this imply that the maximum compatible JDK version for HiBench is limited to JDK 11? If I aim to conduct tests on JDK 17 or newer versions, what would be the recommended approach? As this is my initial endeavor with HiBench, I may have misunderstood some aspects, so I welcome any corrections or guidance [1] https://github.com/Intel-bigdata/HiBench/blob/master/sparkbench/pom.xml#L174 [2] https://community.cloudera.com/t5/Community-Articles/Spark-and-Java-versions-Supportability-Matrix/ta-p/383669

carsonwang commented 5 months ago

We haven't tried JDK 17 or newer versions. You can first find out which Spark version supports the JDK version that you want. After that, update HiBench to support that Spark version. You can update the pom file to include a new profile the Spark version.