Open a140262 opened 1 year ago
We support Spark 3.0 currently, and we are doing the support for Spark 3.2
We have tested it, spark-shuffle-manager-3 is also suitable for Spark 3.2 @a140262
how to compile it for Spark 3.2? these didn't work for me when compiling the CSS .
<java.version>11</java.version>
<spark.version>3.2.0</spark.version>
<hadoop.version>3.2.1</hadoop.version>
Please provide more details on your test.
What would you like to be added?
I am trying to build a CSS docker image to support Spark3.2 based on java11 (FROM amazoncorretto:11 ), not able to compile it so far. How can I make it work with Spark3.2? It is acceptable if CSS works with Spark3.2 & java8. Please help.
Here is the list of version changes in pom.xml file:
Error:
The replication step is:
The jdk11 Dockerfile looks like this:
install maven
RUN yum update -y && yum install -y git tar RUN mkdir -p /usr/share/maven /usr/share/maven/ref \ && curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \ && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \ && rm -f /tmp/apache-maven.tar.gz \ && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn ENV MAVEN_HOME /usr/share/maven ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
git clone https://github.com/bytedance/CloudShuffleService.git /tmp/CloudShuffleService cd /tmp/CloudShuffleService vi pom.xml
./build.sh