search

Intel-bigdata / HiBench

HiBench is a big data benchmark suite.
Other
1.45k stars 761 forks source link

Spark3 and Scala2.12 are not supported when compiling SparkBench #662

Open JustDoCoder opened 3 years ago

JustDoCoder commented 3 years ago

As shown in the title.

minmingzhu commented 3 years ago

@JustDoCoder There is no problem with our previous compilation. Can you give me more details information, for example mvn command and error information.

JustDoCoder commented 3 years ago

@minmingzhu The mvn compilation command is as follows: mvn clean package -Dspark=3.1 -Dscala=2.12 -DskipTests The error information is shown in the figure below: info

minmingzhu commented 3 years ago

Hi, JustDoCoder, we have not enabled spark 3.1+ on HiBench . We have enabled spark 3.0, so you can use this command as follows: mvn clean package -Dspark=3.0 -Dscala=2.12 -DskipTests

JustDoCoder commented 3 years ago

@michaelmior Thanks, are there any plans to enable Spark 3.1.1 in the near future?

michaelmior commented 3 years ago

@JustDoCoder I'm not involved with this project, so unfortunately I don't know. Did you mean to mention @minmingzhu?

JustDoCoder commented 3 years ago

@michaelmior I'm very sorry to bother you

JustDoCoder commented 3 years ago

@minmingzhu Thanks, are there any plans to enable Spark 3.1.1 in the near future?

minmingzhu commented 3 years ago

Yes , In this week, we should be upgrade spark to 3.1.

JustDoCoder commented 3 years ago

Sounds great!

QWQyyy commented 3 years ago

您好，spark3.1.3是还不能用吗，spark集群正常高可用方式搭建，端口没有问题，就是运行不了 spark.conf的master mode设置为yarn-client的话，又识别不出来spark集群，说没有运行在yarn,moses什么上 这个问题是不是我的版本问题，希望解答

image

minmingzhu commented 3 years ago

@QWQyyy Hibench have already enabled spark3.1.X. Your workload didn't run successful, because of you set master mode was wrong. image