Closed li7hui closed 6 years ago
Hi, seems that you are running SparkRdma in a local mode (that's basically running executor in a separate thread and no network shuffle involved). Can you please provide command that you use to submit spark job. Basically you need to have:
Let me know if you'll have questions.
Hi, Thanks for quick response. I used following submit scripts, and the problem has been fixed.
./bin/spark-submit --class com.github.ehiggs.spark.terasort.TeraSort --master yarn --deploy-mode cluster /root/spark-terasort-1.0-SNAPSHOT-jar-with-dependencies.jar hdfs:///input hdfs:///output
Hi @li7hui ,
Have you run SparkRDMA in yarn-cluster mode? We have some problems when running the jobs in SparkRDMA because of it always use the wrong IP in another NIC. Do you have multiple network interface cards in the server? Or have you set the LOCAL_SPARK_IP
in spark-env.sh?
Hi, I was trying to run the terasort app on spark rdma. The terasort code is forked from https://github.com/ehiggs/spark-terasort
I can run the terasort datagen code successfully. However, the program fails when it uses the RDMA utilities.![rdmaerror](https://user-images.githubusercontent.com/1718938/45195183-e9cfd700-b245-11e8-87cf-355bcde7e16f.png)
I checked the spark rdma install guide, but find no clue.