search

Mellanox / SparkRDMA

This is archive of SparkRDMA project. The new repository with RDMA shuffle acceleration for Apache Spark is here: https://github.com/Nvidia/sparkucx
Apache License 2.0
240 stars 70 forks source link

Add libdisni.so in wiki to test performance of SparkRDMA #29

Closed tobegit3hub closed 5 years ago

tobegit3hub commented 5 years ago

We have found some issue about the wiki page in https://github.com/Mellanox/SparkRDMA/wiki/Running-HiBench-with-SparkRDMA which may be fixed easily.

In section "Experiment #1: TeraSort", we need to add not only spark-rdma-3.1-for-spark-SPARK_VERSION-jar-with-dependencies.jar but also libdisni.so.

Add to HiBench/conf/spark.conf:
spark.driver.extraClassPath /PATH/TO/spark-rdma-3.1-for-spark-SPARK_VERSION-jar-with-dependencies.jar
spark.executor.extraClassPath /PATH/TO/spark-rdma-3.1-for-spark-SPARK_VERSION-jar-with-dependencies.jar
spark.shuffle.manager org.apache.spark.shuffle.rdma.RdmaShuffleManager
spark.shuffle.compress false
spark.shuffle.spill.compress false
spark.broadcast.compress false
spark.broadcast.checksum false
spark.locality.wait 0

It should be changed to this.

Add to HiBench/conf/spark.conf:
spark.driver.extraLibraryPath /PATH/TO/libdisni.so
spark.executor.extraLibraryPath /PATH/TO/libdisni.so
spark.driver.extraClassPath /PATH/TO/spark-rdma-3.1-for-spark-SPARK_VERSION-jar-with-dependencies.jar
spark.executor.extraClassPath /PATH/TO/spark-rdma-3.1-for-spark-SPARK_VERSION-jar-with-dependencies.jar
spark.shuffle.manager org.apache.spark.shuffle.rdma.RdmaShuffleManager
spark.shuffle.compress false
spark.shuffle.spill.compress false
spark.broadcast.compress false
spark.broadcast.checksum false
spark.locality.wait 0
petro-rudenko commented 5 years ago

On the README page we've mentioned:

libdisni.so must be in java.library.path on every Spark Master and Worker (usually in /usr/lib)

So if libdisni is in default jaava.library.path on each node (e.g. in /usr/lib) then no need to configurre additionally.