search

Mellanox / SparkRDMA

This is archive of SparkRDMA project. The new repository with RDMA shuffle acceleration for Apache Spark is here: https://github.com/Nvidia/sparkucx
Apache License 2.0
240 stars 70 forks source link

ERROR RdmaNode: libdisni not found! It must be installed within the java.library.path on each Executor and Driver instance #35

Open RummySugar opened 5 years ago

RummySugar commented 5 years ago

can you give a more detailed configuration method?

petro-rudenko commented 5 years ago

Installation:

  1. Download spark-rdma release
  2. In archive you'll find libdisni.so
  3. Put it on every executor in the same folder. Let it be '/usr/lib/disni`
  4. Add to spark next configurations: 
    spark.driver.extraJavaOptions         -Djava.library.path=/usr/lib/disni/
spark.executor.extraJavaOptions    -Djava.library.path=/usr/lib/disni/

    Let me know if you need help.

RummySugar commented 5 years ago

Following your steps, my problem is still unsolved. in my spark-defaults-conf :

spark.driver.extraClassPath /usr/local/spark-rdma/spark-rdma-3.1-for-spark-2.3.0-jar-with-dependencies.jar
spark.executor.extraClassPath /use/local/spark-rdma/spark-rdma-3.1-for-park-2.3.0-jar-with-dependencies.jar
spark.driver.extraJavaOptions -Djava.library.path=/usr/lib/disni/
spark.executor.extraJavaOptions -Djava.library.path=/usr/lib/disni/
spark.shuffle.manager org.apache.spark.shuffle.rdma.RdmaShuffleManager

in my /etc/profile

export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib/disni/
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/lib/disni/
export SPARK_YARN_USER_ENV="JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH,LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
finalTestin commented 5 years ago

the same