Open RummySugar opened 5 years ago
libdisni.so
spark.driver.extraJavaOptions -Djava.library.path=/usr/lib/disni/
spark.executor.extraJavaOptions -Djava.library.path=/usr/lib/disni/
Let me know if you need help.
Following your steps, my problem is still unsolved. in my spark-defaults-conf :
spark.driver.extraClassPath /usr/local/spark-rdma/spark-rdma-3.1-for-spark-2.3.0-jar-with-dependencies.jar
spark.executor.extraClassPath /use/local/spark-rdma/spark-rdma-3.1-for-park-2.3.0-jar-with-dependencies.jar
spark.driver.extraJavaOptions -Djava.library.path=/usr/lib/disni/
spark.executor.extraJavaOptions -Djava.library.path=/usr/lib/disni/
spark.shuffle.manager org.apache.spark.shuffle.rdma.RdmaShuffleManager
in my /etc/profile
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib/disni/
export JAVA_LIBRARY_PATH=${JAVA_LIBRARY_PATH}:/usr/lib/disni/
export SPARK_YARN_USER_ENV="JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH,LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
the same
can you give a more detailed configuration method?