Closed tobegit3hub closed 5 years ago
SparkRDMA uses native off-heap memory. So yes, need to add memoryOverhead
parameter. The size of memoryOverhead depends on the total dataset size, the number of executors, cores per executor.
Thanks @petro-rudenko . Do you have any guide to set the memoryOverhead? For example, if the shuffle read or shuffle write is 10G for each executor, do we need about 10G memoryOverhead for SparkRDMA?
@tobegit3hub you would need 10Gb + # cores / executor spark.shuffle.rdma.MaxBytesInFlight
E.g. 10Gb + 20 cores 500Mb = ~20 Gb.
Thanks for your detailed exploration. That helps a lot @petro-rudenko !
We use
SparkRDMA
by addinglibdisni.so
andspark-rdma-3.1-for-spark-2.3.0-jar-with-dependencies.jar
in command of spark-submit. But some tasks always fail because of the issue ofjava.io.IOException: getCmEvent() failed
. Here is the complete log.It seems to require more memory to run
SparkRDMA
and it may work by adding--conf spark.executor.memoryOverhead=10g
when using this. We are confused about this and how much memory should we add for this task?