Open haojinIntel opened 3 years ago
The performance data showed below:
The cluster contains 3 workers and each contains 384G DRAM. The configuration of spark showed below: spark.memory.pmem.extension.enabled true hibench.streambench.spark.checkpointPath /var/tmp spark.storage.unrollMemoryThreshold 1048576 hibench.streambench.spark.receiverNumber 4 spark.yarn.historyServer.address vsr219:18080 spark.memory.pmem.initial.size 450GB spark.executor.extraJavaOptions -Xms50G -XX:InitialBootClassLoaderMetaspaceSize=128m -XX:MetaspaceSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=10 -XX:ParallelGCThreads=10 -XX:ConcGCThreads=10 hibench.yarn.executor.cores 45 spark.executor.memory 90g hibench.streambench.spark.useDirectMode true spark.eventLog.dir hdfs://vsr219:9000/spark-history-server spark.driver.memory 10g spark.eventLog.enabled true spark.memory.spill.pmem.enabled false spark.driver.extraClassPath /opt/Beaver/OAP/oap_jar/pmem-spill-1.1.0-with-spark-3.0.0.jar:/opt/Beaver/OAP/oap_jar/pmem-common-1.1.0-with-spark-3.0.0.jar spark.kryo.unsafe true hibench.yarn.executor.num 6 spark.history.fs.logDirectory hdfs://vsr219:9000/spark-history-server spark.files /opt/Beaver/OAP/oap_jar/pmem-spill-1.1.0-with-spark-3.0.0.jar,/opt/Beaver/OAP/oap_jar/pmem-common-1.1.0-with-spark-3.0.0.jar spark.executor.extraClassPath ./pmem-spill-1.1.0-with-spark-3.0.0.jar:./pmem-common-1.1.0-with-spark-3.0.0.jar spark.history.fs.cleaner.enabled true spark.default.parallelism ${hibench.default.map.parallelism} spark.serializer.bufferedInputStreamSize 4096 hibench.streambench.spark.storageLevel 2 hibench.streambench.spark.batchInterval 100 hibench.spark.master yarn spark.sql.shuffle.partitions 200 spark.history.ui.port 18080 hibench.spark.home /opt/Beaver/spark spark.sql.warehouse.dir hdfs://vsr219:9000/spark-warehouse spark.serializer org.apache.spark.serializer.KryoSerializer spark.memory.pmem.initial.path /mnt/pmem0,/mnt/pmem1 hibench.streambench.spark.enableWAL false
@zhixingheyi-tian @yeyuqiang Please help to tracking the issue.
@haojinIntel,Please refer to this conclusion:RDD cache show low performance comparing with vanilla spark
The performance data showed below:
The cluster contains 3 workers and each contains 384G DRAM. The configuration of spark showed below: spark.memory.pmem.extension.enabled true hibench.streambench.spark.checkpointPath /var/tmp spark.storage.unrollMemoryThreshold 1048576 hibench.streambench.spark.receiverNumber 4 spark.yarn.historyServer.address vsr219:18080 spark.memory.pmem.initial.size 450GB spark.executor.extraJavaOptions -Xms50G -XX:InitialBootClassLoaderMetaspaceSize=128m -XX:MetaspaceSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=10 -XX:ParallelGCThreads=10 -XX:ConcGCThreads=10 hibench.yarn.executor.cores 45 spark.executor.memory 90g hibench.streambench.spark.useDirectMode true spark.eventLog.dir hdfs://vsr219:9000/spark-history-server spark.driver.memory 10g spark.eventLog.enabled true spark.memory.spill.pmem.enabled false spark.driver.extraClassPath /opt/Beaver/OAP/oap_jar/pmem-spill-1.1.0-with-spark-3.0.0.jar:/opt/Beaver/OAP/oap_jar/pmem-common-1.1.0-with-spark-3.0.0.jar spark.kryo.unsafe true hibench.yarn.executor.num 6 spark.history.fs.logDirectory hdfs://vsr219:9000/spark-history-server spark.files /opt/Beaver/OAP/oap_jar/pmem-spill-1.1.0-with-spark-3.0.0.jar,/opt/Beaver/OAP/oap_jar/pmem-common-1.1.0-with-spark-3.0.0.jar spark.executor.extraClassPath ./pmem-spill-1.1.0-with-spark-3.0.0.jar:./pmem-common-1.1.0-with-spark-3.0.0.jar spark.history.fs.cleaner.enabled true spark.default.parallelism ${hibench.default.map.parallelism} spark.serializer.bufferedInputStreamSize 4096 hibench.streambench.spark.storageLevel 2 hibench.streambench.spark.batchInterval 100 hibench.spark.master yarn spark.sql.shuffle.partitions 200 spark.history.ui.port 18080 hibench.spark.home /opt/Beaver/spark spark.sql.warehouse.dir hdfs://vsr219:9000/spark-warehouse spark.serializer org.apache.spark.serializer.KryoSerializer spark.memory.pmem.initial.path /mnt/pmem0,/mnt/pmem1 hibench.streambench.spark.enableWAL false