search

oap-project / pmem-spill

Spark plug-in package for accelerating Spark runtime spill functions using PMem such as RDD cache PMem extension.
Apache License 2.0
7 stars 5 forks source link

PMEM-SPILL of OAP-1.1 cannot support spark-3.0.1. #38

Open haojinIntel opened 3 years ago

haojinIntel commented 3 years ago

We try to run K-means algorithm by using PMEM-SPILL on spark-3.0.1 while meet the issue like following picture: image PMEM-SPILL overwrite the source code of spark("src/main/scala/org/apache/spark/internal/config/package.scala") and add some configs for PMEM image While the package.scala of spark-3.0.0 is different with the one of spark-3.0.1 and the issue is caused by the package.scala of OAP-1.1 PMEM-SPILL not contains EXECUTOR_ALLOW_SPARK_CONTEXT. image So how can we decouple package.scala of spark? If we cannot decouple this file, we need to adapt to all some versions of spark.

haojinIntel commented 3 years ago

@zhixingheyi-tian @winningsix @yma11 @yeyuqiang Please help to track the issue. Thanks!

JustDoCoder commented 3 years ago

@haojinIntel Now, the pmem-spill project has not been adapted to spark 3.0.1 and only supports Spark 3.0.0. Compared with Spark3.0.0, the configuration items in package.scala of Spark3.0.1 are slightly changed. Similarly, Spark 3.1.1 also has the same problem. I submitted a PR to make pmem-spill support Spark 3.1.1, please refer to #33 .If you have any questions, welcome to discuss and communicate