ppml with bigdl 2.0 stable version and features

[glorysdj]

• [x] bigdl assembly jar/zip and classpath issue fix [P0]
• [ ] PySpark/Spark on Graphene-SGX [P1]
  • [x] spark pi/wordcount examples
  • [x] spark sql/dataframe api examples
  • [x] spark sql/dataframe e2e examples
  • [ ] bigdl-dllib examples
  • [x] bigdl-orca examples
  • [x] + k8s support
  • [ ] arguments encryption [P3]
  • [ ] filesystem encryption [P3]
• [ ] PySpark/Spark on Occlum [P2]
  • [x] spark pi/wordcount examples
  • [x] spark sql/dataframe api examples
  • [x] spark sql/dataframe e2e examples
  • [x] bigdl-dllib examples
  • [ ] bigdl-orca examples
  • [x] + k8s support
• [x] real time ml on Graphene-SGX [P1]
  • [x] local tests
  • [x] device plugin deploy/undeploy in k8s job
  • [x] k8s integration tests
• [ ] real time ml on Occlum [P1]
  • [x] local tests
  • [x] + k8s support
  • [ ] k8s integration tests
• [ ] document
• [ ] attestation [P2]
  • [x] design
  • [ ] implementation
• [ ] scalability tests [P2]
  • [x] env/driver setup
  • [x] performance analysis
  • [x] spark vfork/framework improvements
• [ ] stability tests [P2]
  • [ ] design
  • [ ] implementation
• [ ] TPCH on Graphene-SGX performance benchmark
  • [x] env/image prepare
  • [x] benchmark
  • [ ] analysis
  • [ ] improvements
• [ ] KMS [P4]

[jason-dai]

Please break down the list into several milestones, and each milestone corresponds to a version merged into github: 1) Time schedule for merging the milestone 2) What features each version supports

[glorysdj]

PySpark/Spark on Graphene-SGX Milestones

• WW46 Basic PySpark and SQL support
• WW47 Basic Spark SQL support
• WW48 K8S support
• WW49 hadoop/hive support
• WW50 BigDL/Orca support
• WW51 xgboost/ncf support

details

1. Basic PySpark and SQL support features:

   • [x] backward compatibility with Graphene pre-1.2-rc1, bigdl 2.0, spark 3.1.2
   • [x] add simple pyworker feature
   • [x] pyspark sql example tests
   • [x] pyspark sql api tests tests:
   • [x] python helloworld/simple numpy
   • [x] pyspark pi
   • [x] pyspark wordcount
   • [x] pyspark sql basic
   • [x] pyspark sql arrow
   • [x] pyspark sql hive
   • [x] pyspark sql dataframe api
   • [x] pyspark sql sqlcontext api
   • [x] pyspark sql GroupedData api
   • [x] ... sql apis
   • [x] pyspark sql end to end example [1 millon]
   • [x] pyspark sql end to end example [50 millon]
   • [x] pyspark sql end to end example [100 millon]

2. Basic Spark SQL support features:

   • [x] backward compatibility with Graphene pre-1.2-rc1, bigdl 2.0, spark 3.1.2
   • [x] spark sql example tests tests
   • [x] examples listed here(https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples/sql)
   • [x] org.apache.spark.examples.SparkPi
   • [x] org.apache.spark.examples.sql.SparkSQLExample
   • [x] org.apache.spark.examples.sql.RDDRelation
   • [x] org.apache.spark.examples.sql.SimpleTypedAggregator
   • [x] org.apache.spark.examples.sql.UserDefinedScalar
   • [x] org.apache.spark.examples.sql.UserDefinedScalar
   • [x] org.apache.spark.examples.sql.UserDefinedTypedAggregation
   • [x] org.apache.spark.examples.sql.UserDefinedTypedAggregation
   • [x] org.apache.spark.examples.sql.UserDefinedUntypedAggregation
   • [x] org.apache.spark.examples.sql.hive.SparkHiveExample
   • X org.apache.spark.examples.sql.SQLDataSourceExample.scala
   • [x] TPCH 1G
   • [x] TPCH 10G

3. K8S support features:

   • [x] backward compatibility with Graphene pre-1.2-rc1, bigdl 2.0, spark 3.1.2 tests:
   • [x] spark pi
   • [x] spark pi single-executor with spark.kubernetes.sgx.enabled=false
   • [x] spark pi multi-executor with spark.kubernetes.sgx.enabled=false
   • [x] spark pi single-executor with spark.kubernetes.sgx.enabled=true
   • [x] spark pi multi-executor with spark.kubernetes.sgx.enabled=true
   • [x] spark pi single-executor with spark.kubernetes.sgx.enabled=true and driver in SGX
   • [x] spark pi multi-executor with spark.kubernetes.sgx.enabled=true and driver in SGX
   • [x] pyspark sql basic
   • [x] multi-executor with spark.kubernetes.sgx.enabled=true
   • [x] multi-executor with spark.kubernetes.sgx.enabled=true and driver in SGX
   • [x] pyspark sql e2e
   • [x] multi-executor with spark.kubernetes.sgx.enabled=true
   • [x] multi-executor with spark.kubernetes.sgx.enabled=true and driver in SGX
   • [x] tpch

4. BigDL/Orca support features:

   • [ ] backward compatibility with Graphene pre-1.2-rc1, bigdl 2.0 tests:
   • [ ] bigdl lenet
   • [ ] orca data
   • [ ] orca tf basic_text_classification.py
   • [ ] orca tf transfer_learning.py
   • [ ] BigDL API
   • [ ] Orca API

[glorysdj]

real time ml on Graphene-SGX Milestones

• WW46 (11-12) local and k8s integration tests finished

[qiyuangong]

Occlum Spark K8s Image with fork patch

• WW46 (11-12) image ready, local test pass

[glorysdj]

define pyspark sql testsets

1. pyspark examples (including pi, wordcount)
2. pyspark sql examples (basic, arrow, hive)
3. pyspark sql api tests (apis)
4. pysaprk e2e examples (customer_profile 100M)

[glorysdj]

define spark sql testsets

1. spark-examples (spark pi)
2. spark-sql-examples(sql_example, sql_RDDRelation=1, sql_SimpleTypedAggregator, sql_UserDefinedScalar, sql_UserDefinedTypedAggregation, sql_UserDefinedUntypedAggregation, SparkHiveExample)
3. spark sql e2e example
4. TPCH 1G/10G