search

apache / incubator-streampark

Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
https://streampark.apache.org/
Apache License 2.0
3.91k stars 1.01k forks source link

[Feature] Support Packaging Spark Docker Images #4133

Open lenoxzhao opened 1 week ago

lenoxzhao commented 1 week ago

What changes were proposed in this pull request

Brief change log

Added KUBERNETES_NATIVE_CLUSTER and KUBERNETES_NATIVE_CLIENT deployment modes. Updated pod template related fields to spark scheme.

cURL Request For Test:

curl --location 'http://localhost:10000/spark/app/create' \
--header 'Authorization: {your authorization}' \
--header 'Cookie: JSESSIONID={your cookie}' \
--form 'jobType="1"' \
--form 'appType="4"' \
--form 'deployMode="4"' \
--form 'versionId="1"' \
--form 'appName="test-packaging3"' \
--form 'resourceFrom="2"' \
--form 'jar="example3"' \
--form 'mainClass="org.apache.spark.examples.SparkPi"' \
--form 'teamId="100000"' \
--form 'k8sMasterUrl="k8s://https://127.0.0.1:6443"' \
--form 'k8sContainerImage="apache/spark:v3.2.1"' \
--form 'k8sServiceAccount="spark"' \
--form 'k8sNamespace="default"' \
--form 'k8sDriverPodTemplate="apiVersion: v1
kind: Pod
metadata:
  name: test-driver-spark
spec:
  initContainers:
  - name: install
    image: busybox
    command:
    - wget
    - \"-O\"
    - \"/var/index.html\"
    - \"https://www.baidu.com\""'

Test Result Log:

16:13:37.958 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[1/7] running => Create building workspace | appName=test-packaging3
16:13:37.960 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Recreate building workspace: /tmp/streampark/workspace/100004/default | appName=test-packaging3
16:13:37.960 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[1/7] success | appName=test-packaging3
16:13:37.961 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[2/7] running => Export kubernetes pod template | appName=test-packaging3
16:13:37.962 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Export spark podTemplates: /tmp/streampark/workspace/100004/default/driver-pod-template.yaml | appName=test-packaging3
16:13:37.962 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[2/7] success | appName=test-packaging3
16:13:37.963 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[3/7] running => Prepare spark job jar | appName=test-packaging3
16:13:37.968 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Prepared spark job jar: /tmp/streampark/workspace/100004/default/spark-examples_2.12-3.1.2.jar | appName=test-packaging3
16:13:37.969 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[3/7] success | appName=test-packaging3
16:13:37.970 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[4/7] running => Export spark app dockerfile | appName=test-packaging3
16:13:37.974 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Output spark dockerfile: /tmp/streampark/workspace/100004/default/Dockerfile, content:
FROM apache/spark:v3.2.1
USER root
RUN mkdir -p $SPARK_HOME/usrlib
COPY spark-examples_2.12-3.1.2.jar $SPARK_HOME/usrlib/spark-examples_2.12-3.1.2.jar
COPY lib $SPARK_HOME/lib/
 | appName=test-packaging3
16:13:37.974 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[4/7] success | appName=test-packaging3
16:13:37.976 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[5/7] running => Pull spark app base docker image | appName=test-packaging3
16:13:48.176 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Already pulled docker image from remote register, imageTag=apache/spark:v3.2.1 | appName=test-packaging3
16:13:48.178 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[5/7] success | appName=test-packaging3
16:13:48.204 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[6/7] running => Build spark app docker image | appName=test-packaging3
16:13:48.371 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Built docker image, imageId=5f694e272ece, imageTag=crpi-xedl4flgdfs4nujg.cn-shenzhen.personal.cr.aliyuncs.com/streampark_lenox/streampark-sparkjob-default-test-packaging3 | appName=test-packaging3
16:13:48.371 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[6/7] success | appName=test-packaging3
16:13:48.374 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[7/7] running => Push spark app docker image | appName=test-packaging3
16:14:02.487 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Already pushed docker image, imageTag=crpi-xedl4flgdfs4nujg.cn-shenzhen.personal.cr.aliyuncs.com/streampark_lenox/streampark-sparkjob-default-test-packaging3 | appName=test-packaging3
16:14:02.488 [streampark-pipeline-watcher-executor-2] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline step[7/7] success | appName=test-packaging3
16:14:02.512 [streampark-build-pipeline-executor-1] INFO org.apache.streampark.flink.packer.pipeline.impl.SparkK8sApplicationBuildPipeline - [StreamPark] [streampark-packer] Building pipeline has finished successfully. | appName=test-packaging3

Command For Image Test:

./spark-submit \
--master k8s://https://127.0.0.1:6443 \
--deploy-mode cluster \
--name spark-pi  \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.executor.request.cores=0.1 \
--conf spark.kubernetes.container.image=crpi-xedl4flgdfs4nujg.cn-shenzhen.personal.cr.aliyuncs.com/streampark_lenox/streampark-sparkjob-default-test-packaging3 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.driver.podTemplateFile={your_local_workspace}/driver-pod-template.yaml \
local:///opt/spark/usrlib/spark-examples_2.12-3.1.2.jar

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

sonarcloud[bot] commented 1 week ago

Quality Gate Failed Quality Gate failed

Failed conditions
13.6% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud