apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.29k stars 1.16k forks source link

feat: support submit pyspark once job on k8s and add clusterlabel to combinedlabel #4906

Closed lenoxzhao closed 11 months ago

lenoxzhao commented 11 months ago

What is the purpose of the change

Test command:

linkis-cli --mode once \
-engineType spark-3.2.1 \
-labelMap engineConnMode=once \
-jobContentMap runType='py' \
-sourceMap jobName=OnceJobTest \
-k8sCluster 'K8S-default' \
-confMap spark.app.name='spark-submit-py' \
-confMap spark.app.resource='local:///opt/spark/examples/src/main/python/pi.py' \
-confMap spark.submit.deployMode='cluster' \
-confMap spark.master='k8s-native' \
-confMap linkis.spark.k8s.serviceAccount='spark' \
-confMap linkis.spark.k8s.master.url='k8s://https://172.31.226.155:6443' \
-confMap linkis.spark.k8s.config.file='/home/hadoop/.kube/config' \
-confMap linkis.spark.k8s.imagePullPolicy='IfNotPresent' \
-confMap linkis.spark.k8s.namespace='default' \
-confMap linkis.spark.k8s.image="apache/spark-py:v3.2.1" \
-confMap spark.submit.pyFiles='local:///opt/spark/examples/src/main/python/wordcount.py'

Checklist