spark-operator

[GunSik2]

목적

• Spark operator 적용 시험

사전조건

• RKE 환경구성: #5

절차

1. helm repo 등록

   helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator

2. Operator 설치

   helm install my-release spark-operator/spark-operator --namespace spark-operator --create-namespace

3. rbac 설정

   kubectl create serviceaccount spark
   kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default

4. sample 실행 spark-pi.yaml spark-py-pi.yaml

   kubectl apply -f spark-pi.yaml
   kubectl apply -f spark-py-pi.yaml

5. 실행 결과

   
   $ kubectl get sparkapplications spark-pi 
   NAME       STATUS    ATTEMPTS   START                  FINISH       AGE
   spark-pi   RUNNING   1          2021-09-17T04:42:44Z   <no value>   11s

$ kubectl describe sparkapplications spark-pi Status: Application State: State: RUNNING .. Events: Type Reason Age From Message

---

Normal SparkApplicationAdded 3m2s spark-operator SparkApplication spark-pi was added, enqueuing it for submission Normal SparkApplicationSubmitted 2m59s spark-operator SparkApplication spark-pi was submitted successfully Normal SparkDriverRunning 2m55s spark-operator Driver spark-pi-driver is running Normal SparkExecutorPending 2m49s (x2 over 2m49s) spark-operator Executor spark-pi-3cc1457bf2107d9e-exec-1 is pending Normal SparkExecutorPending 2m49s (x2 over 2m49s) spark-operator Executor spark-pi-3cc1457bf2107d9e-exec-2 is pending Normal SparkExecutorRunning 2m46s spark-operator Executor spark-pi-3cc1457bf2107d9e-exec-1 is running Normal SparkExecutorRunning 2m45s spark-operator Executor spark-pi-3cc1457bf2107d9e-exec-2 is running Normal SparkExecutorCompleted 13s spark-operator Executor spark-pi-3cc1457bf2107d9e-exec-2 completed Normal SparkExecutorCompleted 13s (x2 over 13s) spark-operator Executor spark-pi-3cc1457bf2107d9e-exec-1 completed Normal SparkDriverCompleted 13s (x2 over 13s) spark-operator Driver spark-pi-driver completed Normal SparkApplicationCompleted 13s spark-operator SparkApplication spark-pi completed

$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.43.0.1 443/TCP 27h spark-pi-24da4f7bf25935da-driver-svc ClusterIP None 7078/TCP,7079/TCP,4040/TCP 4m41s spark-pi-ui-svc ClusterIP 10.43.196.240 4040/TCP 4m41s

6. Operator 삭제

helm uninstall my-release kubectl delete serviceaccounts my-release-spark-operator -n spark-operator kubectl delete serviceaccounts default -n spark-operator

참고자료
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

[GunSik2]

ToDo

• 모니터링
• Spark UI
• Advanced Setting :
  • Node affinity 1
  • batch scheduler volcano

참고

• https://dev.to/stack-labs/my-journey-with-spark-on-kubernetes-in-python-2-3-5e8j
• https://docs.d2iq.com/dkp/kommander/1.4/projects/platform-services/platform-services-catalog/kudo-spark/
• https://artifacthub.io/packages/helm/spot/spark-operator
• https://kudo.dev/docs/runbooks/spark/installation.html
• https://github.com/big-data-europe/docker-spark

[GunSik2]

모니터링

./helm install spark-operator incubator/sparkoperator --namespace spark-operator \
--set enableWebhook=true --set enableBatchScheduler=true \
--set ingressUrlFormat="\{\{\$appName\}\}.ingress.cluster.com"