search

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Apache License 2.0
2.7k stars 1.35k forks source link

Helm Upgrade Causes the Spark Operator To Stop Working #1554

Open estiller opened 2 years ago

estiller commented 2 years ago

Hi everyone, We install the Spark Operator Helm template (v1.1.24) as part of our system's umbrella chart. Our problem is that we run helm upgrade on the umbrella chart, the upgrade causes the Spark Operator template to recreate the operator's Service Account, Cluster Role and Cluster Role Binding. This is because of the hook-delete-policy introduced in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/1384.

This causes the Spark Operator itself to stop functioning since the pod itself is not recreated/restarted. Since the service account was recreated, its token is different after the upgrade, and the spark operator pod can no longer access the Kubernetes API. You can see it in the logs after it happens:

E0623 01:55:52.879781      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: the server has asked for the client to provide credentials (get [sparkapplications.sparkoperator.k8s.io](http://sparkapplications.sparkoperator.k8s.io/))
E0623 01:55:53.808330      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:55:55.566717      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:55:59.732919      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:56:06.900876      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: the server has asked for the client to provide credentials (get pods)
E0623 01:56:07.875561      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:56:09.434049      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:56:10.088295      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:56:14.792479      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:56:23.007482      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:56:26.308825      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:56:39.562384      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:57:09.297253      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:57:10.760446      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:57:45.396531      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:57:51.592719      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized
E0623 01:58:32.519788      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1beta2.SparkApplication: failed to list *v1beta2.SparkApplication: Unauthorized
E0623 01:58:47.470333      11 reflector.go:127] pkg/mod/[k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156](http://k8s.io/client-go@v0.19.6/tools/cache/reflector.go:156): Failed to watch *v1.Pod: failed to list *v1.Pod: Unauthorized

This is very similar to an issue I found in another project.

The issue is only resolved by manually restarting the pod, or waiting for 60 minutes until the access token is refreshed and the Spark Operator resumes operation.

Our current workaround is to restart the pod after each deployment. This is not ideal as it requires us to run an extra "manual" command after the Helm deployment completes.

Possible solutions include:

  1. Not deleting the service account after every deployment/upgrade.
  2. Restarting the pod after every deployment as described in the Helm documentation.

Assuming that option 2 is the better choice due to Helm hook limitations, I can open a PR implementing this solution. What do you think?

anry007 commented 1 year ago

Proposed solution # 2 can be implemented via setting spark-operator.podAnnotations.checksum/build to constantly changing version in your deployment script, e.g.:

helm upgrade my-umbrella-chart . --set spark-operator.podAnnotations.checksum/build=${my-build-version}

This will trigger spark operator to restart.

julienlau commented 1 year ago

Is it not a problem of out of order operation ? I mean I think the problem is that the seviceaccount is always recreated on helm upgrade. Except sparkoperator pods do not wait for the serviceaccount to be recreated before restarting on upgrade. Then most of the time the pods starts with the previous serviceaccount before it is trashed and recreated...

Maybe not recreating the SA would be a better solution ? I see that the chart already has options :