[BUG] Report: ScheduledSparkApplication Fails to Execute After Helm Chart Upgrade

[TheDevilDan]

Description : I am deploying my ScheduledSparkApplication and SparkApplication using a custom Helm chart. When I upgrade my Helm chart and change spec.schedule (the execution time or date) of my ScheduledSparkApplication, the new time is correctly reflected in Rancher. However, the ScheduledSparkApplication does not execute when the specified time arrives.

To resolve this issue, I have to manually delete the existing ScheduledSparkApplication, and then perform the Helm upgrade again to recreate them. After this manual process, the applications execute as expected at the new scheduled time.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: ScheduledSparkApplication
metadata:
  name: collect-{{ .nameCollector }}
  namespace: {{ $.Release.Namespace }}
spec:
  schedule: {{ .timeScheduled | quote }}
  concurrencyPolicy: Forbid
  template:
    type: Java
[......................................................]

On my values : 
  timeScheduled: "0 7 * * *"

Steps to Reproduce

1. Deploy a ScheduledSparkApplication using a custom Helm chart.
2. Upgrade the Helm chart and change the spec.schedule (the execution time or date).
3. Check that the new time is correctly reflected in Rancher (UI shows the updated schedule).
4. Wait for the scheduled execution time to arrive.
5. Observe that the Spark scheduled application does not execute at the scheduled time.

Expected Behavior The Spark scheduled application should execute at the new time after upgrading the Helm chart and modifying the schedule, without the need to manually delete and recreate the scheduled Spark application.

Actual Behavior The Spark scheduled application does not execute at the updated time unless the existing scheduled application is deleted and recreated through a Helm upgrade.

Environment Details Kubernetes version: 1.28 Spark-operator version: 2.0.2 Helm chart version: spark-operator-2.0.2 Rancher version: 2.9.2

Possible Cause It seems that the Helm upgrade correctly updates the schedule metadata in the UI (Rancher), but the actual Spark scheduled application does not receive the trigger or update needed to execute the job at the newly set time.

Workaround The current workaround is to manually delete the existing scheduled Spark applications and perform the Helm upgrade again. This recreates the scheduled applications with the correct execution time, allowing them to run as expected.

[TheDevilDan]

Maybe it has something to do with this issue? https://github.com/kubeflow/spark-operator/issues/2285 since you have to remove the resources and upgrade the helm chart to push back the resources? (whatever the status)

[sebschmeister]

Hi, are there any updates on this? I am also facing this issue where a change to spec.schedule does not cause the app to start at the expected time. When changing the schedule it will be logged in the controller with Reconciling ScheduledSparkApplication {"name": ...