Robustness to driver pod taking time to create

[Tom-Newton]

Purpose of this PR

Improve reliability. Closes: #2302

Proposed changes:

• Add a grace period for the driver pod to be created.
• Grace period is controlled by a new config option driver-pod-creation-grace-period. Default is 10 seconds.
• Add 3 new tests around reconciling submitted status, including the new grace period functionality. This is the majority of the lines changed.
• Expose the new config option in the helm chart and add a helm test

Change Category

• [ ] Bugfix (non-breaking change which fixes an issue)
• [x] Feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that could affect existing functionality)
• [ ] Documentation update

Checklist

• [x] I have conducted a self-review of my own code.
• [x] I have updated documentation accordingly - I think its sufficient just to add a description on the new argument on the helm chart and regenerate the helm docs.
• [x] I have added tests that prove my changes are effective or that my feature works.
• [x] Existing unit tests pass locally with my changes.

Additional Notes

Some logs of a real example on our prod cluster where the spark application was saved by the grace period added in this PR. Explore-logs-2024-11-11 12_09_13.txt

[google-oss-prow[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubeflow/spark-operator/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[Tom-Newton]

Sorry for the direct ping. @ChenYi015 are you the right person to review this?