Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil

Cian911 commented 1 week ago

Purpose of this PR

Proposed changes:

This is a follow up fix from work already done in https://github.com/kubeflow/spark-operator/pull/2305 by @ChenYi015

This PR adds a check at submission on executors pods if an emptyDir sizeLimit is nil.

Change Category

[x] Bugfix (non-breaking change which fixes an issue)
[ ] Feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that could affect existing functionality)
[ ] Documentation update

Rationale

Following some further testing of an older SparkApplication template wherein the volumeName (see issue https://github.com/kubeflow/spark-operator/issues/2199 for more details) had not been changed, I noticed behaviour wherein only the driver pod would get created and no executor pods would spawn. I believe this is due to the same issue outlined in https://github.com/kubeflow/spark-operator/pull/2305 which would make sense, as the check was only added on the driver pod submission, and not the executors.

Checklist

[x] I have conducted a self-review of my own code.
[ ] I have updated documentation accordingly.
[ ] I have added tests that prove my changes are effective or that my feature works.
[x] Existing unit tests pass locally with my changes.

Additional Notes

google-oss-prow[bot] commented 1 week ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubeflow/spark-operator/blob/master/OWNERS)~~ [ChenYi015] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

ChenYi015 commented 1 week ago

@Cian911 Thanks for fixing this. /lgtm

kubeflow / spark-operator