Support Kubernetes v1.29 - v1.31 or v1.28 - v1.31

What you would like to be added?

I would like to support the kubernetes v1.29 - v1.31, and stop the v1.27 and the v1.28 supporting before we release the final training-operator v1 version.

But, based on the v1.28 deprecation date, we may want to support the 4 Kubernetes versions (v1.28 - v1.31). @kubeflow/wg-training-leads WDYT?

What we need to do:

Upgrade the Go libs versions: https://github.com/kubeflow/training-operator/blob/6965c1a92462d46981071748936eb135a4584f3d/go.mod#L15-L22
Upgrade the CI env versions:
- https://github.com/kubeflow/training-operator/blob/6965c1a92462d46981071748936eb135a4584f3d/.github/workflows/integration-tests.yaml#L29-L55
- https://github.com/kubeflow/training-operator/blob/6965c1a92462d46981071748936eb135a4584f3d/.github/workflows/unittests.yaml#L20-L21
Upgrade the tools versions:
- https://github.com/kubeflow/training-operator/blob/6965c1a92462d46981071748936eb135a4584f3d/Makefile#L89
- https://github.com/kubeflow/training-operator/blob/6965c1a92462d46981071748936eb135a4584f3d/Makefile#L125
Switch the new code-generator: https://github.com/kubeflow/training-operator/blob/6965c1a92462d46981071748936eb135a4584f3d/hack/update-codegen.sh

Note that we should upgrade the versions step by step (1.29 -> 1.30 -> 1.31) above tasks so that we can easily revert the commit once we face the any version specific bugs and regressions.

Why is this needed?

Currently, we support the Kubernetes v1.27 - v1.29, but these versions will / have been deprecated: https://kubernetes.io/releases/ So, we should support newer versions.

EoL for 1.27: 2024-07-08
EoL for 1.28: 2024-10-28
EoL for 1.29: 2025-02-28
EoL for 1.30: 2025-06-28
EoL for 1.31: 2025-10-28

Love this feature?

Give it a 👍 We prioritize the features with most 👍

kubeflow / training-operator