search

kubernetes-sigs / kueue

Kubernetes-native Job Queueing
https://kueue.sigs.k8s.io
Apache License 2.0
1.46k stars 262 forks source link

Document Support Policy on Kueue Releases #3588

Open kannon92 opened 4 days ago

kannon92 commented 4 days ago

As Kueue matures, there is some questions on when Kueue decides that a release is no longer supported. I don't really know when we decide a release is no longer in support.

My hope would be some kind of documented policy on when we stop maintaining release branches and force people to use later versions.

mimowo commented 4 days ago

cc @mwielgus @tenzen-y @mwysokin @dgrove-oss

Good question, I think we generally support only the last released version, but not sure this is documented anywhere.

It may also be a bit tricky to set rigid rules because the release cadence isn't as fixed. In the past we had 1 to 4 months intervals between releases, which could be making some difference also.

kannon92 commented 4 days ago

One rough proposal I talked with @tenzen-y about is maybe linking support to the k8s release version the APIs are based on. This does mean that we are supporting much longer release branches. Generally Kueue is pretty good about upgrading the k8s dependencies once the newest patch of the latest version is released (ie 1.32.1, we upgrade k8s release. Maybe that can be a release?).

For openshift, we offer LTS on k8s release branches and eventually we will end up not being able to support certain kueue releases as the skew increases but I think LTS for Kueue may be linked with the k8s version eventually. I would see folks wanting to run/support Kueue on 1.28 as long as 1.28 is still in support. It isn't clear to me what Kueue release we would support in this case.

dgrove-oss commented 1 day ago

Summarizing my input from the wg call today. Wearing my "consumer of Kueue" hat, the minimum expectation would be that at some point relatively soon we'd get into a steady state where there would always be a release of Kueue that is supported by this community that would work with each version of Kubernetes that is being supported by the Kubernetes community.

This doesn't mean that as a consumer of Kueue I expect the Kueue community to support every Kueue release on every supported version of Kubernetes. What I do expect is:

  1. There is some well-defined structure of Kueue release streams (eg, fast and stable). Not every Kueue release needs to be a stable release.
  2. The stable stream would have always a release that is supported by the Kueue community on each version of Kubernetes that is supported by the Kubernetes community.
  3. If a cluster is running a supported version of Kubernetes, then the Kueue community supports upgrading Kueue "in place" on that cluster from the Nth to N+1 release in Kueue's stable release stream.
kannon92 commented 16 hours ago

Floating this idea with @mrunalp, we are happy with @dgrove-oss's proposal.

I would go a little farther and ask for a formal commitment for n and n - 1 support. And to me, I think that means we should make sure we have periodic tests that verify that n and n - 1 run on supported versions of Kubernetes. I'm happy to help here on the release side. I opened up https://github.com/kubernetes/test-infra/pull/33833 as one option for how this would look in practice.

We will need to revisit this once Kueue hits GA but until that time, I think n and n - 1 support would be sufficient.