search

kubernetes-sigs / descheduler

Descheduler for Kubernetes
https://sigs.k8s.io/descheduler
Apache License 2.0
4.23k stars 645 forks source link

Investigate `kind` replacement for e2e #1391

Open a7i opened 1 month ago

a7i commented 1 month ago

Is your feature request related to a problem? Please describe.

Descheduler release are typically blocked by waiting for a kind node image.

Describe the solution you'd like

  1. Either build our own kind node image
  2. Pursue alternatives such as k3d or minikube, which keep up with latest versions

Describe alternatives you've considered

Stay behind on releases

Additional context

https://github.com/kubernetes-sigs/kind/issues/3589

BenTheElder commented 1 month ago

As an upstream kubernetes project you should really consider testing against upcoming Kubernetes before it releases, by doing kind build node image within the e2e pipeline and then kind create cluster --image=kindest/node:latest (from the locally built image, more in the docs).

EDIT: 1.30 is out anyhow.

damemi commented 1 month ago

Thanks @BenTheElder, that's a great suggestion and I think it's exactly what we've been looking for. It has always felt a bit weird that we were dependent on waiting until after the kind release to test and publish our next release. I didn't know about that option

a7i commented 1 month ago

cc: @pravarag since you wanted to look into this.

I think we could do something like:

docker pull kindest/node:v1.30.0 || build_kind_image_from_source
damemi commented 1 month ago

@a7i I'd say our master branch could always be building from source, with our tag branches using the released version

The downside to that is we run the risk of getting master blocked on bugs from kind though

a7i commented 1 month ago

Wouldn't that still get us blocked on kind image being released? Unless that's intentional?

damemi commented 1 month ago

It would block new PRs to the tagged branch until that image was available, so maybe the release branches could use the conditional switch like you're suggesting

BenTheElder commented 1 month ago

For K/K we're fetching kind from HEAD to stay compatible with any kubernetes breaking changes, but we're also running equivalent CI jobs, it's possible we'd have a breaking change that you'd have found in the release notes.

That said, we're avoiding those as much as possible, and when we're planning one they've generally been pre-announced (like the containerd 2.0 style registry config) in previous release notes similar to Kubernetes style deprecations. This isn't always possible when we have to react to e.g. the runc misc cgroup changes but generally speaking upgrading to those changes should be desired in a typical CI environment.

We're discussing continuous image builds in the future, right now we build releases with https://github.com/kubernetes-sigs/kind/blob/0a7403e49c529d22cacd3a3f3606b9d8a5c16ae7/hack/release/build/push-node.sh which amongst other things is making the images smaller by compiling out dockershim and cloud providers ... which won't be necessary vs standard builds in 1.31+