Repeated errors in kueue-manager log when Kueue 0.9 is installed on Kubernetes 1.29

[dgrove-oss]

What happened:

I created a fresh kind 0.24 cluster with Kubernetes 1.27.17. I installed the Kueue 0.9 release on the cluster.

Every 30 seconds (+/-) the kueue-controller-manager log contains the following stanza:

W1108 22:04:17.191809       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.FlowSchema: the server could not find the requested resource
E1108 22:04:17.191842       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:23.165653       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E1108 22:04:23.165688       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:23.370784       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1108 22:04:23.370836       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:52.216220       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1108 22:04:52.216261       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:58.002480       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E1108 22:04:58.002507       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource" logger="UnhandledError"
W1108 22:05:02.844004       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1108 22:05:02.844044       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"

What you expected to happen:

I do not expect to see error logs when running Kueue 0.9.0 on Kubernetes 1.27. This is a regression vs. Kueue 0.8.3.

How to reproduce it (as minimally and precisely as possible):

Create 1.27 cluster with kind.

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.9.0/manifests.yaml

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.27.17
• Kueue version (use git describe --tags --dirty --always): 0.9.0
• Cloud provider or hardware configuration: MacOS (arm64)

[kannon92]

1.27 is out of support in open source. Kueue 0.9.0 was probably built with 1.31 apis so I’m not sure the skew will even be supporting at this point.

[dgrove-oss]

On Kubernetes 1.29 (which I believe is still in support until 2025-02-28), I get a reduced set of repeating error messages but there is still a problem:

W1111 18:40:52.560632       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1111 18:40:52.560667       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1111 18:41:35.548617       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1111 18:41:35.548652       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1111 18:41:36.038435       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1111 18:41:36.038474       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"
W1111 18:42:11.505717       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1111 18:42:11.505754       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1111 18:42:15.087195       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1111 18:42:15.087231       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"

[dgrove-oss]

I verified that on Kubernetes 1.30 there are no error messages in the log.

[dgrove-oss]

retitled issue to only mention supported Kubernetes version (1.29).

[mbobrovskyi]

As I can see here https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/, ValidatingAdmissionPolicy is stable starting from version v1.30. Kueue v0.9 build with v1.31 apis. That's why we have this error on 1.29.

[mimowo]

@mbobrovskyi do you know if this is just a weird logging, or ValidatingAdmissionPolicy does not work in Kueue in 1.29? I would suppose it should still work even while in Beta, but then it is still surprising to see this error.