Open robcharlwood opened 11 months ago
I don't have any immediate solutions to your problem, but it looks like the root cause is that kube-apiserver cannot communicate with the container running on the cluster.
To verify the problem is not with the webhook, you can validate the webhook configuration was created successfully
kubectl get validatingwebhookconfigurations rancher.cattle.io -o yaml
Your other solution, which is available on Rancher:v2.7-head but has not been released yet would be to have the webhook run on port 443 https://github.com/rancher/rancher/issues/41142#issuecomment-1670123499
@KevinJoiner Thanks! I will check this and get back to you.
@KevinJoiner - So I ran the suggested command and YAML was returned successfully. I can't see anything problematic in the output. Is there anything specific I should be looking for?
@robcharlwood No, if the resource exists and the webhook is not logging any errors we can have higher confidence that the problem is with the connection between the kube-apiserver and the rancher-webhook pod.
I would double checks the steps for adding the firewall rule to make sure it is correctly configured since the symptoms seem to match https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#api_request_that_triggers_admission_webhook_timing_out
You can try to edit the deployment of the Webhook and remove the startupProbe
and livelinessProbe
and see if things start to work. I don't expect this to fix the problem since other requests will most likely time out when you try to create a RoleTemplate, but if it does work, we might have a bug on our side.
@KevinJoiner Thanks! I will investigate and report back!
We are experiencing the same issue.
Firewall rules allow any communication between nodes (trusted)
Adding some extra info:
For the moment being we will try and remove the startupProbe
and livelinessProbe
I have very same issue:
Rancher UI: 2.7.9
RKE version: v1.5.1
K8s: v1.25.16
kubectl describe pod -n cattle-system rancher-webhook-7879bb6c5-vb7ss
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 85s (x28071 over 2d1h) kubelet Startup probe failed: HTTP probe failed with statuscode: 500
Problem started after upgrading from previous version.
I have same issue
Rancher chart: rancher-2.8.1
Rancher webhook chart: rancher-webhook-103.0.1+up0.4.2
Kubernetes: v1.27.13
When try to hit the /healthz
endpoint, I got this log message:
[-]Config Applied failed: reason withheld
healthz check failed
I'm struggle with reason withheld
error because can't find out what the root cause.
Hi
We are seeing a problem with the latest version of the rancher-webhook (0.3.5) when running alongside the latest rancher (2.7.6). In both the Rancher HA cluster and imported K3S and GKE downstream clusters, the webhook pod has a warning about startup probe checks failing with status code 500.
If left for long enough, it eventually starts failing with a liveness probe error:
This is only ever generated as a warning and the pod itself never becomes unhealthy. The pod itself also does not give any useful logs:
This rancher is deployed in the following manner:
Any help or advice on this issue would be appreciated.
Many thanks!