fmoral2
commented
4 months ago Validated on Version:
-$ rke2 version v1.27.15-rc5+rke2r1 (1ccd6ea04ae76519ec0bad94e30deeecfd0a8674)
Environment Details
Infrastructure Cloud EC2 instance
Node(s) CPU architecture, OS, and Version: ubuntu AMD
Cluster Configuration: -3 node server -1 node agents
- 1 servers
Steps to validate the fix
- Install rke2
- Install helm rancher webhooks
- Join a new node on a upgraded version
- Validate rke2 is up and running
- Validate that no error from webhook is seen in the logs
- Validate pods
Reproduction Issue:
```
rke2 version v1.27.2+rke2r1 (300a06dabe679c779970112a9cb48b289c17536c)
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=rancher.yourdomain.com
kubectl create namespace kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno --namespace kyverno
kubectl get validatingwebhookconfigurations
NAME WEBHOOKS AGE
cert-manager-webhook 1 3m12s
rke2-ingress-nginx-admission 1 21m
rke2-snapshot-validation-webhook 1 21m
validating-webhook-configuration 12 86s
:~> kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
cert-manager-webhook 1 3m21s
mutating-webhook-configuration 9 95s
On a new node joining the cluster upgrading version.
sudo curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.28.8+rke2r1 sh -
sudo journalctl -u rke2-server -f | grep "failed to call webhook"
Jun 21 12:01:31 rke2[2060]: time="2024-06-21T12:01:31Z" level=warning msg="Failed to create Kubernetes secret: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s\": context deadline exceeded"
```
## **Validation Results:**
```
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--set hostname=rancher.yourdomain.com
kubectl create namespace kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno --namespace kyverno
kubectl get validatingwebhookconfigurations
NAME WEBHOOKS AGE
cert-manager-webhook 1 3m12s
rke2-ingress-nginx-admission 1 21m
rke2-snapshot-validation-webhook 1 21m
validating-webhook-configuration 12 86s
:~> kubectl get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
cert-manager-webhook 1 3m21s
mutating-webhook-configuration 9 95s
On a new node joining the cluster upgrading version.
sudo curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION=v1.28.8+rke2r1 sh -
sudo journalctl -u rke2-server -f | grep "failed to call webhook"
<>
$ kubectl get pods -n cattle-system -l app=rancher-webhook
NAME READY STATUS RESTARTS AGE
rancher-webhook-dbdbf746-5fmdq 1/1 Running 0 6m1s
```
Backport fix for RKE2 failing to start: fatal, Failed to apply network policy default-network-ingress-webhook-policy to namespace kube-system
5693