paddy-hack
commented
2 years ago Curious, rebooting the worker node fixes the Ingress ... :open_mouth:
I have no idea why the Ingress becomes non-functional as a result of the upgrade. I also have no idea how to prevent that from happening during the upgrade but it seems rebooting worker nodes after the upgrade makes the issue go away.
Additional testing with a multi-worker cluster has shown that you can reboot worker nodes one at a time and wait for the rebooting node to become Ready again. This also worked for me in multi-master node cluster.
You may not need to reboot all worker nodes. Even just rebooting worker nodes not associated with the Ingress fixed things for me.
timarandras
lopf
aendi123
wzrdtales
RKE version: Upgrading from
v1.3.10tov1.3.12Docker version: (
docker version,docker infopreferred)docker info
docker version
Operating system and kernel: (
cat /etc/os-release,uname -rpreferred)Using RancherOS v1.5.8
uname -r
cat /etc/os-release
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Using QEMU (
qemu-system-x86_64), version 7.0.0, on Devuan GNU/Linuxdaedalus. The node VMs are started withCluster info
cluster.yml
nodes: - address: 10.11.17.208 user: rancher role: - controlplane - etcd - address: 10.11.17.209 user: rancher role: - workermanifest.yml
apiVersion: apps/v1 kind: Deployment metadata: name: welcome spec: replicas: 1 selector: matchLabels: app: welcome-nginx template: metadata: name: welcome labels: app: welcome-nginx spec: containers: - image: nginx:1.21.6-alpine name: nginx ports: - name: http containerPort: 80 protocol: TCP --- apiVersion: v1 kind: Service metadata: name: welcome spec: ports: - port: 80 targetPort: 80 selector: app: welcome-nginx --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: welcome spec: rules: - host: welcome.example.org http: paths: - path: / pathType: Prefix backend: service: name: welcome port: number: 80Steps to Reproduce:
Log of the RKE v1.3.10 invocation
Adjust IP addresses in
cluster.ymlto match your VMs. Wait for all pods to beRunningorCompletedafter therke upinvocation. Also wait for the Ingress to get an IP address. It should get the IP address of the sole worker node.kubectl -n welcome describe ingress
$ kubectl -n welcome describe ingress Name: welcome Labels: <none> Namespace: welcome Address: 10.11.17.209 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules: Host Path Backends ---- ---- -------- welcome.example.org / welcome:80 (10.42.1.7:80) Annotations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 16m (x2 over 17m) nginx-ingress-controller Scheduled for syncAdd that IP address to your
/etc/hostswith a hostname ofwelcome.example.org. After that thewgetcommand returns the Nginx welcome page.So far, so good.
Now upgrade to RKE v1.3.12, wait for all pods to be
RunningorCompletedand re-run thewgetcommand. It should return the Nginx welcome page again.Log of the RKE v1.3.12 invocation
Results:
The
wgetinvocation does not return the Nginx welcome page. Instead it just sits there and I eventually killed it after about 20 minutes.The Ingress hasn't changed except for its age, of course, which is now older than all the Pods that the
rke updeployed.kubectl -n welcome describe ingress
I can access the Nginx welcome page without trouble from Pods in the cluster's
defaultnamespace. All of the following return the expected page:If I change the Service to use a NodePort, accessing the assigned port on any of the cluster's nodes also works as expected.
Removing the namespace and redeploying, like so
does not change the situation either.