search

clincha-org / clincha

Configuration and monitoring of clinch-home infrastructure
https://clinch-home.com
1 stars 1 forks source link

Unable to reach nginx on all nodes when using nodeport #53

Closed clincha closed 1 year ago

clincha commented 1 year ago

When I try and reach the NGINX pod from nodes that aren't hosting it I get an HTTP error. 

[clincha@bri-runner-01 ~]$ curl 192.168.1.24:30689
curl: (7) Failed to connect to 192.168.1.24 port 30689: No route to host
[clincha@bri-runner-01 ~]$ curl 192.168.1.21:30689
curl: (7) Failed to connect to 192.168.1.21 port 30689: No route to host
[clincha@bri-runner-01 ~]$ curl 192.168.1.22:30689
curl: (7) Failed to connect to 192.168.1.22 port 30689: No route to host
[clincha@bri-runner-01 ~]$ curl 192.168.1.2:30689
3^C
[clincha@bri-runner-01 ~]$ curl 192.168.1.23:30689
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.23.2</center>
</body>
</html>
[clincha@bri-runner-01 ~]$
clincha commented 1 year ago

This was recommended but did nothing

systemctl stop kubelet
systemctl stop cri-o
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start cri-o
clincha commented 1 year ago

Had to pin it to a node. Not happy about that...

clincha commented 1 year ago

Urg this is happening to all services not just NGINX

clincha commented 1 year ago

https://github.com/kubernetes/kubernetes/issues/100434

This suggests that moving to Flannel instead of Calico should work. Giving that a go now

clincha commented 1 year ago

Remove Calico

kubectl delete -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl delete -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Install Flannel

kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
clincha commented 1 year ago

vi /etc/kubernetes/manifests/kube-controller-manager.yaml

https://gist.github.com/rkaramandi/44c7cea91501e735ea99e356e9ae7883

https://github.com/flannel-io/flannel/issues/728

ip link set cni0 down && ip link set flannel.1 down 
ip link delete cni0 && ip link delete flannel.1
systemctl restart cri-o && systemctl restart kubelet
clincha commented 1 year ago

After switching over to Flannel the issue is still not resolved. Apparently Weave has sorted it out so I guess I'll try them next. Although setting sudo iptables -P FORWARD ACCEPT didn't seem to help which was supposed to if Weave was going to help

clincha commented 1 year ago

This worked https://stackoverflow.com/questions/62540512/k8s-1-18-1-api-not-reachable-since-update-to-1-18-1

I needed to set the NET_ADMIN to privileged in the flannel YAML

clincha commented 1 year ago

It's so good to see it working. I'll get everything written up properly later but the gist of it is this: