search

justmeandopensource / kubernetes

Kubernetes playground
1.66k stars 2.95k forks source link

LXD provisioing - Error running ProxyServer" err="write /sys/module/nf_conntrack/parameters/hashsize: operation not supported #144

Open Cloud-Mak opened 3 weeks ago

Cloud-Mak commented 3 weeks ago

Hi Venkat @justmeandopensource

https://github.com/justmeandopensource/kubernetes/tree/master/lxd-provisioning

Looks like issue with this method now. the kube-proxy remains into crashloop and eventually error 

root@master:/etc/kubernetes/manifests# kubectl -n kube-system logs kube-proxy-m6xgl
I0823 17:40:52.912090       1 server_linux.go:69] "Using iptables proxy"
I0823 17:40:52.924381       1 server.go:1062] "Successfully retrieved node IP(s)" IPs=["10.2.7.122"]
I0823 17:40:52.932417       1 conntrack.go:59] "Setting nf_conntrack_max" nfConntrackMax=786432
I0823 17:40:52.933129       1 conntrack.go:90] "Setting conntrack hashsize" conntrackHashsize=196608
E0823 17:40:52.933169       1 server.go:558] "Error running ProxyServer" err="write /sys/module/nf_conntrack/parameters/hashsize: operation not supported"
E0823 17:40:52.933190       1 run.go:74] "command failed" err="write /sys/module/nf_conntrack/parameters/hashsize: operation not supported"

I already set value - sysctl -w net.netfilter.nf_conntrack_max=786432. Its not helping at all.

root@master:/etc/kubernetes/manifests# kubectl -n kube-system logs kube-proxy-m6xgl
I0823 17:41:22.913788       1 server_linux.go:69] "Using iptables proxy"
I0823 17:41:22.922042       1 server.go:1062] "Successfully retrieved node IP(s)" IPs=["10.2.7.122"]
I0823 17:41:22.930034       1 conntrack.go:59] "Setting nf_conntrack_max" nfConntrackMax=786432
I0823 17:41:22.931151       1 conntrack.go:90] "Setting conntrack hashsize" conntrackHashsize=196608
E0823 17:41:22.931302       1 server.go:558] "Error running ProxyServer" err="write /sys/module/nf_conntrack/parameters/hashsize: operation not supported"
E0823 17:41:22.931432       1 run.go:74] "command failed" err="write /sys/module/nf_conntrack/parameters/hashsize: operation not supported"

More info - Host ubu 20.04.6 LTS LXD - stable 5.21 I am setting up 1st master node which is running inside ubuntu 22 lxd container.

I am getting node in ready status (I have already deployed flannel network)

Cloud-Mak commented 3 weeks ago

Update - there is already PR merged for ignoring this /sys/module/nf_conntrack/parameters/hashsize. So not sure how this is till an issue

One more PR - https://github.com/kubernetes/kubernetes/pull/19303

I tried patching kube-proxy ds in kube-system. Just added --conntrack-max-per-core=0 to container of kube-proxy ds. After this the k-proxy pod went int error state. can't see any logs from same pod now. strange

Addtional info - Linux does not support writing to /sys/module/nf_conntrack/parameters/hashsize when the writer process is not in the initial network namespace (https://github.com/torvalds/linux/blob/v4.10/net/netfilter/nf_conntrack_core.c#L1795-L1796).

Usually that's fine. But in some configurations such as (with https://github.com/kinvolk/kubeadm-nspawn) or in current LXD container kube-proxy is in another netns)

Therefore, check if writing in hashsize is necessary and skip the writing if not.

Referenes - https://github.com/kinvolk/kube-spawn/issues/14 https://github.com/kubernetes/kubernetes/issues/90083