Failed to execute iptables-restore: exit status 1 in kube-proxy

buptliuwei commented 6 years ago

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST /kind bug

Versions

kubeadm version (use kubeadm version):1.10.0

Environment:

Kubernetes version (use kubectl version):1.10.0
Cloud provider or hardware configuration:an amd64 master and 1 arm nodes (nvidia tagra TX2)
OS (e.g. from /etc/os-release): ubuntu 16.04
Kernel (e.g. uname -a): Linux ubuntu 4.13.0-39-generic # 44~16.04.1-Ubuntu SMP Thu Apr 5 16:43:10 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux for amd64 && Linux tegra-ubuntu 4.4.38 # 1 SMP PREEMPT Sun Apr 22 02:51:59 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux for arm
Others: iptables version: 1.6.0

What happened?

my iptables-save shows that kube-proxy did not set up a rule for 10.96.0.1 in arm node. But the kube-proxy pod in arm node is running.

 kubectl get pod --all-namespaces
NAMESPACE     NAME                             READY     STATUS    RESTARTS   AGE
kube-system   etcd-ubuntu                      1/1       Running   0          6d
kube-system   kube-apiserver-ubuntu            1/1       Running   0          6d
kube-system   kube-controller-manager-ubuntu   1/1       Running   0          6d
kube-system   kube-dns-86f4d74b45-pgwb8        0/3       Pending   0          6d
kube-system   kube-proxy-arm-bzrvg             1/1       Running   0          5s
kube-system   kube-proxy-pjnwn                 1/1       Running   0          6d
kube-system   kube-scheduler-ubuntu            1/1       Running   0          6d

iptables-save
# Generated by iptables-save v1.6.0 on Wed May  2 21:19:13 2018
*nat
:PREROUTING ACCEPT [4:905]
:INPUT ACCEPT [4:905]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
COMMIT
# Completed on Wed May  2 21:19:13 2018
# Generated by iptables-save v1.6.0 on Wed May  2 21:19:13 2018
*filter
:INPUT ACCEPT [85:25923]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [23:2048]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.32.0.0/12 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.32.0.0/12 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Wed May  2 21:19:13 2018

and i checked the logs for kube-proxy . The output is:

kubectl logs kube-proxy-arm-bzrvg -n=kube-system
I0502 13:19:00.679884       1 feature_gate.go:226] feature gates: &{{} map[]}
W0502 13:19:00.699625       1 server_others.go:290] Can't use ipvs proxier, trying iptables proxier
I0502 13:19:00.702634       1 server_others.go:140] Using iptables Proxier.
I0502 13:19:00.740414       1 server_others.go:174] Tearing down inactive rules.
I0502 13:19:01.034561       1 server.go:444] Version: v1.10.1
I0502 13:19:01.099829       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0502 13:19:01.100324       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0502 13:19:01.101273       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0502 13:19:01.101430       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0502 13:19:01.102335       1 config.go:202] Starting service config controller
I0502 13:19:01.102946       1 controller_utils.go:1019] Waiting for caches to sync for service config controller
I0502 13:19:01.103395       1 config.go:102] Starting endpoints config controller
I0502 13:19:01.104678       1 controller_utils.go:1019] Waiting for caches to sync for endpoints config controller
I0502 13:19:01.206422       1 controller_utils.go:1026] Caches are synced for service config controller
I0502 13:19:01.206422       1 controller_utils.go:1026] Caches are synced for endpoints config controller
E0502 13:19:01.415172       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed
)
E0502 13:19:31.288315       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed
)
E0502 13:20:01.540382       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed
)
E0502 13:20:31.758797       1 proxier.go:1285] Failed to execute iptables-restore: exit status 1 (iptables-restore: line 27 failed

What you expected to happen?

the iptables output should be:

# Generated by iptables-save v1.6.0 on Wed May  2 14:38:09 2018
*nat
:PREROUTING ACCEPT [1:68]
:INPUT ACCEPT [1:68]
:OUTPUT ACCEPT [2:120]
:POSTROUTING ACCEPT [2:120]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-FSVANR5HWKZAEIMM - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-FSVANR5HWKZAEIMM -s 10.108.48.92/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-FSVANR5HWKZAEIMM -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-FSVANR5HWKZAEIMM --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.108.48.92:6443
-A KUBE-SERVICES ! -s 10.32.0.0/12 -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-FSVANR5HWKZAEIMM --mask 255.255.255.255 --rsource -j KUBE-SEP-FSVANR5HWKZAEIMM
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-FSVANR5HWKZAEIMM
COMMIT
# Completed on Wed May  2 14:38:09 2018
# Generated by iptables-save v1.6.0 on Wed May  2 14:38:09 2018
*filter
:INPUT ACCEPT [630:176580]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [628:184037]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.32.0.0/12 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.32.0.0/12 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Wed May  2 14:38:09 2018

How to reproduce it (as minimally and precisely as possible)?

I create a kube-proxy for arm nodes using "node selector". such as Multiplatform (amd64 and arm) Kubernetes cluster setup

Anything else we need to know?

Because of this, i could not deploy weave.

buptliuwei commented 6 years ago

I re-compiled the TX2 kernel and loaded some kernel modules needed for netfliter and weave. The problem is solved.

martwetzels commented 6 years ago

@buptliuwei Which kernel modules did you add? I have the same issue with the TX2 but cannot pinpoint the kernel modules for weave.

buptliuwei commented 6 years ago

@martwetzels Hi, this is my modules after re-complied

nf_conntrack_netlink    24755  0
xt_nat                  2320  5
xt_recent              10058  2
ipt_REJECT              1951  1
nf_reject_ipv4          3438  1 ipt_REJECT
ip_set                 33915  0
nfnetlink               7318  2 ip_set,nf_conntrack_netlink
xt_comment              1348  32
xt_mark                 1663  5
fuse                   83099  2
ipt_MASQUERADE          2115  2
nf_nat_masquerade_ipv4     2931  1 ipt_MASQUERADE
iptable_nat             2285  1
nf_nat_ipv4             6554  1 iptable_nat
xt_addrtype             3298  3
iptable_filter          2119  1
ip_tables              18322  2 iptable_filter,iptable_nat
xt_conntrack            3551  3
nf_nat                 16285  3 nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4
br_netfilter           13923  0
overlay                33899  2
openvswitch            85585  2
bcmdhd               7447670  0
pci_tegra              60337  0
bluedroid_pm           11195  0

and you can read my gist . Hope it works.

martwetzels commented 6 years ago

Thanks! I just found the requirement for openvswitch and vxlan on a weave documentation page. Your gist is very useful, too bad it already took me a few hours to reinvent the wheel this week. Do you mind if I do a detailed write-up on Medium?

Btw, did you also manage to get the GPU capabilities visible on the node within the cluster from the TX2?

buptliuwei commented 6 years ago

@martwetzels Never mind. We are working hard to get the GPU capabilities visible.

StupidYe commented 6 years ago

@buptliuwei hi，brother，where is your gist，I can not find it,can you give me the url?

buptliuwei commented 6 years ago

@StupidYe hi this is my gist: https://gist.github.com/buptliuwei/8a340cc151507cb48a071cda04e1f882

buptliuwei commented 6 years ago

@StupidYe 你好，gist没有邮件提醒，现在才看到，不好意思啊。首先proxy的manifest,我是这样做的，通过编辑器打开原来的kube-proxy文件，然后复制到自己新建的文本中，改成kube-proxy-arm，然后做一些修改，主要是nodeselector。关于flannel部署失败的问题，从报的error上看也是不能路由到service ip。很大关系和kube-proxy有关，我不知道你arm上装的系统是什么，ubuntu的话，可能是有些内核模块没打开，flannel也是overlay的解决方案，需要openvswitch等内核模块的支持，你可以lsmod下，看看模块是否都加载了。最后，kubeadm的文档里面建议在arm上部署网络插件的话，weave的兼容性最好。所以你也可以考虑下weave.

StupidYe commented 6 years ago

@buptliuwei 非常谢谢你的回复，我会尝试使用weave。Thanks

yeliuang commented 6 years ago

Thanks! I just found the requirement for openvswitch and vxlan on a weave documentation page. Your gist is very useful, too bad it already took me a few hours to reinvent the wheel this week. Do you mind if I do a detailed write-up on Medium?

Btw, did you also manage to get the GPU capabilities visible on the node within the cluster from the TX2?

hi buddy! Have you made any progress on getting the GPU capabilities visible on the node within the cluster from the TX2? Beacause of the lack of official support ,I think it is a hard work to manage the GPU on TX2 nodes by k8s master. I am getting stuck in , do you mind giving me some instructions ?

martwetzels commented 6 years ago

@yeliuang I did not proceed with getting the GPUs visible in K8s because @buptliuwei said he was working on it; it already cost me quite some time. To finish up the project we used a different approach, but I am still interested in getting this to work.

alexrashed commented 5 years ago

Just for the record (as I've been struggling quite some time to get this running) here's my working config:

Jetson TX2
Jetpack 4.2 / L4T 32.1.0 / Kernel 4.9.140 (has just recently been released)
Kube-Proxy in iptables mode (with flannel as overlay network)

CONFIG_NETFILTER_XT_SET=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_HASH_IP=m
CONFIG_IP_SET_HASH_NET=m
CONFIG_NF_NAT_REDIRECT=m
CONFIG_IP_NF_TARGET_REDIRECT=m

I cannot say for sure if all of those modules are necessary, my issue (non accessible services running on the same minion) has been resolved with adding xt_physdev (after some tedious iptables debugging).

kubernetes / kubeadm