search

containernetworking / plugins

Some reference and example networking plugins, maintained by the CNI team.
Apache License 2.0
2.23k stars 788 forks source link

portmap nftables backend invalid rules (dnat from input hook + invalid ipv6 rules) #1115

Closed champtar closed 1 week ago

champtar commented 2 weeks ago

Trying the new nftables backend, it fails on EL 9.4

# nft --version
nftables v1.0.9 (Old Doc Yak #3)
# uname -r
5.14.0-427.40.1.el9_4.x86_64

conf

{
  "type": "portmap",
  "capabilities": {"portMappings": true},
  "backend": "nftables",
  "conditionsV4": ["ip", "daddr", "!=", "{ 127.0.0.0/8, 198.19.254.254 }"]
},

Error

Nov 05 17:15:34 atsc2 kubelet[4258]: E1105 17:15:34.352838    4258 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err=<
Nov 05 17:15:34 atsc2 kubelet[4258]:         rpc error: code = Unknown desc = failed to setup network for sandbox "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621": plugin type="multus-cni" name="multus-cni-network" failed (add): [traefik/appliance-ingress-traefik-698cd97568-5xg88/3908060c-0419-481c-865d-3230b95c3a84:mgmt]: error adding container to network "mgmt": plugin type="portmap" failed (add): unable to set up nftables rules for port mappings: /dev/stdin:13:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.1.194 . 10081 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:14:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.1.194 . 18444 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:15:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.1.194 . 10082 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:16:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.1.194 . 18445 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:17:1-182: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.1.194 . 10083 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:18:1-184: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.1.194 . 18446 comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:19:1-168: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 198.18.1.194 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:         /dev/stdin:20:1-165: Error: Could not process rule: Operation not supported
Nov 05 17:15:34 atsc2 kubelet[4258]:         add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.1.194 masquerade comment "9fa1b2975bf1dc51c273d7b5e53e7f4500faa1f931d8552e2761fd534b37d621"
Nov 05 17:15:34 atsc2 kubelet[4258]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 05 17:15:34 atsc2 kubelet[4258]:  >

Using strace, the rules are:

add table ip cni_hostport { comment "CNI portmap plugin" ; }
add chain ip cni_hostport hostports
add chain ip cni_hostport hostip_hostports
add chain ip cni_hostport input { type nat hook input priority -100 ; }
flush chain ip cni_hostport input
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport input ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostports
add chain ip cni_hostport output { type nat hook output priority -100 ; }
flush chain ip cni_hostport output
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } jump hostip_hostports
add rule ip cni_hostport output ip daddr != { 127.0.0.0/8, 198.19.254.254 } fib daddr type local jump hostports
add chain ip cni_hostport masquerading { type nat hook postrouting priority 100 ; }
add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.0.56 . 10081 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8444 dnat ip addr . port to 198.18.0.56 . 18444 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 82 dnat ip addr . port to 198.18.0.56 . 10082 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8445 dnat ip addr . port to 198.18.0.56 . 18445 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 83 dnat ip addr . port to 198.18.0.56 . 10083 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport hostports ip protocol tcp th dport 8446 dnat ip addr . port to 198.18.0.56 . 18446 comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 198.18.0.56 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"
add rule ip cni_hostport masquerading ip saddr 127.0.0.1 ip daddr 198.18.0.56 masquerade comment "4b624583ded419060960435aec2264eda03b1b0ac54c009a2ac98dc8492bf729"

I think instead of

add chain ip cni_hostport input { type nat hook input priority -100 ; }

we should use

add chain ip cni_hostport input { type nat hook prerouting priority -100 ; }

(and rename input to prerouting)

champtar commented 2 weeks ago

The ip6 rules are also invalid,

kubelet[4258]:         /dev/stdin:18:41-48: Error: syntax error, unexpected string
kubelet[4258]:         add rule ip6 cni_hostport hostports ip6 protocol tcp th dport 8446 dnat ip6 addr . port to fd61:7465:6d65:1000::3cd . 18446 comment "30ba0c9d53d293e8ed7b7dcdc37cd0cb8380f78cbd2b3977bc57f8e9931e04e7"
kubelet[4258]:                                                 ^^^^^^^^

ip6 protocol tcp gives a syntax error, we can use meta l4proto tcp th dport, but I'm curious why aren't we just using tcp dport ? @danwinship ?

champtar commented 2 weeks ago 
nft 'add rule ip cni_hostport hostports meta l4proto tcp th dport 81 dnat ip addr . port to 198.18.3.235 . 10081 comment "1"'
nft 'add rule ip cni_hostport hostports ip protocol tcp th dport 81 dnat ip addr . port to 198.18.3.235 . 10081 comment "2"'
nft 'add rule ip cni_hostport hostports tcp dport 81 dnat ip addr . port to 198.18.3.235 . 10081 comment "3"'
# nft list ruleset
        tcp dport 81 dnat to 198.18.3.235:10081 comment "1"
        tcp dport 81 dnat to 198.18.3.235:10081 comment "2"
        tcp dport 81 dnat to 198.18.3.235:10081 comment "3"
danwinship commented 2 weeks ago

hm... I had tested all of this at one point... I must have made a "minor fix" at some point and broken it

I think instead of

need to think about this and compare with iptables portmap and nftables kube-proxy rules...

I'm curious why aren't we just using tcp dport

probably I copied from somewhere else and simplified/rewrote and didn't notice it ended up having unnecessary clauses in the end.