search

ldx / python-iptables

Python bindings for iptables
731 stars 183 forks source link

nftables / CentOS8 support #306

Closed iptizer closed 4 years ago

iptizer commented 4 years ago

I am not 100% sure how things are related between iptables and nftables, so I though I'll highlight my problem here, maybe there is a solution.

Setup

I am using Kubernetes with Calico on CentOS8. CentOS8 uses nftables. Felix (the Calico component for setting iptables on a host) is setup with backend=NFT. This works, but some things are not working 100% so I thought let's fix this using python.

My iptables command is linked to xtables:

[root@myserver myuser]# ll /usr/sbin/iptables
lrwxrwxrwx. 1 root root 17 Apr 24 13:51 /usr/sbin/iptables -> xtables-nft-multi

Problem

Listing chains with iptables -L -t nat and iptc.easy.dump_table('nat', ipv6=False) show different results:

>>> iptc.easy.dump_table('nat', ipv6=False)
{'PREROUTING': [{'comment': {'comment': 'kubernetes service portals'}, 'target': 'KUBE-SERVICES', 'counters': (810333, 46201389)}], 'INPUT': [], 'OUTPUT': [{'comment': {'comment': 'kubernetes service portals'}, 'target': 'KUBE-SERVICES', 'counters': (4088850, 246510358)}], 'POSTROUTING': [{'comment': {'comment': 'kubernetes postrouting rules'}, 'target': 'KUBE-POSTROUTING', 'counters': (4542742, 273781783)}], 'KUBE-FIREWALL': [{'target': 'KUBE-MARK-DROP', 'counters': (0, 0)}], 'KUBE-LOAD-BALANCER': [{'target': 'KUBE-MARK-MASQ', 'counters': (0, 0)}], 'KUBE-MARK-DROP': [], 'KUBE-MARK-MASQ': [{'target': {'MARK': {'set-xmark': '0x4000/0x4000'}}, 'counters': (0, 0)}], 'KUBE-NODE-PORT': [{'protocol': 'tcp', 'comment': {'comment': 'Kubernetes nodeport TCP port for masquerade purpose'}, 'set': {'match-set': ['KUBE-NODE-PORT-TCP', 'dst']}, 'target': 'KUBE-MARK-MASQ', 'counters': (0, 0)}], 'KUBE-POSTROUTING': [{'comment': {'comment': 'kubernetes service traffic requiring SNAT'}, 'mark': {'mark': '0x4000/0x4000'}, 'target': {'MASQUERADE': {'random-fully': ''}}, 'counters': (0, 0)}, {'comment': {'comment': 'Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose'}, 'set': {'match-set': ['KUBE-LOOP-BACK', 'dst,dst,src']}, 'target': 'MASQUERADE', 'counters': (2, 120)}], 'KUBE-SERVICES': [{'src': '!10.233.64.0/18', 'comment': {'comment': 'Kubernetes service cluster ip + port for masquerade purpose'}, 'set': {'match-set': ['KUBE-CLUSTER-IP', 'dst,dst']}, 'target': 'KUBE-MARK-MASQ', 'counters': (0, 0)}, {'addrtype': {'dst-type': 'LOCAL'}, 'target': 'KUBE-NODE-PORT', 'counters': (53, 2948)}, {'set': {'match-set': ['KUBE-CLUSTER-IP', 'dst,dst']}, 'target': 'ACCEPT', 'counters': (11, 660)}]}
[root@ns3032787 moritz]# iptables -L -t nat | grep -i chain
Chain PREROUTING (policy ACCEPT)
Chain INPUT (policy ACCEPT)
Chain POSTROUTING (policy ACCEPT)
Chain OUTPUT (policy ACCEPT)
Chain KUBE-MARK-DROP (0 references)
Chain KUBE-MARK-MASQ (0 references)
Chain KUBE-POSTROUTING (1 references)
Chain KUBE-KUBELET-CANARY (0 references)
Chain cali-PREROUTING (1 references)
Chain cali-POSTROUTING (1 references)
Chain cali-OUTPUT (1 references)
Chain cali-fip-dnat (2 references)
Chain cali-fip-snat (1 references)
Chain cali-nat-outgoing (1 references)
Chain CNI-HOSTPORT-SETMARK (19 references)
Chain CNI-HOSTPORT-MASQ (1 references)
Chain CNI-HOSTPORT-DNAT (2 references)
Chain CNI-DN-ff88843368fae4d5cdfae (0 references)
Chain CNI-DN-f0f13904d22032106b1e7 (0 references)
Chain CNI-DN-442e7af5e08e6bf46621a (0 references)
Chain CNI-DN-e4582bd2b3ff7ed196a28 (0 references)
Chain CNI-DN-365d9a185cff3e5ea9379 (0 references)
Chain CNI-DN-90b43263af9d462e04f9b (0 references)
Chain CNI-DN-f54bc33afc414d6e64133 (1 references)
Chain CNI-DN-fc95996fce81df369bc47 (1 references)
Chain CNI-DN-ea59c359e9d718f7116dc (0 references)
Chain CNI-DN-9a7663babfc285ea01c15 (1 references)
Chain CNI-DN-fc387103aa44e3b189d09 (1 references)

So, some chains are showing up, some not.

Question: Is this even supposed to work? Or are the differences between iptables and nftables so large, that using a different python library is recommended?

jllorente commented 4 years ago

Hi @iptizer , can you please provide the output of # iptables -t nat -S ?

iptizer commented 4 years ago

Hi @iptizer , can you please provide the output of # iptables -t nat -S ?

sure thing!

[root@myserver ~]# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P POSTROUTING ACCEPT
-P OUTPUT ACCEPT
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-POSTROUTING
-N KUBE-KUBELET-CANARY
-N cali-PREROUTING
-N cali-POSTROUTING
-N cali-OUTPUT
-N cali-fip-dnat
-N cali-fip-snat
-N cali-nat-outgoing
-N CNI-HOSTPORT-SETMARK
-N CNI-HOSTPORT-MASQ
-N CNI-HOSTPORT-DNAT
-N CNI-DN-ff88843368fae4d5cdfae
-N CNI-DN-f0f13904d22032106b1e7
-N CNI-DN-442e7af5e08e6bf46621a
-N CNI-DN-e4582bd2b3ff7ed196a28
-N CNI-DN-365d9a185cff3e5ea9379
-N CNI-DN-90b43263af9d462e04f9b
-N CNI-DN-f54bc33afc414d6e64133
-N CNI-DN-fc95996fce81df369bc47
-N CNI-DN-ea59c359e9d718f7116dc
-N CNI-DN-9a7663babfc285ea01c15
-N CNI-DN-fc387103aa44e3b189d09
-N CNI-DN-bcfdb00a9541b4df781a0
-N CNI-DN-2a1d84389734e54983abb
-N CNI-DN-38f4d538cd34246b81019
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" -j cali-POSTROUTING
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE --random-fully
-A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" -j cali-fip-dnat
-A cali-POSTROUTING -m comment --comment "cali:Z-c7XtVd2Bq7s_hA" -j cali-fip-snat
-A cali-POSTROUTING -m comment --comment "cali:nYKhEzDlr11Jccal" -j cali-nat-outgoing
-A cali-POSTROUTING -o tunl0 -m comment --comment "cali:SXWvdsbh4Mw7wOln" -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL -j MASQUERADE --random-fully
-A cali-OUTPUT -m comment --comment "cali:GBTAv2p5CwevEyJm" -j cali-fip-dnat
-A cali-nat-outgoing -m comment --comment "cali:flqWnvo8yq4ULQLa" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE --random-fully
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cni0\" id: \"e05c0b7dc624c20da56c8db60492744381eb27b7449e28e87f8efa6da8210a9e\"" -m multiport --dports 10081,8443 -j CNI-DN-bcfdb00a9541b4df781a0
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cni0\" id: \"24b63cb1cbda96c6c89e5fb89700141460323010decfb98160b6434700e3a9a4\"" -m multiport --dports 10081,8443 -j CNI-DN-2a1d84389734e54983abb
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cni0\" id: \"c6578acfe6765587433b73388f55424e1f6e8996f3bb38c1e7ab573995f34245\"" -m multiport --dports 10081,8443 -j CNI-DN-38f4d538cd34246b81019
-A CNI-DN-f0f13904d22032106b1e7 -s 10.233.124.132/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f0f13904d22032106b1e7 -s 127.0.0.1/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK

-A CNI-DN-f0f13904d22032106b1e7 -p tcp -m tcp --dport 10080 -j DNAT --to-destination 10.233.124.132:80
-A CNI-DN-442e7af5e08e6bf46621a -s 10.233.124.122/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-442e7af5e08e6bf46621a -s 127.0.0.1/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-442e7af5e08e6bf46621a -p tcp -m tcp --dport 10080 -j DNAT --to-destination 10.233.124.122:80
-A CNI-DN-e4582bd2b3ff7ed196a28 -s 10.233.124.141/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-e4582bd2b3ff7ed196a28 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-365d9a185cff3e5ea9379 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-365d9a185cff3e5ea9379 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.245:80
-A CNI-DN-90b43263af9d462e04f9b -s 10.233.124.192/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.192:80
-A CNI-DN-90b43263af9d462e04f9b -s 10.233.124.192/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.192:443
-A CNI-DN-9a7663babfc285ea01c15 -s 10.233.124.151/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.151:80
-A CNI-DN-9a7663babfc285ea01c15 -s 10.233.124.151/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.151:443
-A CNI-DN-fc387103aa44e3b189d09 -s 10.233.124.36/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.36:80
-A CNI-DN-fc387103aa44e3b189d09 -s 10.233.124.36/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.36:443
-A CNI-DN-bcfdb00a9541b4df781a0 -s 10.233.124.50/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.50:80
-A CNI-DN-bcfdb00a9541b4df781a0 -s 10.233.124.50/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.50:443
-A CNI-DN-2a1d84389734e54983abb -s 10.233.124.73/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.73:80
-A CNI-DN-2a1d84389734e54983abb -s 10.233.124.73/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.73:443
-A CNI-DN-38f4d538cd34246b81019 -s 10.233.124.236/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.236:80
-A CNI-DN-38f4d538cd34246b81019 -s 10.233.124.236/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.236:443
# Warning: iptables-legacy tables present, use iptables-legacy to see them
ldx commented 4 years ago

My understanding is, even though nftables provides a compatibility layer and the CLI is similar, under the hood it's a different library. Python-iptables interacts with iptables via the C libraries, so it doesn't work with nftables.

Recently the official nftables project added python bindings it seems: https://git.netfilter.org/nftables/tree/py so you might want to give that a try. I haven't tested it personally though.