What wrong with snat (nftables)?

MrSorcus commented 7 years ago

Issue description

Snat doesn't work with latest NixOS master branch. But works in Archlinux. Archlinux has similar configuration.

Steps to reproduce

Build server & client ISO image: nix-build -A config.system.build.isoImage -I nixos-config=./server.nix ./nixpkgs/nixos/default.nix nix-build -A config.system.build.isoImage -I nixos-config=./client.nix ./nixpkgs/nixos/default.nix https://gist.github.com/MrSorcus/472f5ddaad28d01a92811c49d2f86398 (nix files)
Create virtual machines:

Server:

virt-install \          
--name NixOSVS05 \
--ram 1024 \
--vcpus 1 \
--cdrom /tmp/nixos_05.iso \
--os-type linux \
--nodisks \
--network bridge=br0 \
--graphics vnc,password='ABCDE',port=5905,listen=2a01:4f8:xx:xx::36 \
--autostart \
--noautoconsole

Client:

virt-install \          
--name NixOSVS10 \
--ram 1024 \
--vcpus 1 \
--cdrom /tmp/nixos_10.iso \
--os-type linux \
--nodisks \
--network bridge=br0 \
--graphics vnc,password='ABCDE',port=5910,listen=2a01:4f8:xx:xx::36 \
--autostart \
--noautoconsole

Ping 2001:19f0:7400:87a2::64 (https://ipv6.net/ for example)
SNAT doesn't work correctly. See logs: https://gist.github.com/MrSorcus/d45df56e62741719fbbcd2919505abc9 - logs (nft monitor trace, ip a, ip -6 route, route -6, tshark) https://gist.github.com/MrSorcus/1e912bfadc0662f40852d9ea0f3ebc83 - compare /proc/config.gz from NixOS and Archlinux.

Technical details

System: 17.09.git.d4fbb47 (Hummingbird)
Nix version: nix-env (Nix) 1.11.8
Nixpkgs version: "17.09.git.d4fbb47"
Sandboxing enabled: build-use-sandbox = false

MrSorcus commented 7 years ago

Work with iptables ip6tables -t nat -A POSTROUTING -o ens3 -j SNAT --to-source 2a01:4f8:xx:xx::5, not with nftables...

MrSorcus commented 7 years ago

Tcpdump logs. https://gist.github.com/MrSorcus/3dca950513c454c62ff66b6965c29fc3 (with nftables doesn't work) https://gist.github.com/MrSorcus/3598e588bb02d06d21e4e66735362093 (working with iptables)

Mic92 commented 7 years ago

I have not yet tested yet tested your example, but I use source nat for ipv6 with nftables on nixos:

table ip6 nat {
        chain postrouting {
                type nat hook postrouting priority 0; policy accept;
                oifname "vpn0" snat to fdda:d0d0:cafe:1300::1007
        }
}

My kernel version is 4.9.27 at the moment on this machine.

Mic92 commented 7 years ago

I would say the snat already worked in your case. On your servers eth0 are the following packets:

26 3.499453880 2a01:4f8:xx:xx::5 → 2001:19f0:xx:xx::64 ICMPv6 118 Echo (ping) request id=0x03e6, seq=1, hop limit=63
27 3.515236598 2001:19f0:xx:xx::64 → 2a01:4f8:xx:xx::5 ICMPv6 118 Echo (ping) reply id=0x03e6, seq=1, hop limit=55 (request in 26)

They could only come from the ping of your client.

MrSorcus commented 7 years ago

@Mic92 , doesn't work with any kernel version... Packets must leave from enp0s3 to wg0, but it doesn't.....

Mic92 commented 7 years ago

I don't know why this happens. I only know that it requires conntrack to reverse SNAT:

$ lsmod | grep conntrack                                                                                                                                             
nf_conntrack_netlink    32768  0                                                                                                                                                              
nf_conntrack_ipv4      16384  3                                                                                                                                                               
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4                                                                                                                                             
nf_conntrack_ipv6      20480  3                                                                                                                                                               
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6                                                                                                                                             
nf_conntrack           86016  8 nft_ct,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_conntrack_netlink,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4,nf_nat                                         
nfnetlink              16384  3 nfnetlink_log,nf_conntrack_netlink,nf_tables                                                                                                                  
ipv6                  389120  1475 esp6,nf_conntrack_ipv6,nf_reject_ipv6,ip6_gre,nf_defrag_ipv6,wireguard,nf_nat_ipv6,ip6_tunnel,bridge

tracked connection can be also listed with conntrack -L from conntrack_tools. Apart from that the only two workarounds comes to my mind are:

Use public ipv6 addresses and do routing
Try to dnat the reverse path (and make it stateless this way).

MrSorcus commented 7 years ago

@Mic92 i have public ipv6 address. But why with iptables snat all works and doesn't with nftables? What wrong with nftables?

MrSorcus commented 7 years ago

dmesg - https://gist.github.com/MrSorcus/f364ed907c218adcf3f3b16b202d99cc lsmod - https://gist.github.com/MrSorcus/0da1b249bcc3bc41e357826165c13f11 And tcpdump logs before and after command ip6tables -t nat -A POSTROUTING -o enp0s3 -j SNAT --to-source 2a01:4f8:xx:xx::5 Before (nftables used only) - https://gist.github.com/MrSorcus/91569ffd39f5d5b1c74056583fce1bba After (ip6tables used) - https://gist.github.com/MrSorcus/97d49e7a3e14f7008d3dafd71244ffcc

MrSorcus commented 7 years ago

tracked connection can be also listed with conntrack -L from conntrack_tools.
[root@nixos:~]# conntrack -L
conntrack v1.4.4 (conntrack-tools): 0 flow entries have been shown.

[root@nixos:~]# conntrack -S cpu=0 found=0 invalid=22 ignore=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0

Mic92 commented 7 years ago

I meant, you could use public ipv6 subnets inside your vpn instead of snat. Actually conntrack should have flows or something went wrong. It fails to track the connection that's why it does not translate back. I cannot tell why as it is working for me with nftables and the rules seems correct to me. You could also try netmap, which is some kind of symmetrical NAT and does not require state.

MrSorcus commented 7 years ago

Can you share your ip -6 route output from vpn server, please?

Mic92 commented 7 years ago

It would not help you since it contains 600 unrelated routes I import from BGP routing protocol, while the actual interesting routes are in a different routing table and get accessed via source routing rules. But believe me, it is not your routing table. Your one is pretty standard.

Mic92 commented 7 years ago

But if you want to help from network people, who also play with VPNs a lot, you can ask in #dn42 on freenode.

MrSorcus commented 7 years ago

@Mic92 no answer in #dn42... And nftables doesn't work correctly in archlinux too... Only after next steps:

nftables enabled, ping any ipv6 resource (ipv6 from google, for example) doesn't work.
Disable nftables and enable ip6tables. Now you can ping any resource.
Disable ip6tables and enable nftables. It's work again... I don't know why.

Maybe nftables can't create some entries, but can read it after ip6tables?

Mic92 commented 7 years ago

Usually loading iptables kernel module will mess up nftables. I have blacklisted iptables on my system for that reason. I am not sure about other modules on which both firewall subsystems might depend. You should check the difference of loaded kernel modules in this case.

MrSorcus commented 7 years ago

How can i do it? (disable iptables) networking.firewall.enable = false; - i have this now.

Mic92 commented 7 years ago

This usually enough, however some tools might iptables itself (like docker). For that reason, I use:

boot.blacklistedKernelModules = [ "iptables_nat" "ip_tables" ];

Mic92 commented 7 years ago

These are the netfilter related modules on my system:

$ lsmod | grep nf_
nf_conntrack_netlink    32768  0                                                                                                                                            
nf_reject_ipv4         16384  1 nft_reject_inet                                                                                                                             
nf_reject_ipv6         16384  1 nft_reject_inet                                                                                                                             
nf_nat_masquerade_ipv4    16384  1 nft_masq_ipv4                                                                                                                            
nf_conntrack_ipv4      16384  3                                                                                                                                             
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4                                                                                                                           
nf_nat_ipv4            16384  1 nft_chain_nat_ipv4                                                                                                                          
nf_conntrack_ipv6      20480  3                                                                                                                                             
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6                                                                                                                           
nf_nat_ipv6            16384  1 nft_chain_nat_ipv6                                                                                                                          
nf_nat                 24576  4 nft_nat,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4                                                                                      
nf_conntrack           86016  8 nft_ct,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_conntrack_netlink,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4,nf_nat                       
nf_tables_inet         16384  4                                                                                                                                             
nf_tables_ipv6         16384  3 nf_tables_inet                                                                                                                              
nf_tables_ipv4         16384  3 nf_tables_inet                                                                                                                              
nf_tables              69632  2707 nft_ct,nft_nat,nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_reject,nft_set_hash,nf_tables_ipv6,nf_tables_ipv4,nft_masq,nft_reject_inet,nft_meta,nft_set_rbtree,nft_masq_ipv4,nft_counter,nft_log,nf_tables_inet                                                                                                          
nfnetlink              16384  3 nfnetlink_log,nf_conntrack_netlink,nf_tables                                                                                                
ipv6                  389120  1497 esp6,nf_conntrack_ipv6,nf_reject_ipv6,ip6_gre,nf_defrag_ipv6,wireguard,nf_nat_ipv6,ip6_tunnel,bridge

nf_conntrack might be important for your use case.

MrSorcus commented 7 years ago

@Mic92 nothing changed... I don't know how to fix it. Can you show me your nix files for installation? I think that i'm doing something wrong... But what?

MrSorcus commented 7 years ago

With git-version of nftables, libmnl & libnftnl doesn't work too...

MrSorcus commented 7 years ago

Solved. http://marc.info/?l=netfilter-devel&m=150027256708621&w=2

You are probably lacking the reply NAT chain, which needs to be registered. https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT) I'm updating right now the wiki to put this in bold.

NixOS / nixpkgs