Could not find a global IPv6 address on enp2s0 from which to derive a CLAT IPv6 address

Lucaslah commented 1 year ago

Hi, I have been unable to get this work, when I start clatd I get this error:

Starting clatd v1.5 by Tore Anderson <tore@fud.no>
Performing DNS64-based PLAT prefix discovery (cf. RFC 7050)
Using PLAT (NAT64) prefix: 64:ff9b::/96
Device facing the PLAT: enp2s0
Attempting to derive a CLAT IPv6 address from an IPv6 address on 'enp2s0'
<error> Could not find a global IPv6 address on enp2s0 from which to derive a CLAT IPv6 address (try setting 'clat-v6-addr')

enp2s0 interface info

2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 74:86:e2:24:82:7a brd ff:ff:ff:ff:ff:ff
    inet6 [removed]::a5/128 scope global dynamic noprefixroute 
       valid_lft 5254sec preferred_lft 2554sec
    inet6 fe80::5a93:449d:1c08:bd8d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

How would I get this working? do I need to set clat-v6-addr to something? if so what?

timriker commented 1 year ago

Read the LIMITATIONS in the README. You need to be using slaac so that the clat app can generate it's own ip address. It looks like you are either usijng dhcpv6 or a single static address. This won't work from what I'm reading.

toreanderson commented 1 year ago

@timriker is correct. Actually, the LIMITATIONS section is a bit misleading and should be updated, as clatd will (since 81f2c61364143f898e53b69625e1a4cb9d5f5b71) try to auto-generate a clat-v6-addr for any prefixes /120 or larger - SLAAC or no SLAAC.

Anyway, @Lucaslah: you need to set clat-v6-addr to an IPv6 address that is routed to your host by the surrounding network. It might be that the link does have a prefix routed to it, and if so you might be able to set the address to some adjacent address to the one you already have, such as [removed]::a6. Try it and see?

Lucaslah commented 1 year ago

@timriker is correct. Actually, the LIMITATIONS section is a bit misleading and should be updated, as clatd will (since 81f2c61) try to auto-generate a clat-v6-addr for any prefixes /120 or larger - SLAAC or no SLAAC.

Anyway, @Lucaslah: you need to set clat-v6-addr to an IPv6 address that is routed to your host by the surrounding network. It might be that the link does have a prefix routed to it, and if so you might be able to set the address to some adjacent address to the one you already have, such as [removed]::a6. Try it and see?

Setting the clat-v6-addr allows clatd to start but still cannot access the IPv4 internet

Starting clatd v1.5 by Tore Anderson <tore@fud.no>
Performing DNS64-based PLAT prefix discovery (cf. RFC 7050)
Using PLAT (NAT64) prefix: 64:ff9b::/96
Device facing the PLAT: enp2s0
Using CLAT IPv4 address: 192.0.0.1
Using CLAT IPv6 address: [removed]::a6
Enabling Proxy-ND for [removed]::a6on enp2s0
Creating and configuring up CLAT device 'clat'
Created persistent tun device clat
Adding IPv4 default route via the CLAT
Starting up TAYGA, using config file '/tmp/wD0rfl6_YT'

~/Desktop > ping 1.1.1.1 -I clat
PING 1.1.1.1 (1.1.1.1) from 192.0.0.1 clat: 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
22 packets transmitted, 0 received, 100% packet loss, time 21486ms

The IP address is allocated via DHCPv6, I don't want to use SLAAC, is there any other way to make this work? (other than allocating a /120 or larger to every client, if that's the only way would prefix delegation work?)

timriker commented 1 year ago

Side note: Android does not support DHCPv6 specifically because it only gives one IP and the xlat code wants another IP. Android will only use SLAAC. https://issuetracker.google.com/issues/36949085

toreanderson commented 1 year ago

@Lucaslah I got an idea. Could you try the following?

nft "add table ip6 clatd"
nft "add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }"
nft "add rule ip6 clatd napt66 iifname clat masquerade"
clatd clat-v6-addr=fd00::c1a7 # or another random address, doesn't really matter

Lucaslah commented 1 year ago

Sure, Ill try it sometime today.

Lucaslah commented 1 year ago

Side note: Android does not support DHCPv6 specifically because it only gives one IP and the xlat code wants another IP. Android will only use SLAAC. https://issuetracker.google.com/issues/36949085

Oh I was unaware of that, just enabled SLAAC on vlans that have mobile devices on them, I just don't want SLAAC on some particular networks.

Lucaslah commented 1 year ago

@Lucaslah I got an idea. Could you try the following?

nft "add table ip6 clatd"
nft "add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }"
nft "add rule ip6 clatd napt66 iifname clat masquerade"
clatd clat-v6-addr=fd00::c1a7 # or another random address, doesn't really matter

This worked! thanks so much for your help.

Lucaslah commented 1 year ago

Only one issue, not sure if this is related to clatd or not but on ubuntu building from source this worked fine, however on fedora 36 installing with dnf install clatd clatd started but is still unable to reach the internet on the clat interface.

Edit: still getting the same issue on fedora when installing from source.

toreanderson commented 1 year ago

Great to hear that it worked! You can make this happen automatically by using script-up and script-down, for example by creating an /etc/clatd.conf file containing the following:

clat-v6-addr=fd00::c1a7
proxynd-enable=false
script-up=nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade'
script-down=nft 'delete table ip6 clatd'

I will consider implementing native support for this in clatd in the future.

Lucaslah commented 1 year ago

@toreanderson Alright thanks again, any ideas about it not working on fedora?

toreanderson commented 1 year ago

No. Please send a debug log (clatd -d -d [whatever other options you normally use]) from when reproducing the issue.

Lucaslah commented 1 year ago

eadconf('/etc/clatd.conf')
Configuration successfully read, dumping it:
  clat-dev=clat
  clat-v4-addr=192.0.0.1
  clat-v6-addr=fd00::c1a7
  cmd-ip=ip
  cmd-ip6tables=ip6tables
  cmd-tayga=tayga
  debug=2
  dns64-servers=[removed]
  forwarding-enable=1
  ip6tables-enable=<undefined>
  plat-dev=<undefined>
  plat-prefix=<undefined>
  proxynd-enable=false
  quiet=0
  script-down=nft 'delete table ip6 clatd'
  script-up=nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade'
  tayga-conffile=<undefined>
  tayga-v4-addr=192.0.0.2
  v4-conncheck-delay=10
  v4-conncheck-enable=false
  v4-defaultroute-advmss=0
  v4-defaultroute-enable=1
  v4-defaultroute-metric=2048
  v4-defaultroute-mtu=1260
  v4-defaultroute-replace=0
Starting clatd v1.6 by Tore Anderson <tore@fud.no>
Performing DNS64-based PLAT prefix discovery (cf. RFC 7050)
cfg(dns64-servers)
Looking up 'ipv4only.arpa' using DNS64 server 2001:4860:4860::6464
check_wka(): Testing to see if 64:ff9b:0:0:0:0:c000:aa was DNS64-synthesised
Looking for Well-Known Addresses at prefix length /96
Looking for WKA ::c000:aa
Found it!
Inferred PLAT prefix 64:ff9b::/96 from AAAA record 64:ff9b:0:0:0:0:c000:aa
check_wka(): Testing to see if 64:ff9b:0:0:0:0:c000:ab was DNS64-synthesised
Looking for Well-Known Addresses at prefix length /96
Looking for WKA ::c000:aa
Didn't find it
Looking for WKA ::c000:ab
Found it!
Inferred PLAT prefix 64:ff9b::/96 from AAAA record 64:ff9b:0:0:0:0:c000:ab
Using PLAT (NAT64) prefix: 64:ff9b::/96
get_plat_dev(): finding which network dev faces the PLAT
cfg(plat-prefix)
cfg(cmd-ip)
get_plat_dev(): Found PLAT-facing device: enp2s0
Device facing the PLAT: enp2s0
Using CLAT IPv4 address: 192.0.0.1
Using CLAT IPv6 address: fd00::c1a7
cfgint(v4-defaultroute-mtu)
cfgbool(v4-conncheck-enable)
Skipping IPv4 connectivity check at user request
cfg(tayga-conffile)
Using temporary conffile for TAYGA: /tmp/cnX1Xan0se
cfg(clat-dev)
cfg(plat-prefix)
cfg(tayga-v4-addr)
cfg(clat-v4-addr)
cfg(clat-v6-addr)
cfgbool(forwarding-enable)
Reading sysctl /proc/sys/net/ipv6/conf/all/forwarding
/proc/sys/net/ipv6/conf/all/forwarding is set to '1'
cfgbool(ip6tables-enable)
cfgbool(proxynd-enable)
cfg(clat-dev)
Creating and configuring up CLAT device 'clat'
cfg(cmd-tayga)
cfg(tayga-conffile)
cfgint(debug)
cmd(tayga --config /tmp/cnX1Xan0se --mktun -d)
Created persistent tun device clat
cfg(cmd-ip)
cfg(clat-dev)
cmd(ip link set up dev clat)
cfg(cmd-ip)
cfg(clat-v4-addr)
cfg(clat-dev)
cmd(ip -4 address add 192.0.0.1 dev clat)
cfg(cmd-ip)
cfg(clat-v6-addr)
cfg(clat-dev)
cmd(ip -6 route add fd00::c1a7 dev clat)
cfgbool(v4-defaultroute-replace)
cfgbool(v4-defaultroute-enable)
cfg(clat-dev)
cfgint(v4-defaultroute-metric)
cfgint(v4-defaultroute-metric)
cfgint(v4-defaultroute-mtu)
cfgint(v4-defaultroute-mtu)
cfgint(v4-defaultroute-advmss)
cfgint(v4-defaultroute-advmss)
Adding IPv4 default route via the CLAT
cfg(cmd-ip)
cmd(ip -4 route add default dev clat metric 2048 mtu 1260 advmss 1220)
Script env: clat-dev=clat
Script env: clat-v4-addr=192.0.0.1
Script env: clat-v6-addr=fd00::c1a7
Script env: cmd-ip=ip
Script env: cmd-ip6tables=ip6tables
Script env: cmd-tayga=tayga
Script env: debug=2
Script env: dns64-servers=[removed]
Script env: forwarding-enable=1
Script env: ip6tables-enable=
Script env: plat-dev=enp2s0
Script env: plat-prefix=64:ff9b::/96
Script env: proxynd-enable=false
Script env: quiet=
Script env: script-down=nft 'delete table ip6 clatd'
Script env: script-up=nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade'
Script env: tayga-conffile=/tmp/cnX1Xan0se
Script env: tayga-v4-addr=192.0.0.2
Script env: v4-conncheck-delay=10
Script env: v4-conncheck-enable=false
Script env: v4-defaultroute-advmss=1220
Script env: v4-defaultroute-enable=1
Script env: v4-defaultroute-metric=2048
Script env: v4-defaultroute-mtu=1260
Script env: v4-defaultroute-replace=
cfg(script-up)
cfg(script-up)
Running custom startup script: nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade'
cfg(script-up)
cmd(nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade')
Starting up TAYGA, using config file '/tmp/cnX1Xan0se'
cfg(cmd-tayga)
cfg(tayga-conffile)
cfgint(debug)
cmd(tayga --config /tmp/cnX1Xan0se --nodetach -d)
starting TAYGA 0.9.2
Using tun device clat with MTU 1500
TAYGA's IPv4 address: 192.0.0.2
TAYGA's IPv6 address: 64:ff9b::c000:2
NAT64 prefix: 64:ff9b::/96
Note: traffic between IPv6 hosts and private IPv4 addresses (i.e. to/from 64:ff9b::10.0.0.0/104, 64:ff9b::192.168.0.0/112, etc) will be dropped.  Use a translation prefix within your organization's IPv6 address space instead of 64:ff9b::/96 if you need your IPv6 hosts to communicate with private IPv4 addresses.

toreanderson commented 1 year ago

This looks pretty normal. The next steps are:

Verify that traffic is going into the clat device as IPv4, passing through the TAYGA translation engine, and coming back out as IPv6. This you can do by running tcpdump -ni clat port 80 in one shell while sending some IPv4 traffic from another (e.g. nc -zv4 github.com 80
Verify that the IPv6 traffic that came out of the clat interface is forwarded out the enp2s0 towards the NAT64 after undergoing NAPT66. Verify that by running another tcpdump -ni enp2s0 port 80 in a third shell while sending the IPv4 traffic.
Verify that the responses come back in the reverse direction. The output from the two tcpdump processes should tell me all of that, so just send me the output from all three commands during a single test.

Lucaslah commented 1 year ago

Here are the outputs, looks like the traffic is not forwarded to ens160 (interface is different because I'm testing in a VM now)

[user@fedora ~]$ sudo tcpdump -ni clat port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on clat, link-type RAW (Raw IP), snapshot length 262144 bytes
16:38:02.008982 IP 192.0.0.1.58356 > 140.82.113.3.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105218179 ecr 0,nop,wscale 7], length 0
16:38:02.009033 IP6 fd00::c1a7.58356 > 64:ff9b::8c52:7103.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105218179 ecr 0,nop,wscale 7], length 0
16:38:03.045348 IP 192.0.0.1.58356 > 140.82.113.3.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105219215 ecr 0,nop,wscale 7], length 0
16:38:03.045512 IP6 fd00::c1a7.58356 > 64:ff9b::8c52:7103.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105219215 ecr 0,nop,wscale 7], length 0
16:38:05.093428 IP 192.0.0.1.58356 > 140.82.113.3.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105221263 ecr 0,nop,wscale 7], length 0
16:38:05.093710 IP6 fd00::c1a7.58356 > 64:ff9b::8c52:7103.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105221263 ecr 0,nop,wscale 7], length 0
16:38:09.125317 IP 192.0.0.1.58356 > 140.82.113.3.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105225295 ecr 0,nop,wscale 7], length 0
16:38:09.125589 IP6 fd00::c1a7.58356 > 64:ff9b::8c52:7103.http: Flags [S], seq 569728093, win 64660, options [mss 1220,sackOK,TS val 105225295 ecr 0,nop,wscale 7], length 0

[user@fedora ~]$ sudo tcpdump -ni ens160 port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes

[user@fedora ~]$ nc -zv4 github.com 80
Ncat: Version 7.93 ( https://nmap.org/ncat )
Ncat: TIMEOUT.

toreanderson commented 1 year ago

Interesting. Can you (while clatd is running), verify the state of the IPv6 forwarding sysctls with grep . /proc/sys/net/ipv6/conf/*/forwarding? If enabled (they should be), do you have some kind of firewall running that would block this traffic? Check nft list ruleset and/or ip6tables-save.

Lucaslah commented 1 year ago

Output of grep . /proc/sys/net/ipv6/conf/*/forwarding

/proc/sys/net/ipv6/conf/all/forwarding:1
/proc/sys/net/ipv6/conf/clat/forwarding:1
/proc/sys/net/ipv6/conf/default/forwarding:1
/proc/sys/net/ipv6/conf/ens160/forwarding:1
/proc/sys/net/ipv6/conf/lo/forwarding:1

Output of nft list ruleset

[user@fedora ~]$ sudo nft list ruleset
[sudo] password for user: 
table inet firewalld {
    ct helper helper-netbios-ns-udp {
        type "netbios-ns" protocol udp
        l3proto ip
    }

    chain mangle_PREROUTING {
        type filter hook prerouting priority mangle + 10; policy accept;
        jump mangle_PREROUTING_ZONES
    }

    chain mangle_PREROUTING_POLICIES_pre {
        jump mangle_PRE_policy_allow-host-ipv6
    }

    chain mangle_PREROUTING_ZONES {
        iifname "ens160" goto mangle_PRE_FedoraWorkstation
        goto mangle_PRE_FedoraWorkstation
    }

    chain mangle_PREROUTING_POLICIES_post {
    }

    chain nat_PREROUTING {
        type nat hook prerouting priority dstnat + 10; policy accept;
        jump nat_PREROUTING_ZONES
    }

    chain nat_PREROUTING_POLICIES_pre {
        jump nat_PRE_policy_allow-host-ipv6
    }

    chain nat_PREROUTING_ZONES {
        iifname "ens160" goto nat_PRE_FedoraWorkstation
        goto nat_PRE_FedoraWorkstation
    }

    chain nat_PREROUTING_POLICIES_post {
    }

    chain nat_POSTROUTING {
        type nat hook postrouting priority srcnat + 10; policy accept;
        jump nat_POSTROUTING_ZONES
    }

    chain nat_POSTROUTING_POLICIES_pre {
    }

    chain nat_POSTROUTING_ZONES {
        oifname "ens160" goto nat_POST_FedoraWorkstation
        goto nat_POST_FedoraWorkstation
    }

    chain nat_POSTROUTING_POLICIES_post {
    }

    chain nat_OUTPUT {
        type nat hook output priority -90; policy accept;
        jump nat_OUTPUT_POLICIES_pre
        jump nat_OUTPUT_POLICIES_post
    }

    chain nat_OUTPUT_POLICIES_pre {
    }

    chain nat_OUTPUT_POLICIES_post {
    }

    chain filter_PREROUTING {
        type filter hook prerouting priority filter + 10; policy accept;
        icmpv6 type { nd-router-advert, nd-neighbor-solicit } accept
        meta nfproto ipv6 fib saddr . mark . iif oif missing drop
    }

    chain filter_INPUT {
        type filter hook input priority filter + 10; policy accept;
        ct state { established, related } accept
        ct status dnat accept
        iifname "lo" accept
        ct state invalid drop
        jump filter_INPUT_ZONES
        reject with icmpx admin-prohibited
    }

    chain filter_FORWARD {
        type filter hook forward priority filter + 10; policy accept;
        ct state { established, related } accept
        ct status dnat accept
        iifname "lo" accept
        ct state invalid drop
        ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 addr-unreachable
        jump filter_FORWARD_ZONES
        reject with icmpx admin-prohibited
    }

    chain filter_OUTPUT {
        type filter hook output priority filter + 10; policy accept;
        ct state { established, related } accept
        oifname "lo" accept
        ip6 daddr { ::/96, ::ffff:0.0.0.0/96, 2002::/24, 2002:a00::/24, 2002:7f00::/24, 2002:a9fe::/32, 2002:ac10::/28, 2002:c0a8::/32, 2002:e000::/19 } reject with icmpv6 addr-unreachable
        jump filter_OUTPUT_POLICIES_pre
        jump filter_OUTPUT_POLICIES_post
    }

    chain filter_INPUT_POLICIES_pre {
        jump filter_IN_policy_allow-host-ipv6
    }

    chain filter_INPUT_ZONES {
        iifname "ens160" goto filter_IN_FedoraWorkstation
        goto filter_IN_FedoraWorkstation
    }

    chain filter_INPUT_POLICIES_post {
    }

    chain filter_FORWARD_POLICIES_pre {
    }

    chain filter_FORWARD_ZONES {
        iifname "ens160" goto filter_FWD_FedoraWorkstation
        goto filter_FWD_FedoraWorkstation
    }

    chain filter_FORWARD_POLICIES_post {
    }

    chain filter_OUTPUT_POLICIES_pre {
    }

    chain filter_OUTPUT_POLICIES_post {
    }

    chain filter_IN_FedoraWorkstation {
        jump filter_INPUT_POLICIES_pre
        jump filter_IN_FedoraWorkstation_pre
        jump filter_IN_FedoraWorkstation_log
        jump filter_IN_FedoraWorkstation_deny
        jump filter_IN_FedoraWorkstation_allow
        jump filter_IN_FedoraWorkstation_post
        jump filter_INPUT_POLICIES_post
        meta l4proto { icmp, ipv6-icmp } accept
        reject with icmpx admin-prohibited
    }

    chain filter_IN_FedoraWorkstation_pre {
    }

    chain filter_IN_FedoraWorkstation_log {
    }

    chain filter_IN_FedoraWorkstation_deny {
    }

    chain filter_IN_FedoraWorkstation_allow {
        ip6 daddr fe80::/64 udp dport 546 ct state { new, untracked } accept
        tcp dport 22 ct state { new, untracked } accept
        udp dport 137 ct helper set "helper-netbios-ns-udp"
        udp dport 137 ct state { new, untracked } accept
        udp dport 138 ct state { new, untracked } accept
        ip daddr 224.0.0.251 udp dport 5353 ct state { new, untracked } accept
        ip6 daddr ff02::fb udp dport 5353 ct state { new, untracked } accept
        udp dport 1025-65535 ct state { new, untracked } accept
        tcp dport 1025-65535 ct state { new, untracked } accept
    }

    chain filter_IN_FedoraWorkstation_post {
    }

    chain nat_POST_FedoraWorkstation {
        jump nat_POSTROUTING_POLICIES_pre
        jump nat_POST_FedoraWorkstation_pre
        jump nat_POST_FedoraWorkstation_log
        jump nat_POST_FedoraWorkstation_deny
        jump nat_POST_FedoraWorkstation_allow
        jump nat_POST_FedoraWorkstation_post
        jump nat_POSTROUTING_POLICIES_post
    }

    chain nat_POST_FedoraWorkstation_pre {
    }

    chain nat_POST_FedoraWorkstation_log {
    }

    chain nat_POST_FedoraWorkstation_deny {
    }

    chain nat_POST_FedoraWorkstation_allow {
    }

    chain nat_POST_FedoraWorkstation_post {
    }

    chain filter_FWD_FedoraWorkstation {
        jump filter_FORWARD_POLICIES_pre
        jump filter_FWD_FedoraWorkstation_pre
        jump filter_FWD_FedoraWorkstation_log
        jump filter_FWD_FedoraWorkstation_deny
        jump filter_FWD_FedoraWorkstation_allow
        jump filter_FWD_FedoraWorkstation_post
        jump filter_FORWARD_POLICIES_post
        reject with icmpx admin-prohibited
    }

    chain filter_FWD_FedoraWorkstation_pre {
    }

    chain filter_FWD_FedoraWorkstation_log {
    }

    chain filter_FWD_FedoraWorkstation_deny {
    }

    chain filter_FWD_FedoraWorkstation_allow {
    }

    chain filter_FWD_FedoraWorkstation_post {
    }

    chain nat_PRE_FedoraWorkstation {
        jump nat_PREROUTING_POLICIES_pre
        jump nat_PRE_FedoraWorkstation_pre
        jump nat_PRE_FedoraWorkstation_log
        jump nat_PRE_FedoraWorkstation_deny
        jump nat_PRE_FedoraWorkstation_allow
        jump nat_PRE_FedoraWorkstation_post
        jump nat_PREROUTING_POLICIES_post
    }

    chain nat_PRE_FedoraWorkstation_pre {
    }

    chain nat_PRE_FedoraWorkstation_log {
    }

    chain nat_PRE_FedoraWorkstation_deny {
    }

    chain nat_PRE_FedoraWorkstation_allow {
    }

    chain nat_PRE_FedoraWorkstation_post {
    }

    chain mangle_PRE_FedoraWorkstation {
        jump mangle_PREROUTING_POLICIES_pre
        jump mangle_PRE_FedoraWorkstation_pre
        jump mangle_PRE_FedoraWorkstation_log
        jump mangle_PRE_FedoraWorkstation_deny
        jump mangle_PRE_FedoraWorkstation_allow
        jump mangle_PRE_FedoraWorkstation_post
        jump mangle_PREROUTING_POLICIES_post
    }

    chain mangle_PRE_FedoraWorkstation_pre {
    }

    chain mangle_PRE_FedoraWorkstation_log {
    }

    chain mangle_PRE_FedoraWorkstation_deny {
    }

    chain mangle_PRE_FedoraWorkstation_allow {
    }

    chain mangle_PRE_FedoraWorkstation_post {
    }

    chain filter_IN_policy_allow-host-ipv6 {
        jump filter_IN_policy_allow-host-ipv6_pre
        jump filter_IN_policy_allow-host-ipv6_log
        jump filter_IN_policy_allow-host-ipv6_deny
        jump filter_IN_policy_allow-host-ipv6_allow
        jump filter_IN_policy_allow-host-ipv6_post
    }

    chain filter_IN_policy_allow-host-ipv6_pre {
    }

    chain filter_IN_policy_allow-host-ipv6_log {
    }

    chain filter_IN_policy_allow-host-ipv6_deny {
    }

    chain filter_IN_policy_allow-host-ipv6_allow {
        icmpv6 type nd-neighbor-advert accept
        icmpv6 type nd-neighbor-solicit accept
        icmpv6 type nd-router-advert accept
        icmpv6 type nd-redirect accept
    }

    chain filter_IN_policy_allow-host-ipv6_post {
    }

    chain nat_PRE_policy_allow-host-ipv6 {
        jump nat_PRE_policy_allow-host-ipv6_pre
        jump nat_PRE_policy_allow-host-ipv6_log
        jump nat_PRE_policy_allow-host-ipv6_deny
        jump nat_PRE_policy_allow-host-ipv6_allow
        jump nat_PRE_policy_allow-host-ipv6_post
    }

    chain nat_PRE_policy_allow-host-ipv6_pre {
    }

    chain nat_PRE_policy_allow-host-ipv6_log {
    }

    chain nat_PRE_policy_allow-host-ipv6_deny {
    }

    chain nat_PRE_policy_allow-host-ipv6_allow {
    }

    chain nat_PRE_policy_allow-host-ipv6_post {
    }

    chain mangle_PRE_policy_allow-host-ipv6 {
        jump mangle_PRE_policy_allow-host-ipv6_pre
        jump mangle_PRE_policy_allow-host-ipv6_log
        jump mangle_PRE_policy_allow-host-ipv6_deny
        jump mangle_PRE_policy_allow-host-ipv6_allow
        jump mangle_PRE_policy_allow-host-ipv6_post
    }

    chain mangle_PRE_policy_allow-host-ipv6_pre {
    }

    chain mangle_PRE_policy_allow-host-ipv6_log {
    }

    chain mangle_PRE_policy_allow-host-ipv6_deny {
    }

    chain mangle_PRE_policy_allow-host-ipv6_allow {
    }

    chain mangle_PRE_policy_allow-host-ipv6_post {
    }
}
table ip6 clatd {
    chain napt66 {
        type nat hook postrouting priority srcnat; policy accept;
        iifname "clat" masquerade
    }
}

toreanderson commented 1 year ago

Looks to me that filter_FORWARD rejects all packets. Does it work if you stop firewalld? No output from ip6tables-save? (clatd should have added the necessary rules there)

Lucaslah commented 1 year ago

Yeah ip6tables-save returns nothing, Ill try stopping firewalld

Lucaslah commented 1 year ago

As soon as I stopped firewalld it started working.

toreanderson commented 1 year ago

Could you try with ip6tables-enable=true in the configuration file? I think what is happening is that you did not have the ip6table_filter kernel module loaded, so clatd skips adding the rules. This is based on an assumption that with no ip6table_filter, all forwarded traffic is permitted by default. However, that's not necessarily true since nft became a thing.

ip6tables-enable=true should force clatd to add the rules no matter what. (Note that you running ip6table-save might have loaded the ip6table_filter module, so you might want to test after a reboot to make sure it works in all cases.)

Lucaslah commented 1 year ago

After adding ip6tables-enable=true and rebooting, I still cannot access the internet on the clat interface unless I disable firewalld

user@fedora ~]$ sudo ip6tables-save
[sudo] password for user: 
# Generated by ip6tables-save v1.8.7 on Tue Mar 14 09:57:53 2023
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -i ens160 -o clat -j ACCEPT
-A FORWARD -i clat -o ens160 -j ACCEPT
COMMIT
# Completed on Tue Mar 14 09:57:53 2023

toreanderson commented 1 year ago

Ok, I guess that ip6tables and nft aren't really compatible with each other. You can try getting around that by setting ip6tables-enable=false (to not bother with ip6tables at all) and instead try the following when clatd (and firewalld) is running:

nft 'add chain ip6 clatd forwarding { type filter hook forward priority filter; }; add rule ip6 clatd forwarding iifname "clatd" accept; add rule ip6 clatd forwarding oifname "clatd" accept'

If that works, you can add these commands to your clatd.conf so that it reads:

script-up=nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade; add chain ip6 clatd forwarding { type filter hook forward priority filter; }; add rule ip6 clatd forwarding iifname "clat" accept; add rule ip6 clatd forwarding oifname "clat" accept'

Edit: it might be that you need to increase the priority in order to short-circuit firewalld. If it doesn't work, try again with priority 120 and see if that is better. Let me know how it goes!

Lucaslah commented 1 year ago

Set ip6tables-enable=false in /etc/clatd.conf and then started firewalld and clatd, and then ran nft 'add chain ip6 clatd forwarding { type filter hook forward priority filter; }; add rule ip6 clatd forwarding iifname "clatd" accept; add rule ip6 clatd forwarding oifname "clatd" accept' still cannot access internet on the clat interface.

Where would I add priority 120 to?

toreanderson commented 1 year ago

Argh, I made a mistake, sorry. It should have been iifname "clat" and oifname "clat" (not clat*d*). That is:

nft 'add chain ip6 clatd forwarding { type filter hook forward priority filter; }; add rule ip6 clatd forwarding iifname "clat" accept; add rule ip6 clatd forwarding oifname "clat" accept'

If that doesn't work, delete what you have with nft 'delete table ip6 clatd' and try again with an increased priority:

nft 'add chain ip6 clatd forwarding { type filter hook forward priority 120; }; add rule ip6 clatd forwarding iifname "clat" accept; add rule ip6 clatd forwarding oifname "clat" accept'

Lucaslah commented 1 year ago

Unfortunately I tried with both and still cannot get it to work.

toreanderson commented 1 year ago

It is possible that it can't work. I am not sure, but it might be that the reject with icmpx admin-prohibited rule in firewalld's chain filter_FORWARD cannot be overridden by other nft tables. If so you would need to reconfigure firewalld to allow that traffic. If you find out how, I would be interested in knowing about it.

thpryrchn commented 1 year ago

Great to hear that it worked! You can make this happen automatically by using script-up and script-down, for example by creating an /etc/clatd.conf file containing the following:
clat-v6-addr=fd00::c1a7
proxynd-enable=false
script-up=nft 'add table ip6 clatd; add chain ip6 clatd napt66 { type nat hook postrouting priority srcnat; }; add rule ip6 clatd napt66 iifname clat masquerade'
script-down=nft 'delete table ip6 clatd'
I will consider implementing native support for this in clatd in the future.

This did not work for me... I don't have any firewall installed. Im in Debian 12..

But I did find something that does!

clat-v6-addr=fd00::c1a7
proxynd-enable=false
script-up=nft "add table ip6 clatd" ; nft "add chain ip6 clatd POSTROUTING { type nat hook postrouting priority srcnat; }" ; nft "add rule ip6 clatd POSTROUTING masquerade"
script-down=nft 'delete table ip6 clatd'

toreanderson commented 1 year ago

This did not work for me... I don't have any firewall installed. Im in Debian 12..

But I did find something that does!
clat-v6-addr=fd00::c1a7
proxynd-enable=false
script-up=nft "add table ip6 clatd" ; nft "add chain ip6 clatd POSTROUTING { type nat hook postrouting priority srcnat; }" ; nft "add rule ip6 clatd POSTROUTING masquerade"
script-down=nft 'delete table ip6 clatd'

Interesting. The only actual difference between the two, as far as I can tell, is that you do not match on iifname clat. The other differences, namely that you call your chain something else than I did (POSTROUTING vs napt66), and that you inject the rules using multiple invocations of nft instead of a single one, should not make any difference.

Have you renamed the CLAT device by setting clat-dev to something else than its default value clat? That would explain why it didn't work for you.

toreanderson / clatd

Could not find a global IPv6 address on enp2s0 from which to derive a CLAT IPv6 address #23