Closed Firstyear closed 4 years ago
Hmm. I do not quite understand why you need this change..
On my system, when the net/ipv6/conf/all/forwarding
sysctl is changed (as clatd already does), all the corresponding per-interface forwarding syscts are changed also, as demonstrated below (with NetworkManager running):
[:~] $ grep . /proc/sys/net/ipv6/conf/*/forwarding
/proc/sys/net/ipv6/conf/all/forwarding:0
/proc/sys/net/ipv6/conf/default/forwarding:0
/proc/sys/net/ipv6/conf/enp0s31f6/forwarding:0
/proc/sys/net/ipv6/conf/lo/forwarding:0
/proc/sys/net/ipv6/conf/msvpn/forwarding:0
/proc/sys/net/ipv6/conf/virbr0/forwarding:0
/proc/sys/net/ipv6/conf/virbr0-nic/forwarding:0
/proc/sys/net/ipv6/conf/wlp2s0/forwarding:0
/proc/sys/net/ipv6/conf/wwp0s20f0u3c3/forwarding:0
[:~] $ sudo sysctl net/ipv6/conf/all/forwarding=1
net.ipv6.conf.all.forwarding = 1
[:~] $ grep . /proc/sys/net/ipv6/conf/*/forwarding
/proc/sys/net/ipv6/conf/all/forwarding:1
/proc/sys/net/ipv6/conf/default/forwarding:1
/proc/sys/net/ipv6/conf/enp0s31f6/forwarding:1
/proc/sys/net/ipv6/conf/lo/forwarding:1
/proc/sys/net/ipv6/conf/msvpn/forwarding:1
/proc/sys/net/ipv6/conf/virbr0/forwarding:1
/proc/sys/net/ipv6/conf/virbr0-nic/forwarding:1
/proc/sys/net/ipv6/conf/wlp2s0/forwarding:1
/proc/sys/net/ipv6/conf/wwp0s20f0u3c3/forwarding:1
As I understand it, your code would additionally perform sysctl net/ipv6/conf/${plat_dev}/forwarding=1
at this point. That seems to be a pointless no-op, since net/ipv6/conf/${plat_dev}/forwarding
is already 1
.
What am I missing?
On my system I'm seeing situations where after network manager starts and clatd starts, that net/ipv6/conf/${plat_dev}/forwarding is 0, but /proc/sys/net/ipv6/conf/all/forwarding is 1, which causes the traffic to fail to operate. That's why I created this patch to resolve that situation.
# grep . /proc/sys/net/ipv6/conf/*/forwarding
/proc/sys/net/ipv6/conf/all/forwarding:0
/proc/sys/net/ipv6/conf/default/forwarding:0
/proc/sys/net/ipv6/conf/eno1/forwarding:0
/proc/sys/net/ipv6/conf/lo/forwarding:0
# After ifup
/proc/sys/net/ipv6/conf/all/forwarding:1
/proc/sys/net/ipv6/conf/clat/forwarding:1
/proc/sys/net/ipv6/conf/default/forwarding:1
/proc/sys/net/ipv6/conf/eno1/forwarding:0
/proc/sys/net/ipv6/conf/lo/forwarding:1
Interesting. Are you sure that it is NetworkManager that disables forwarding on the eno1
device? Can you send me the output of these commands (without clatd running, performing ifup/ifdown or anything like that)?
grep . /proc/sys/net/ipv6/conf/*/forwarding
sudo sysctl net/ipv6/conf/all/forwarding=1
grep . /proc/sys/net/ipv6/conf/*/forwarding
I have a hypothesis about what is actually going on here. I believe that at the point in time where clatd starts, your net/ipv6/conf/all/forwarding
is already 1, while net/ipv6/conf/eno1/forwarding
is 0.
When net/ipv6/conf/all/forwarding
is already 1, the code that would set this sysctl is skipped:
if(cfgbool("forwarding-enable")) {
if(sysctl("net/ipv6/conf/all/forwarding") == 0) { # <-- this condition evaluates to false
p("Enabling IPv6 forwarding");
# not reached!
…
You could confirm this hypothesis by checking if clatd emits the «Enabling IPv6 forwarding» message or not.
You could also try running sysctl net/ipv6/conf/all/forwarding=0
before starting (unpatched) clatd, and confirm that if it works correctly then (outputting the «Enabling IPv6 forwarding» message as it starts up).
# Boot without clatd + nm
# grep . /proc/sys/net/ipv6/conf/*/forwarding
/proc/sys/net/ipv6/conf/all/forwarding:1
/proc/sys/net/ipv6/conf/default/forwarding:1
/proc/sys/net/ipv6/conf/eno1/forwarding:0
/proc/sys/net/ipv6/conf/lo/forwarding:1
# start clatd
# grep . /proc/sys/net/ipv6/conf/*/forwarding
/proc/sys/net/ipv6/conf/all/forwarding:1
/proc/sys/net/ipv6/conf/clat/forwarding:1
/proc/sys/net/ipv6/conf/default/forwarding:1
/proc/sys/net/ipv6/conf/eno1/forwarding:0
/proc/sys/net/ipv6/conf/lo/forwarding:1
# Log
-- Reboot --
Apr 10 14:14:24 spotinuc systemd[1]: Started 464XLAT CLAT daemon.
Apr 10 14:14:24 spotinuc clatd[1003]: Starting clatd v1.5 by Tore Anderson <tore@fud.no>
Apr 10 14:14:24 spotinuc clatd[1003]: Performing DNS64-based PLAT prefix discovery (cf. RFC 7050)
Apr 10 14:14:26 spotinuc.prd.blackhats.net.au systemd[1]: Stopping 464XLAT CLAT daemon...
Apr 10 14:14:26 spotinuc.prd.blackhats.net.au systemd[1]: clatd.service: Succeeded.
Apr 10 14:14:26 spotinuc.prd.blackhats.net.au systemd[1]: Stopped 464XLAT CLAT daemon.
Apr 10 14:14:26 spotinuc.prd.blackhats.net.au systemd[1]: Started 464XLAT CLAT daemon.
Apr 10 14:14:26 spotinuc.prd.blackhats.net.au clatd[1544]: Starting clatd v1.5 by Tore Anderson <tore@fud.no>
Apr 10 14:14:26 spotinuc.prd.blackhats.net.au clatd[1544]: Performing DNS64-based PLAT prefix discovery (cf. RFC 7050)
Apr 10 14:14:31 spotinuc.prd.blackhats.net.au clatd[1544]: No PLAT prefix could be discovered. Your ISP probably doesn't provide NAT64/DNS64 PLAT service. Exiting.
Apr 10 14:14:31 spotinuc.prd.blackhats.net.au systemd[1]: clatd.service: Succeeded.
Apr 10 14:14:35 spotinuc.prd.blackhats.net.au systemd[1]: Started 464XLAT CLAT daemon.
Apr 10 14:14:35 spotinuc.prd.blackhats.net.au clatd[2573]: Starting clatd v1.5 by Tore Anderson <tore@fud.no>
Apr 10 14:14:35 spotinuc.prd.blackhats.net.au clatd[2573]: Performing DNS64-based PLAT prefix discovery (cf. RFC 7050)
Apr 10 14:14:35 spotinuc.prd.blackhats.net.au clatd[2573]: Using PLAT (NAT64) prefix: 2001:44b8:2155:2c64::/96
Apr 10 14:14:35 spotinuc.prd.blackhats.net.au clatd[2573]: Device facing the PLAT: eno1
Apr 10 14:14:35 spotinuc.prd.blackhats.net.au clatd[2573]: Attempting to derive a CLAT IPv6 address from an IPv6 address on 'eno1'
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Using CLAT IPv4 address: 192.0.0.1
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Using CLAT IPv6 address: 2001:44b8:2155:2c13:96c6:91c1:a717:53ef
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Checking if this system already has IPv4 connectivity in 10 sec(s)
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Enabling Proxy-ND for 2001:44b8:2155:2c13:96c6:91c1:a717:53ef on eno1
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Creating and configuring up CLAT device 'clat'
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2835]: Created persistent tun device clat
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Adding IPv4 default route via the CLAT
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au clatd[2573]: Starting up TAYGA, using config file '/tmp/mUMgsuqDkm'
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au tayga[2842]: starting TAYGA 0.9.2
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au tayga[2842]: Using tun device clat with MTU 1500
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au tayga[2842]: TAYGA's IPv4 address: 192.0.0.2
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au tayga[2842]: TAYGA's IPv6 address: 2001:44b8:2155:2c64::c000:2
Apr 10 14:14:46 spotinuc.prd.blackhats.net.au tayga[2842]: NAT64 prefix: 2001:44b8:2155:2c64::/96
Apr 10 18:11:11 spotinuc.prd.blackhats.net.au tayga[2842]: exiting on signal 15
Apr 10 18:11:11 spotinuc.prd.blackhats.net.au systemd[1]: Stopping 464XLAT CLAT daemon...
Apr 10 18:11:11 spotinuc.prd.blackhats.net.au clatd[2573]: TAYGA terminated, cleaning up and exiting
Apr 10 18:11:11 spotinuc.prd.blackhats.net.au clatd[4772]: Removed persistent tun device clat
Apr 10 18:11:12 spotinuc.prd.blackhats.net.au clatd[4780]: RTNETLINK answers: No such file or directory
Apr 10 18:11:12 spotinuc.prd.blackhats.net.au clatd[2573]: <warn> cmd(ip -6 neighbour delete proxy 2001:44b8:2155:2c13:96c6:91c1:a717:53ef dev eno1) returned 0
Apr 10 18:11:12 spotinuc.prd.blackhats.net.au systemd[1]: clatd.service: Succeeded.
Apr 10 18:11:12 spotinuc.prd.blackhats.net.au systemd[1]: Stopped 464XLAT CLAT daemon.
Apr 10 18:11:12 spotinuc.prd.blackhats.net.au systemd[1]: Started 464XLAT CLAT daemon.
Here's the output, and yep, there is no "Enabling IPv6 forwarding" message. I checked and I have nothing in sysctl.d that's doing the forward=1, but the forward=0 on eno1 certainly comes from network manager.
the forward=0 on eno1 certainly comes from network manager.
The command output you pasted says otherwise:
# Boot without clatd + nm # grep . /proc/sys/net/ipv6/conf/*/forwarding /proc/sys/net/ipv6/conf/all/forwarding:1 /proc/sys/net/ipv6/conf/default/forwarding:1 /proc/sys/net/ipv6/conf/eno1/forwarding:0
In any case, this situation should be handled better. I think a better solution might be to only change the sysctl on the PLAT device, though (ignoring the all
one completely). I will look into it.
Could you adapt the patch I provided to only set the forwarding on the PLAT device?
Yes, something like this. I haven't tested it as I don't have NAT64 service where I am right now. Does it work for you?
diff --git a/clatd b/clatd
index 1003d00..8bbfca9 100755
--- a/clatd
+++ b/clatd
@@ -579,7 +579,7 @@ sub get_clat_v6_addr {
my $cleanup_remove_clat_dev; # true if having created it
my $cleanup_delete_taygaconf; # true if having made a temp confile
my $cleanup_zero_forwarding_sysctl; # zero forwarding sysctl if set
-my @cleanup_accept_ra_sysctls; # accept_ra sysctls to be reset to '1'
+my $cleanup_accept_ra_sysctl; # reset accept_ra sysctl tto '1'
my $cleanup_zero_proxynd_sysctl; # zero proxy_ndp sysctl if set
my $cleanup_remove_proxynd_entry, # true if having added proxynd entry
my $cleanup_remove_ip6tables_rules; # true if having added ip6tables rules
@@ -599,11 +599,11 @@ sub cleanup_and_exit {
}
if(defined($cleanup_zero_forwarding_sysctl)) {
d("Cleanup: Resetting forwarding sysctl to 0");
- sysctl("net/ipv6/conf/all/forwarding", 0);
+ sysctl("net/ipv6/conf/" . cfg("plat-dev") . "/forwarding", 0);
}
- for my $sysctl (@cleanup_accept_ra_sysctls) {
- d("Cleanup: Resetting $sysctl to 1");
- sysctl($sysctl, 1);
+ if(defined($cleanup_accept_ra_sysctl)) {
+ d("Cleanup: Resetting accept_ra sysctl to 1");
+ sysctl("net/ipv6/conf/" . cfg("plat-dev") . "/accept_ra", 1);
}
if(defined($cleanup_zero_proxynd_sysctl)) {
d("Cleanup: Resetting proxy_ndp sysctl to 0");
@@ -803,20 +803,13 @@ close($tayga_conffile_fh) or err("close($tayga_conffile_fh: $!");
# Enable IPv6 forwarding if necessary
#
if(cfgbool("forwarding-enable")) {
- if(sysctl("net/ipv6/conf/all/forwarding") == 0) {
- p("Enabling IPv6 forwarding");
- for my $ctl (glob("/proc/sys/net/ipv6/conf/*/accept_ra")) {
-
- # Don't touch the ctl for the "all" interface, as that will probably
- # change interfaces that have accept_ra set to 0 also.
- next if($ctl eq "/proc/sys/net/ipv6/conf/all/accept_ra");
-
- if(sysctl($ctl) == 1) {
- d("Changing $ctl from 1 to 2 to prevent connectivity loss after ",
- "enabling IPv6 forwarding");
- sysctl($ctl, 2);
- push(@cleanup_accept_ra_sysctls, $ctl);
- }
+ if(sysctl("net/ipv6/conf/" . cfg("plat-dev") . "/forwarding") == 0) {
+ p("Enabling IPv6 forwarding on PLAT device");
+ if(sysctl("/proc/sys/net/ipv6/conf/" . cfg("plat-dev") . "/accept_ra") == 1) {
+ d("Changing accept_ra from 1 to 2 to prevent connectivity loss after ",
+ "enabling IPv6 forwarding");
+ sysctl("/proc/sys/net/ipv6/conf/" . cfg("plat-dev") . "/accept_ra", 2);
+ $cleanup_accept_ra_sysctl = 1;
}
sysctl("net/ipv6/conf/all/forwarding", 1);
$cleanup_zero_forwarding_sysctl = 0;
@@ -855,13 +848,15 @@ if(cfgbool("proxynd-enable")) {
}
#
-# Create the CLAT tun interface, add the IPv4 address to it as well as the
-# route to the corresponding IPv6 address, and possibly an IPv4 default route
+# Create the CLAT tun interface, enable forwarding, add the IPv4 address to it
+# as well as the route to the corresponding IPv6 address, and possibly an IPv4
+# default route
#
p("Creating and configuring up CLAT device '", cfg("clat-dev"), "'");
cmd(\&err, cfg("cmd-tayga"), "--config", cfg("tayga-conffile"), "--mktun",
cfgint("debug") ? "-d" : "");
$cleanup_remove_clat_dev = 1;
+sysctl("net/ipv6/conf/" . cfg("clat-dev") . "/forwarding", 1);
cmd(\&err, cfg("cmd-ip"), qw(link set up dev), cfg("clat-dev"));
cmd(\&err, cfg("cmd-ip"), qw(-4 address add), cfg("clat-v4-addr"),
"dev", cfg("clat-dev"));
Do you have some extra commits in your repo that aren't in master? The patch doesn't apply cleanly in my repo, and it looks like the git indexes in your diff don't match any commits I have in the repo ...
commit 41a312f9080e987f2fe8d615b58c3baa9db3da4d (HEAD -> plat-dev, origin/master, origin/HEAD, master)
Anyway I manually applied your changes, and it appears that they work as expected. Thanks so much!
I was missing one patch in my local checkout, but that only had changes to Makefile
, so I am not sure why the patch did not apply cleanly for you. Whitespace mismatch due to copy&paste, maybe?
In any case, I pushed it to a branch now instead, so you can get it that way instead: https://github.com/toreanderson/clatd/tree/forwarding_sysctl_on_plat_dev_only
I'll test some more in the weekend and merge to master if everything works OK.
Sounds great, thank you very much for your help! It's much appreciated!
So I've done a bit of testing, and realised that the patch I sent contains a bug: I forgot to update the sysctl("net/ipv6/conf/all/forwarding", 1);
line to use cfg("clat-dev")
instead of all
. :man_facepalming:
Anyway, I tried to reproduce your original issue, but I failed to. If net/ipv6/conf/all/forwarding=1
on my system, clatd works - even though all the other net/ipv6/conf/$dev/forwarding
sysctls are 0
. In other words:
[:~] $ grep . /proc/sys/net/ipv6/conf/*/forwarding
/proc/sys/net/ipv6/conf/all/forwarding:1
/proc/sys/net/ipv6/conf/clat/forwarding:0
/proc/sys/net/ipv6/conf/default/forwarding:0
/proc/sys/net/ipv6/conf/enp0s31f6/forwarding:0
/proc/sys/net/ipv6/conf/lo/forwarding:0
/proc/sys/net/ipv6/conf/virbr0/forwarding:0
/proc/sys/net/ipv6/conf/virbr0-nic/forwarding:0
/proc/sys/net/ipv6/conf/wlp2s0/forwarding:0
/proc/sys/net/ipv6/conf/wwp0s20f0u3c3/forwarding:0
[:~] $ ip r get 1.1.1.1
1.1.1.1 dev clat src 192.0.0.1 uid 1000
cache mtu 1260 advmss 1220
[:~] $ ping -c1 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=49 time=108 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 107.777/107.777/107.777/0.000 ms
This appears to be the documented behaviour. The net/ipv6/conf/all/forwarding
is a global on/off toggle that determines whether or not IPv4 traffic is forwarded or not, while the per-interface forwarding sysctls is only supposed to control whether or not the interface processes Neighbour Discovery packets as if it was a host or a router by default - the per-interface knobs do not control whether or not IPv6 packets are being forwarded - and this is the reason why clatd sets the all
knob and don't touch any of the per-interface ones.
Therefore, the behaviour in your report is very strange. If the all
sysctl is set, clatd should have worked.
Could you reproduce the problem using the master branch, please? Once clatd is running, start a ping process towards 1.1.1.1
or something, and then in a different terminal run sudo sysctl net/ipv6/conf/eno1/forwarding
. Return to the ping, does answers now come in?
It would also be interesting to repeat the test, but instead run sudo sysctl net/ipv6/conf/all/forwarding
. Even though this sysctl should already be set to 1 at this point, I am curious to learn whether or not this actually makes a difference regardless.
Okay, so the series of steps with the unpatched clatd is:
Boot system
Start clatd + nm
nmcli conn down if && nmcli conn up if
restart clatd
At this point traffic fails. After these steps the sysctls are:
# sysctl -a | grep -i ipv6 | grep -i forward
net.ipv6.conf.all.forwarding = 1
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.clat.forwarding = 1
net.ipv6.conf.clat.mc_forwarding = 0
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.eno1.forwarding = 0
net.ipv6.conf.eno1.mc_forwarding = 0
net.ipv6.conf.lo.forwarding = 1
net.ipv6.conf.lo.mc_forwarding = 0
Running this command resolves it:
spotinuc:/tmp # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1012ms
spotinuc:/tmp # sysctl -w net.ipv6.conf.eno1.forwarding=1
net.ipv6.conf.eno1.forwarding = 1
spotinuc:/tmp # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=30.4 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 30.384/30.384/30.384/0.000 ms
However, after running that if I do:
spotinuc:/tmp # sysctl -w net.ipv6.conf.eno1.forwarding=0
net.ipv6.conf.eno1.forwarding = 0
spotinuc:/tmp # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=23.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=49 time=27.4 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 23.268/25.321/27.375/2.053 ms
So I suspect there is something else suspect occurring here. Perhaps there is a kernel level issue that I am not aware of ....
Linux spotinuc.prd.blackhats.net.au 5.6.2-1-default #1 SMP Thu Apr 2 06:31:32 UTC 2020 (c8170d6) x86_64 x86_64 x86_64 GNU/Linux
That is quite strange.
Could you use tcpdump
try to figure out exactly where the packets disappear? While pinging to 8.8.8.8 when it does not work, look for packets as follows:
tcpdump -Q out -eni clat dst host 8.8.8.8
tcpdump -Q in -eni clat dst host 2001:44b8:2155:2c64::8.8.8.8
tcpdump -Q out -eni eno1 dst host 2001:44b8:2155:2c64::8.8.8.8
tcpdump -Q in -eni eno1 src host 2001:44b8:2155:2c64::8.8.8.8
tcpdump -Q out -eni clat src host 2001:44b8:2155:2c64::8.8.8.8
tcpdump -Q in -eni clat src host 8.8.8.8
It might be easier to figure out what is going on if we figure out exactly where the packets vanish.
I'm really confused now - I can't reproduce this error at all. It appears that my system did a full update on the 21st which upgraded the kernel from 5.6.0-1.2
to 5.6.4-1.2
, and I no longer see the behaviour with network manager setting eno1 to forward=0, and ifdown/up, suspend/resume, and network manager restarts all seem to be able to have no issue with the unpatched original clatd code.
So I wonder if this was a kernel issue instead ...
Anyway, I guess that at this point, no change is needed, so I'm sorry for the noise :(
No worries, thank you for your report! The behaviour you described seemed to run counter to the documentation, so it would not surprise me if it was indeed a kernel bug that got fixed.
Systems that use network manager (typically clients, which is where clatd will be deployed) have some unique issues with clatd, because nm controls the whole stack in ways that break clatd. Generally this means that nm disables the forwarding flags on the interfaces, which causes traffic to not operate as expected.
This adds support to manually reset the plat dev forwarding flag in the situation that nm or other tools have disabled it.