search

systemd / systemd

The systemd System and Service Manager
https://systemd.io
GNU General Public License v2.0
13.29k stars 3.8k forks source link

networkd: IPv6 rapid rotation of temporary addresses #20050

Closed ghen2 closed 3 years ago

ghen2 commented 3 years ago

systemd version the issue has been seen with

249rc2

Used distribution

Arch Linux

Linux kernel version used (uname -a)

5.12.13-arch1-2

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see

stable IPv6 privacy addresses for (long running) outbound connections

Unexpected behaviour you saw

rapid rotation of privacy addresses, breaking current connections

Steps to reproduce the problem

upgrade to systemd 249, restart networkd

Additional program output to the terminal or log subsystem illustrating the issue

Below is a diff of networkctl status 3 with just a few minutes in a between. A new temporary privacy address has been added and the old one immediatly removed, breaking all existing connections with that old address. This happens every few minutes, despite the much longer valid_lifetime indicated by ip addr list.

Normally, old addresses are first put in "deprecated" state, so they are no longer used for new connections, and removed later. Plus rotation should be much less frequent, respecting valid_lifetime of the addresses (hours for global addresses, days for ULA).

I noticed this behaviour with systemd 249rc2, coming from 248 where it was stable.

 * 3: eno1                       
                      Link File: /usr/lib/systemd/network/99-default.link
                   Network File: /etc/systemd/network/10-ethernet.network
                           Type: ether
                          State: routable (configured)
                   Online state: online
              Alternative Names: enp0s31f6
                           Path: pci-0000:00:1f.6
                         Driver: e1000e
                         Vendor: Intel Corporation
                          Model: Ethernet Connection (6) I219-V
                     HW Address: 1c:69:7a:0b:e1:fe (EliteGroup Computer Systems Co., LTD)
                            MTU: 1500 (min: 68, max: 9000)
                          QDisc: fq_codel
   IPv6 Address Generation Mode: eui64
           Queue Length (Tx/Rx): 1/1
               Auto negotiation: yes
                          Speed: 1Gbps
                         Duplex: full
                           Port: tp
                        Address: 172.16.1.3
                                 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe
-                                2a02:1810:4d12:7000:d342:cf65:aee:5e5e
+                                2a02:1810:4d12:7000:e4d7:c363:6f04:c072
                                 fd1b:229d:8fec::3
-                                fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe
-                                fd1b:229d:8fec:0:d6c5:91d1:211c:5fcb
                                 fe80::1e69:7aff:fe0b:e1fe
                        Gateway: 172.16.1.254
                                 fe80::c43d:c7ff:fea3:20bd
                            DNS: fd1b:229d:8fec::1
              Activation Policy: up
            Required For Online: yes
              DHCP6 Client IAID: 0xb6220feb
              DHCP6 Client DUID: DUID-EN/Vendor:0000ab117803f0a3b327c68e0000
                   Connected To: n/a on port a4:4c:c8:1b:50:63

ip -6 addr list

3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 2a02:1810:4d12:7000:e4d7:c363:6f04:c072/64 scope global temporary dynamic 
       valid_lft 36712sec preferred_lft 7912sec
    inet6 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 36712sec preferred_lft 7912sec
    inet6 fd1b:229d:8fec::3/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::1e69:7aff:fe0b:e1fe/64 scope link 
       valid_lft forever preferred_lft forever
ghen2 commented 3 years ago

Looks like it's starting from an empty state with every cycle:

Jun 28 10:32:10 systemd-networkd[222]: eno1: NDISC: Received Router Advertisement: flags OTHER preference medium lifetime 1800 sec
Jun 28 10:32:10 systemd-networkd[222]: eno1: NDISC: Invoking callback for 'router' event.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Acquiring DHCPv6 lease on NDisc request
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 1s, preferred for 2h 1min 1s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Requesting route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Setting SLAAC addresses.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Setting NDisc routes.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing old NDisc information obtained from fe80::c43d:c7ff:fea3:20bd.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing address: fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe/64 (valid forever, preferred forever), flags: permanent,manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 26s, preferred for 2h 1min 26s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Removing route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Remembering updated address: fd1b:229d:8fec:0:5487:a11b:9754:7d2a/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe/64 (valid forever, preferred forever), flags: permanent,manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec:0:1e69:7aff:fe0b:e1fe/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: fd1b:229d:8fec:0:5487:a11b:9754:7d2a/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec:0:5487:a11b:9754:7d2a/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Remembering updated address: 2a02:1810:4d12:7000:981c:e4a7:7023:46ff/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 2s, preferred for 2h 1min 2s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting address: 2a02:1810:4d12:7000:981c:e4a7:7023:46ff/64 (valid for 0, preferred for 0), flags: temporary
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000:981c:e4a7:7023:46ff/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 1s, preferred for 2h 1min 1s), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Forgetting route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Configuring route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Remembering updated address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min 1s, preferred for 2h 1min 1s), flags: tentative,manage-temporary-address,no-prefixroute
Jun 28 10:32:10 systemd-networkd[222]: eno1: NDisc SLAAC addresses set.
Jun 28 10:32:10 systemd-networkd[222]: eno1: No SLAAC address obtained from fe80::c43d:c7ff:fea3:20bd is ready. The old NDisc information will be removed later.
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: n/a, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: 2a02:1810:4d12:7000::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: fd1b:229d:8fec::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: 2a02:1810:4d12:7000::/56, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024 
Jun 28 10:32:10 systemd-networkd[222]: eno1: Received remembered route: dst: fd1b:229d:8fec::/48, src: n/a, gw: fe80::c43d:c7ff:fea3:20bd, prefsrc: n/a, scope: global, table: main(254), proto: ra, type: unicast, nexthop: 0, priority: 1024
Jun 28 10:32:10 systemd-networkd[222]: eno1: NDisc routes set.
Jun 28 10:32:10 systemd-networkd[222]: eno1: No SLAAC address obtained from fe80::c43d:c7ff:fea3:20bd is ready. The old NDisc information will be removed later. 
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering foreign address: 2a02:1810:4d12:7000:594:31f:d12c:59dd/64 (valid for 10h 1min, preferred for 2h 1min), flags: temporary
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering foreign route: dst: 2a02:1810:4d12:7000:594:31f:d12c:59dd/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering updated address: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/64 (valid for 10h 1min, preferred for 2h 1min), flags: manage-temporary-address,no-prefixroute
Jun 28 10:32:11 systemd-networkd[222]: eno1: Removing old NDisc information obtained from fe80::c43d:c7ff:fea3:20bd.
Jun 28 10:32:11 systemd-networkd[222]: eno1: Remembering foreign route: dst: 2a02:1810:4d12:7000:1e69:7aff:fe0b:e1fe/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0
keszybz commented 3 years ago

Sounds like we need another #19980, but this time for IPv6.

ghen2 commented 3 years ago

Please note that this is a regression in 249(rc2), this behaviour was not present in 248.

keszybz commented 3 years ago

Yeah, but there were 250 non-merge commits touching src/network in the period. If you could bisect this, that'd probably help. Please also paste /etc/systemd/network/10-ethernet.network.

FWIW, I don't see this here.

keszybz commented 3 years ago

This seems to be the last unadressed issue for v249. I think we should leave it for later, unless @yuwata has some good idea how to fix this.

yuwata commented 3 years ago

Hmm, this is caused by 0828a38605975b68c14c9194a1ee2c5c2ff7038f. But it fixes other issues...

ghen2 commented 3 years ago

Please also paste /etc/systemd/network/10-ethernet.network.

[Match]
Name=eno1

[Network]
Address=fd1b:229d:8fec::3/64
IPv6PrivacyExtensions=yes

Address=172.16.1.3/24
Gateway=172.16.1.254
DNS=fd1b:229d:8fec::1

 + SLAAC/privacy addressing based on prefix advertisements (ULA + global, with long lifetimes) from an OpenWRT router, no DHCPv6.

yuwata commented 3 years ago

@ghen2 If possible, could you test #20108?

ghen2 commented 3 years ago

Yes, https://github.com/systemd/systemd/pull/20108 works for me, thanks!

Another difference between 248 and 249 (with or without your patch) is that the SLAAC addresses in the ULA prefix continously appear and disappear (both fixed mac-based in temporary privacy addresses), which is strange. This does not happen for the globally routable prefix though, does networkd distinguish between those?

ghen2 commented 3 years ago

In fact, this causes the same issue for outgoing connections within ULA prefix (thus within my LAN). Your patch fixed it for "outside" connections (sourced from globally routable addresses) but not for local ones.

yuwata commented 3 years ago

Thank you for testing the PR so quickly! I added one more commit in the PR. I hope it fixes the ULA address issue. Please test gain.

ghen2 commented 3 years ago

Yes, the ULA case is fixed as well now. Thanks!

yuwata commented 3 years ago

Thank you! Your help is much appreciated.

vp1981 commented 3 years ago

Same here and PR#20108 fixed the issue (Archlinux, systemd-249rc3).