IPv4 LinkLocal addresses are not reachable when both interfaces have link local IPs

raviteja-b commented 1 year ago

systemd version the issue has been seen with

253

Used distribution

Linux

Linux kernel version used

6.1.12-f20a505

CPU architectures issue was seen on

arm

Component

systemd-networkd

Expected behaviour you didn't see

Configured two ethernet interfaces eth0 & eth1 with LinkLocalAddressing=yes eth1 interface IPv4 link local IP address it not reachable from the machine connected on same link.

Eth1 interface IPv4 link local IP address should be reachable.

Unexpected behaviour you saw

eth1 interface IPv4 link local IP address it not reachable from the machine connected on same link.

Steps to reproduce the problem

Configured eth0 & eth1 interfaces with LinkLocalAddressing=yes
connect a helper machine to eth1 interface via point to point ethernet cable.
ping eth1 interface IPv4 link local IP

Additional program output to the terminal or log subsystem illustrating the issue

ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 08:94:ef:81:ff:84 brd ff:ff:ff:ff:ff:ff inet 169.254.13.135/16 brd 169.254.255.255 scope link eth0 valid_lft forever preferred_lft forever inet6 fe80::a94:efff:fe81:ff84/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 08:94:ef:81:ff:85 brd ff:ff:ff:ff:ff:ff inet 169.254.202.113/16 brd 169.254.255.255 scope link eth1 valid_lft forever preferred_lft forever inet6 fe80::a94:efff:fe81:ff85/64 scope link valid_lft forever preferred_lft forever

route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 169.254.0.0 255.255.0.0 U 2048 0 0 eth0 169.254.0.0 255.255.0.0 U 2048 0 0 eth1

yuwata commented 1 year ago

Bunch of missing information in the report.

Please show .network file(s) you use.
Please show the result of ip route, instead of route command.
Please show the result of networkctl status ehX.
Please show the ping command you use and its result.

raviteja-b commented 1 year ago

cat /etc/systemd/network/00-bmc-eth0.network 
[Network]
LinkLocalAddressing=yes
[Link]
MACAddress=08:aa:ef:bb:00:14
[Match]
Name=Eth0
root@p10bmc:~# cat /etc/systemd/network/00-bmc-eth1.network 
[Network]
LinkLocalAddressing=yes
[Link]
MACAddress=08:aa:ef:bb:00:15
[Match]
Name=eth1

networkctl status eth0 
* 2: eth0                                                                                                     
                     Link File: /lib/systemd/network/99-default.link
                  Network File: /etc/systemd/network/00-bmc-eth0.network
                         State: routable (configured)
                  Online state: online                                                                        
                          Type: ether
                          Path: platform-1e670000.ftgmac
                        Driver: ftgmac100
              Hardware Address: 08:xx:xx:xx:00:14
                           MTU: 1500 (min: 68, max: 1500)
                         QDisc: fifo_fast
  IPv6 Address Generation Mode: eui64
      Number of Queues (Tx/Rx): 1/1
                       Address: 9.x.x.122
                                169.254.30.82
                                fe80::x:xxff:fe82:14
                       Gateway: 9.3.29.1

             Activation Policy: up
           Required For Online: yes

                  Connected To: system.gus.stglabs.ibm.com on port Gi3/10 (GigabitEthernet3/10)

 networkctl status eth1
* 3: eth1                                                                      
                     Link File: /lib/systemd/network/99-default.link
                  Network File: /etc/systemd/network/00-bmc-eth1.network
                         State: routable (configured)
                  Online state: online                                         
                          Type: ether
                          Path: platform-1e690000.ftgmac
                        Driver: ftgmac100
              Hardware Address: 08:xx:ef:x2:00:15
                           MTU: 1500 (min: 68, max: 1500)
                         QDisc: pfifo_fast
  IPv6 Address Generation Mode: eui64
      Number of Queues (Tx/Rx): 1/1
                       Address: 169.254.7.47
                                fe80::a94:efff:fe82:15
                       Gateway: fe80::b36a:f3b0:4933:f2fb
             Activation Policy: up
           Required For Online: yes
             DHCP6 Client IAID: 0x999999f
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab11f3xyzfexxfa0000

ip route
default via 9.3.29.1 dev eth0 
9.3.29.0/24 dev eth0 scope link  src 9.3.29.122 
169.254.0.0/16 dev eth0 scope link  src 169.254.30.82  metric 2048 
169.254.0.0/16 dev eth1 scope link  src 169.254.7.47  metric 2048

Eth1 interface ipv4 linklocal IP address is not reachable.

ping 169.254.7.47
PING 169.254.7.47 (169.254.7.47) 56(84) bytes of data.
From 169.254.13.200 icmp_seq=1 Destination Host Unreachable
From 169.254.13.200 icmp_seq=2 Destination Host Unreachable
From 169.254.13.200 icmp_seq=3 Destination Host Unreachable
From 169.254.13.200 icmp_seq=4 Destination Host Unreachable
From 169.254.13.200 icmp_seq=5 Destination Host Unreachable
From 169.254.13.200 icmp_seq=6 Destination Host Unreachable

yuwata commented 1 year ago

Please also provide the network status of 'helper machine', address, routes, and so on.
Is it networkd issue? Does the network setup work when you manually configure them?
Please try to monitor packets, and investigate in which level the issue occurs.

raviteja-b commented 1 year ago

I don't see any issue on helper machine configuration, helper machine has link local address & routes, will share these details if it helps in debug.

Here is my observation while investigating this issue, I see eth0 IPv4 link local ping works when eth0 route on top of eth1 for 169.254 subnet. if I change eth1 route to keep on top routing eth0 then I see eth1's link local ip starts pinging

169.254.0.0/16 dev eth0 scope link  src 169.254.30.82  metric 2048 
169.254.0.0/16 dev eth1 scope link  src 169.254.7.47  metric 2048

so only one IPv4 link local address reachable anytime based on route order. Systemd-networkd supports LinklocalAddressing at interface level while it does not work if there are multiple interfaces with LinkLocalAddressing enabled. my question here is when two interfaces registered with Systemd-networkd with LinklocalAddressing enabled, how to make sure both interfaces link local IPv4 addresses works?

ssahani commented 1 year ago

you need to configure https://www.freedesktop.org/software/systemd/man/systemd.network.html#%5BRoutingPolicyRule%5D%20Section%20Options. When configuring a Linux host with multiple interface, each with its own default gateway, ensuring route symmetric is a challenging for any given pair of endpoint. Symmetric routing means that traffic leaves the hosts from one interface (eth1) and come back from the same path (eth1). When both interfaces are in same subnet decision must be made as to which one to use. A condition that we should normally try to avoid is ‘asymmetric routing’, whereby traffic sent to a given IP address arrives on one interface, but traffic originating from that address leaves by a different interface (primary interface) which it can't.

something like this

[Match]
Name=eth1

[Network]
Address=192.168.60.70/24

[Route]
PreferredSource=192.168.60.70
Destination=192.168.60.0/24
Table=10

[Route]
Gateway=192.168.60.1
Table=10

[RoutingPolicyRule]
Table=10
To=192.168.60.70/24

[RoutingPolicyRule]
Table=10
From=192.168.60.70/24

raviteja-b commented 1 year ago

ok Thank you! @ssahani I will explore these options for static IP addresses configuration. When both interfaces are in same or different subnet, it expected to use the same interface which receives traffic should be able to send reply on same interface, we are expecting "Symmetric routing" behaviour as you have mentioned above. linklocal address are automatically assigned on the interface when LinkLocalAddressing Enabled, how does Systemd-networkd achieve these RoutingPolicyRule for link local IP addresses when multiple interfaces configured?

ssahani commented 1 year ago

I see your configurations and it's not static. Same as DHCP . Hence here you need a separate daemon to configure routing policy rules. Inside networkd we are missing this feature. After networkd configures some scripts needs to be called to configure routing policy rules

raviteja-b commented 1 year ago

I see your configurations and it's not static. Same as DHCP . Hence here you need a separate daemon to configure routing policy rules. Inside networkd we are missing this feature. For now we are using a third daemon to configure such scenarios.

what do you mean by separate daemon, you mean separate additional application code which populates these [RoutingPolicyRule] for link local addresses as well ? I believe networkd should support this feature, can we have this feature implemented in networkd?

ssahani commented 1 year ago

routing policy rules is same as ip rule. You can configure it manually after networkd configure finishes and test . you need to do

Create a separate routing table for each of the interfaces.
Add policy rules to direct outbound traffic to the appropriate routing table.

❯ echo 200 isp2 >> /etc/iproute2/rt_tables
❯ ip rule add from <interface_IP> table isp2 prio 1
❯ ip route add default via <gateway_IP> dev <interface> table isp2

raviteja-b commented 9 months ago

@yuwata @ssahani Expecting additional routing table configuration for each interface to have linklocal addresses does not look right, this will be additional work for applications which uses systemd-networkd configuration. application needs to identify what's the link local address assigned then configure routing table configuration for linklocal address, then again need to reload systemd-network and then interface may get new link local address which we are not sure, this is not working as expected while using programmatically I think all routing table configuration needs to be handled inside systemd-netwokd internally.

can you please consider implementing and support this feature in systemd-networkd

yuwata commented 9 months ago

Yeah, maybe networkd should automatically add routing policy rules or routes.

raviteja-b commented 9 months ago

Yeah, maybe networkd should automatically add routing policy rules or routes.

Yes, thank you for considering as feature request @yuwata this will be really helpful in supporting linklocal address on multiple ethernet interfaces. looking forward to see this feature implemented soon in systemd-networkd Please consider implementing this feature on priority basis

sunharis commented 9 months ago

Thank you @yuwata Once you discuss this for the feature release plan, please let us know by which version of the systemd we can get this feature supported?

yuwata commented 9 months ago

Hm... In such case, simplest solution I have is setting different network namespace for each interface, and running networkd in each namespace. Related to #14915. I have no other effective way to support such setup...

Possible super non-effective solution is setting route for each node found by ARP. But, the subnet for IPV4LL is too large, 169.254.0.0/16, hence that possibly introduces too many routes, and causes OOM.

raviteja-b commented 9 months ago

Hm... In such case, simplest solution I have is setting different network namespace for each interface, and running networkd in each namespace. Related to #14915. I have no other effective way to support such setup...

Possible super non-effective solution is setting route for each node found by ARP. But, the subnet for IPV4LL is too large, 169.254.0.0/16, hence that possibly introduces too many routes, and causes OOM.

This commit solves this Linklocal address on multiple interfaces issue right? what does this commit exactly does? https://github.com/yuwata/systemd/commit/f2ab96470c0b3175072c9cfcdbac26ae4a521675

yuwata commented 9 months ago

This commit solves this Linklocal address on multiple interfaces issue right? what does this commit exactly does? yuwata@f2ab964

No. I already dropped the commit from my working branch. The kernel add the exactly same route when an IPv4LL address is assigned to an interface. So, it does not change anything.

raviteja-b commented 3 months ago

Hi @yuwata any update on this feature request ? this is really important feature required to address several routing issues across multiple interfaces.

raviteja-b commented 2 months ago

Hi @yuwata any update on this feature request ? this really needs to be done for dhcp addresses and linklocal addresses from systemd-networkd side

raviteja-b commented 2 months ago

you need to configure https://www.freedesktop.org/software/systemd/man/systemd.network.html#%5BRoutingPolicyRule%5D%20Section%20Options. When configuring a Linux host with multiple interface, each with its own default gateway, ensuring route symmetric is a challenging for any given pair of endpoint. Symmetric routing means that traffic leaves the hosts from one interface (eth1) and come back from the same path (eth1). When both interfaces are in same subnet decision must be made as to which one to use. A condition that we should normally try to avoid is ‘asymmetric routing’, whereby traffic sent to a given IP address arrives on one interface, but traffic originating from that address leaves by a different interface (primary interface) which it can't.

something like this
[Match]
Name=eth1

[Network]
Address=192.168.60.70/24

[Route]
PreferredSource=192.168.60.70
Destination=192.168.60.0/24
Table=10

[Route]
Gateway=192.168.60.1
Table=10

[RoutingPolicyRule]
Table=10
To=192.168.60.70/24

[RoutingPolicyRule]
Table=10
From=192.168.60.70/24

This route policy configuration can be done in configuration files for static addresses, but for DHCP and linklocal address system-networkd should handle. @ssahani @yuwata can you please check this on priority since this is blocking when two ethernet interfaces have routes/gateways configured

systemd / systemd