search

sonic-net / SONiC

Landing page for Software for Open Networking in the Cloud (SONiC) - https://sonic-net.github.io/SONiC/
2.24k stars 1.13k forks source link

LLDP-exchanged MAC address information #185

Open erikpu opened 6 years ago

erikpu commented 6 years ago

Hi, I have two switches (Mellanox SN2100 and Arista 7050QX-32S) with SONiC connected to each other (topology here) and I am investigating LLDP behaviour. Mellanox's DEVICE_METADATA is configured as follows:

    "DEVICE_METADATA": {
        "localhost": {
            "hwsku": "ACS-MSN2100",
            "default_bgp_status": "up",
            "hostname": "sonic-mellanox",
            "platform": "x86_64-mlnx_msn2100-r0",
            "mac": "ec:0d:9a:8d:f1:c0",
            "default_pfcwd_status": "disable",
            "bgp_asn": "65100",
            "deployment_id": "None",
            "type": "LeafRouter"
        }
    },

Also, we can see that the MAC address of the connected interface (Ethernet32) is ec:0d:9a:8d:f1:c0:

admin@sonic-mellanox:~$ ip a show dev Ethernet32
15: Ethernet32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc pfifo_fast state UP group default qlen 1000
    link/ether ec:0d:9a:8d:f1:c0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ee0d:9aff:fe8d:f1c0/64 scope link
       valid_lft forever preferred_lft forever

Despite this, Arista has in its LLDP-entries from Mellanox the following Chassis MAC address: ec:0d:9a:8d:f1:e4.

admin@sonic-arista:~$ show lldp neighbors
Command: sudo lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    Ethernet16, via: LLDP, RID: 3, Time: 0 day, 00:22:44
  Chassis:
    ChassisID:    mac ec:0d:9a:8d:f1:e4
    SysName:      sonic-mellanox
    SysDescr:     Debian GNU/Linux 8 (jessie) Linux 3.16.0-5-amd64 #1 SMP Debian 3.16.51-3+deb8u1 (24
    TTL:          120
    MgmtIP:       10.1.0.32
    MgmtIP:       fe80::ee0d:9aff:fe8d:f1e4
    Capability:   Bridge, on
    Capability:   Router, on
    Capability:   Wlan, off
    Capability:   Station, off
  Port:
    PortID:       local Ethernet32
    PortDescr:    Ethernet32

This is the same MAC-address as Mellanox's eth0 uplink interface:

admin@sonic-mellanox:~$ ip a show dev eth0                                                           
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9100 qdisc mq state UP group default qlen 1000
    link/ether ec:0d:9a:8d:f1:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.250.3/24 brd 192.168.250.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ee0d:9aff:fe8d:f1e4/64 scope link
       valid_lft forever preferred_lft forever

I am wondering why LLDP uses this MAC address as Chassis MAC address of the Mellanox machine, and why not the configured MAC address in config_db.json and the MAC address of the connected Mellanox interface (Ethernet32). Can you shed some light on this? Or point me to some related code so I can dig into it? Thanks in advance!

SONiC version:

admin@sonic-mellanox:~$ show version
SONiC Software Version: SONiC.HEAD.574-ed915e3
Distribution: Debian 8.10
Kernel: 3.16.0-5-amd64
Build commit: ed915e3
Build date: Wed Apr  4 06:58:48 UTC 2018
Built by: johnar@jenkins-worker-3

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-orchagent-mlnx      HEAD.574-ed915e3    32203598a5bc        287 MB
docker-orchagent-mlnx      latest              32203598a5bc        287 MB
docker-syncd-mlnx          HEAD.574-ed915e3    b7ab299d2758        350.9 MB
docker-syncd-mlnx          latest              b7ab299d2758        350.9 MB
docker-lldp-sv2            HEAD.574-ed915e3    d8f65d12b406        297.2 MB
docker-lldp-sv2            latest              d8f65d12b406        297.2 MB
docker-dhcp-relay          HEAD.574-ed915e3    46c2a839b5ba        280.1 MB
docker-dhcp-relay          latest              46c2a839b5ba        280.1 MB
docker-database            HEAD.574-ed915e3    622f0f354847        278.8 MB
docker-database            latest              622f0f354847        278.8 MB
docker-teamd               HEAD.574-ed915e3    9dd25e367798        284.1 MB
docker-teamd               latest              9dd25e367798        284.1 MB
docker-snmp-sv2            HEAD.574-ed915e3    4b4277d6cc32        319.3 MB
docker-snmp-sv2            latest              4b4277d6cc32        319.3 MB
docker-router-advertiser   HEAD.574-ed915e3    6c2adf7743ec        276.4 MB
docker-router-advertiser   latest              6c2adf7743ec        276.4 MB
docker-platform-monitor    HEAD.574-ed915e3    9b31194ff812        298.3 MB
docker-platform-monitor    latest              9b31194ff812        298.3 MB
docker-fpm-quagga          HEAD.574-ed915e3    67c77efc3d4a        290.6 MB
docker-fpm-quagga          latest              67c77efc3d4a        290.6 MB

and

admin@sonic-arista:~$ show version
SONiC Software Version: SONiC.HEAD.547-4754b43
Distribution: Debian 8.10
Kernel: 3.16.0-5-amd64
Build commit: 4754b43
Build date: Fri Apr  6 07:38:17 UTC 2018
Built by: johnar@jenkins-worker-4

Docker images:
REPOSITORY                 TAG                 IMAGE ID            SIZE
docker-syncd-brcm          HEAD.547-4754b43    145a93bf2613        358.1 MB
docker-syncd-brcm          latest              145a93bf2613        358.1 MB
docker-orchagent-brcm      HEAD.547-4754b43    32fdab0a0a85        287 MB
docker-orchagent-brcm      latest              32fdab0a0a85        287 MB
docker-lldp-sv2            HEAD.547-4754b43    9ce7dc0f55f6        297.2 MB
docker-lldp-sv2            latest              9ce7dc0f55f6        297.2 MB
docker-dhcp-relay          HEAD.547-4754b43    17fd00cd2091        280.1 MB
docker-dhcp-relay          latest              17fd00cd2091        280.1 MB
docker-database            HEAD.547-4754b43    5af52a038baf        278.8 MB
docker-database            latest              5af52a038baf        278.8 MB
docker-teamd               HEAD.547-4754b43    24d8a02873a7        284.1 MB
docker-teamd               latest              24d8a02873a7        284.1 MB
docker-snmp-sv2            HEAD.547-4754b43    129e2b96b2c4        319.3 MB
docker-snmp-sv2            latest              129e2b96b2c4        319.3 MB
docker-router-advertiser   HEAD.547-4754b43    58be616fcdb1        276.4 MB
docker-router-advertiser   latest              58be616fcdb1        276.4 MB
docker-platform-monitor    HEAD.547-4754b43    6b68d76ae87b        298.3 MB
docker-platform-monitor    latest              6b68d76ae87b        298.3 MB
docker-fpm-quagga          HEAD.547-4754b43    0ff216021fea        290.6 MB
docker-fpm-quagga          latest              0ff216021fea        290.6 MB
volodymyrsamotiy commented 6 years ago

On some platforms (including Mellanox) system MAC and management interface MAC are different (there is some alignment of last byte). You can find the details about getting system MAC here: https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-config-engine/sonic_platform.py#L43

By default "lldpd" will take the first MAC address from all the considered interfaces to compute the chassis ID. So, it looks like in our case first interface is "eth0" and as a result incorrect MAC is taken by "lldpd" (only on platforms with different system and management MAC).

To solve this problem "lddpd" should be configured with the correct system MAC, not sure what is the right way, but probably it is possible to use something like shown below: http://vincentbernat.github.io/lldpd/usage.html

...
       configure system chassisid description

             Override chassis ID with the provided value instead of using
             MAC address from one interface or host name.
...

In SONiC lldpd configuration can be done in "lldp" docker container, using "lldpcli" utility or by editing "/etc/lldpd.conf" config file (it is passed to "lldpd" in "usr/bin/start.sh" script): (all LDDP config files are here https://github.com/Azure/sonic-buildimage/tree/master/dockers/docker-lldp-sv2)

sonic # docker exec -it lldp lldpcli show chassis summary
-------------------------------------------------------------------------------
Local chassis:
-------------------------------------------------------------------------------
Chassis:
  ChassisID:    mac <macaddr>
  SysName:      <sysname>
-------------------------------------------------------------------------------
sonic #
sonic # docker exec -it lldp cat /etc/lldpd.conf
configure ports eth0 lldp portidsubtype local eth0
erikpu commented 6 years ago

Thank you!

liatgrozovik commented 6 years ago

Although there is a WA i believe the issue should be kept as open and hopefully will be handled

pavel-shirshov commented 6 years ago

The lldpd used in SONiC doesn't have a way to set a management MAC address. Latest lldpd has such option

Currently SONiC runs lldpd as /usr/sbin/lldpd -d -I Ethernet,eth -C eth0 https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-lldp-sv2/supervisord.conf The -C option tells lldpd to scan eth0 only as a source of management mac addresses.