search

sonic-net / SONiC

Landing page for Software for Open Networking in the Cloud (SONiC) - https://sonic-net.github.io/SONiC/
2.23k stars 1.12k forks source link

When Vlan packet is egress, Vlan tagged port of SONiC may add 4 bytes garbage characters to the first 4 bytes of the trailing padding. #517

Open robert1030 opened 4 years ago

robert1030 commented 4 years ago

Description

  1. When Vlan packet is egress, Vlan tagged port of SONiC may add 4 bytes non-zero value to the first 4 bytes of the trailing padding.
  2. Payload should be put "\x00"
  3. Fault Rate about: high
  4. Test Server send Tag packet to vlan tagged port of SONiC (append payload is \x00 * 14), Access Port 1 untagged packet and forward to client, client reply untagged packet to Access port 1, SONiC Switch add vlan tagged and payload (padding maybe is non-zero value) to packet, then it egress the packet to Test Server.

Step

  1. create vlan.json files to /etc/sonic/
  2. Disable copp protection on FanOutLeaf Switch (Wedge100bf_65x)
  3. To check copp protection is disable, docker exec -it swss more /etc/swss/config.d/00-copp.config.json []
  4. sonic-cfggen -H -k mavericks -j /etc/sonic/init_cfg.json -j /etc/sonic/vlan.json --print-data > /etc/sonic/config_db.json
  5. config reload -y
  6. Waiting about 3 mins
  7. Stop and Disalbe some service systemctl stop lldp.service systemctl disable lldp.service systemctl stop dhcp_relay.service systemctl disable dhcp_relay.service systemctl stop bgp.service systemctl disable bgp.service systemctl stop teamd.service systemctl disable teamd.service
  8. To check L2 Vlan be deploy to DUT show vlan brief +-----------+--------------+-------------+----------------+-----------------------+ | VLAN ID | IP Address | Ports | Port Tagging | DHCP Helper Address | +===========+==============+=============+================+=======================+ | 1681 | | Ethernet0 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1682 | | Ethernet4 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1683 | | Ethernet8 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1684 | | Ethernet12 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1685 | | Ethernet16 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1686 | | Ethernet20 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1687 | | Ethernet24 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1688 | | Ethernet28 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1689 | | Ethernet32 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1690 | | Ethernet36 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1691 | | Ethernet40 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1692 | | Ethernet44 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1693 | | Ethernet48 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1694 | | Ethernet52 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1695 | | Ethernet56 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1696 | | Ethernet60 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1697 | | Ethernet64 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1698 | | Ethernet68 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1699 | | Ethernet72 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1700 | | Ethernet76 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1701 | | Ethernet80 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1702 | | Ethernet84 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1703 | | Ethernet88 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1704 | | Ethernet92 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1705 | | Ethernet96 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1706 | | Ethernet100 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1707 | | Ethernet104 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1708 | | Ethernet108 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1709 | | Ethernet112 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1710 | | Ethernet116 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1711 | | Ethernet120 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+ | 1712 | | Ethernet124 | untagged | | | | | Ethernet256 | tagged | | +-----------+--------------+-------------+----------------+-----------------------+

show vlan config Name VID Member Mode

Vlan1681 1681 Ethernet0 untagged Vlan1681 1681 Ethernet256 tagged Vlan1682 1682 Ethernet4 untagged Vlan1682 1682 Ethernet256 tagged Vlan1683 1683 Ethernet8 untagged Vlan1683 1683 Ethernet256 tagged Vlan1684 1684 Ethernet12 untagged Vlan1684 1684 Ethernet256 tagged Vlan1685 1685 Ethernet16 untagged Vlan1685 1685 Ethernet256 tagged Vlan1686 1686 Ethernet20 untagged Vlan1686 1686 Ethernet256 tagged Vlan1687 1687 Ethernet24 untagged Vlan1687 1687 Ethernet256 tagged Vlan1688 1688 Ethernet28 untagged Vlan1688 1688 Ethernet256 tagged Vlan1689 1689 Ethernet32 untagged Vlan1689 1689 Ethernet256 tagged Vlan1690 1690 Ethernet36 untagged Vlan1690 1690 Ethernet256 tagged Vlan1691 1691 Ethernet40 untagged Vlan1691 1691 Ethernet256 tagged Vlan1692 1692 Ethernet44 untagged Vlan1692 1692 Ethernet256 tagged Vlan1693 1693 Ethernet48 untagged Vlan1693 1693 Ethernet256 tagged Vlan1694 1694 Ethernet52 untagged Vlan1694 1694 Ethernet256 tagged Vlan1695 1695 Ethernet56 untagged Vlan1695 1695 Ethernet256 tagged Vlan1696 1696 Ethernet60 untagged Vlan1696 1696 Ethernet256 tagged Vlan1697 1697 Ethernet64 untagged Vlan1697 1697 Ethernet256 tagged Vlan1698 1698 Ethernet68 untagged Vlan1698 1698 Ethernet256 tagged Vlan1699 1699 Ethernet72 untagged Vlan1699 1699 Ethernet256 tagged Vlan1700 1700 Ethernet76 untagged Vlan1700 1700 Ethernet256 tagged Vlan1701 1701 Ethernet80 untagged Vlan1701 1701 Ethernet256 tagged Vlan1702 1702 Ethernet84 untagged Vlan1702 1702 Ethernet256 tagged Vlan1703 1703 Ethernet88 untagged Vlan1703 1703 Ethernet256 tagged Vlan1704 1704 Ethernet92 untagged Vlan1704 1704 Ethernet256 tagged Vlan1705 1705 Ethernet96 untagged Vlan1705 1705 Ethernet256 tagged Vlan1706 1706 Ethernet100 untagged Vlan1706 1706 Ethernet256 tagged Vlan1707 1707 Ethernet104 untagged Vlan1707 1707 Ethernet256 tagged Vlan1708 1708 Ethernet108 untagged Vlan1708 1708 Ethernet256 tagged Vlan1709 1709 Ethernet112 untagged Vlan1709 1709 Ethernet256 tagged Vlan1710 1710 Ethernet116 untagged Vlan1710 1710 Ethernet256 tagged Vlan1711 1711 Ethernet120 untagged Vlan1711 1711 Ethernet256 tagged Vlan1712 1712 Ethernet124 untagged Vlan1712 1712 Ethernet256 tagged

  1. Test Server send vlan tagged 1681's arp packet to Ethernet 256 (payload is \x00 *14)
  2. PC client of Ethernet0 got untagged arp packet and it reply arp-reply
  3. observed egress packet of SONiC Switch Ethernet 256(vlan tagged port) be randomly add garbage characters to the first 4 bytes of the 14 bytes payload

Describe the results you expected: Payload filed: "load : StrField = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')",

Describe the results you received: Payload Failed filed:

  1. "load : StrField = '\xc5\x86\xd4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')",
  2. "load : StrField = '\xb1\xbe\xd4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')",
  3. "load : StrField = '\xbd\xf6\xd4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')",

show version SONiC Software Version: SONiC.HEAD.450-dirty-20191018.061922 Distribution: Debian 9.11 Kernel: 4.9.0-9-2-amd64 Build commit: 60a7f4d6 Build date: Fri Oct 18 08:32:29 UTC 2019 Built by: johnar@jenkins-worker-4

l2-32vlan configuration file - vlan.json: https://drive.google.com/open?id=1vkX_vJJPZlutmlb7BmmK-59prjegAnAQ

Wireshark capture Tagged ports https://drive.google.com/open?id=1jTc2sSJcKc5lTACeFl2qdPCowI-9FQA9

Wedge100bf_65x failed syslog https://drive.google.com/open?id=1no8PeBXBB5iIBaBjpeRpzR_U_MG_kl0ohRBsfiJP0x0

Other dump logging information

"AssertionError: Expected packet was not received on device 0, port 0.", "========== EXPECTED ==========", "dst : DestMACField = '00:06:07:08:09:0a' (None)", "src : SourceMACField = '6c:b9:c5:15:be:55' (None)", "type : XShortEnumField = 2054 (0)", "--", "hwtype : XShortField = 1 (1)", "ptype : XShortEnumField = 2048 (2048)", "hwlen : ByteField = 6 (6)", "plen : ByteField = 4 (4)", "op : ShortEnumField = 2 (1)", "hwsrc : ARPSourceMACField = '6c:b9:c5:15:be:55' (None)", "psrc : SourceIPField = '10.10.1.2' (None)", "hwdst : MACField = '00:06:07:08:09:0a' ('00:00:00:00:00:00')", "pdst : IPField = '10.10.1.3' ('0.0.0.0')", "--", "load : StrField = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')", "--", "0000 00 06 07 08 09 0A 6C B9 C5 15 BE 55 08 06 00 01 ......l....U....", "0010 08 00 06 04 00 02 6C B9 C5 15 BE 55 0A 0A 01 02 ......l....U....", "0020 00 06 07 08 09 0A 0A 0A 01 03 00 00 00 00 00 00 ................", "0030 00 00 00 00 00 00 00 00 00 00 00 00 ............", "========== RECEIVED ==========", "1 total packets. Displaying most recent 1 packets:", "------------------------------", "dst : DestMACField = '00:06:07:08:09:0a' (None)", "src : SourceMACField = '6c:b9:c5:15:be:55' (None)", "type : XShortEnumField = 2054 (0)", "--", "hwtype : XShortField = 1 (1)", "ptype : XShortEnumField = 2048 (2048)", "hwlen : ByteField = 6 (6)", "plen : ByteField = 4 (4)", "op : ShortEnumField = 2 (1)", "hwsrc : ARPSourceMACField = '6c:b9:c5:15:be:55' (None)", "psrc : SourceIPField = '10.10.1.2' (None)", "hwdst : MACField = '00:06:07:08:09:0a' ('00:00:00:00:00:00')", "pdst : IPField = '10.10.1.3' ('0.0.0.0')", "--", "load : StrField = '\xc5\x86\xd4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')", "--", "0000 00 06 07 08 09 0A 6C B9 C5 15 BE 55 08 06 00 01 ......l....U....", "0010 08 00 06 04 00 02 6C B9 C5 15 BE 55 0A 0A 01 02 ......l....U....", "0020 00 06 07 08 09 0A 0A 0A 01 03 C5 86 D4 61 00 00 .............a..", "0030 00 00 00 00 00 00 00 00 00 00 00 00 ............", "==============================",

"AssertionError: Expected packet was not received on device 0, port 0.", "========== EXPECTED ==========", "dst : DestMACField = '00:06:07:08:09:00' (None)", "src : SourceMACField = '6c:b9:c5:15:be:55' (None)", "type : XShortEnumField = 2054 (0)", "--", "hwtype : XShortField = 1 (1)", "ptype : XShortEnumField = 2048 (2048)", "hwlen : ByteField = 6 (6)", "plen : ByteField = 4 (4)", "op : ShortEnumField = 2 (1)", "hwsrc : ARPSourceMACField = '6c:b9:c5:15:be:55' (None)", "psrc : SourceIPField = '10.10.1.2' (None)", "hwdst : MACField = '00:06:07:08:09:00' ('00:00:00:00:00:00')", "pdst : IPField = '10.10.1.3' ('0.0.0.0')", "--", "load : StrField = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')", "--", "0000 00 06 07 08 09 00 6C B9 C5 15 BE 55 08 06 00 01 ......l....U....", "0010 08 00 06 04 00 02 6C B9 C5 15 BE 55 0A 0A 01 02 ......l....U....", "0020 00 06 07 08 09 00 0A 0A 01 03 00 00 00 00 00 00 ................", "0030 00 00 00 00 00 00 00 00 00 00 00 00 ............", "========== RECEIVED ==========", "1 total packets. Displaying most recent 1 packets:", "------------------------------", "dst : DestMACField = '00:06:07:08:09:00' (None)", "src : SourceMACField = '6c:b9:c5:15:be:55' (None)", "type : XShortEnumField = 2054 (0)", "--", "hwtype : XShortField = 1 (1)", "ptype : XShortEnumField = 2048 (2048)", "hwlen : ByteField = 6 (6)", "plen : ByteField = 4 (4)", "op : ShortEnumField = 2 (1)", "hwsrc : ARPSourceMACField = '6c:b9:c5:15:be:55' (None)", "psrc : SourceIPField = '10.10.1.2' (None)", "hwdst : MACField = '00:06:07:08:09:00' ('00:00:00:00:00:00')", "pdst : IPField = '10.10.1.3' ('0.0.0.0')", "--", "load : StrField = '\xb1\xbe\xd4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')", "--", "0000 00 06 07 08 09 00 6C B9 C5 15 BE 55 08 06 00 01 ......l....U....", "0010 08 00 06 04 00 02 6C B9 C5 15 BE 55 0A 0A 01 02 ......l....U....", "0020 00 06 07 08 09 00 0A 0A 01 03 B1 BE D4 61 00 00 .............a..", "0030 00 00 00 00 00 00 00 00 00 00 00 00 ............", "=============================="

"AssertionError: Expected packet was not received on device 0, port 0.", "========== EXPECTED ==========", "dst : DestMACField = '00:06:07:08:09:00' (None)", "src : SourceMACField = '6c:b9:c5:15:be:55' (None)", "type : XShortEnumField = 2054 (0)", "--", "hwtype : XShortField = 1 (1)", "ptype : XShortEnumField = 2048 (2048)", "hwlen : ByteField = 6 (6)", "plen : ByteField = 4 (4)", "op : ShortEnumField = 2 (1)", "hwsrc : ARPSourceMACField = '6c:b9:c5:15:be:55' (None)", "psrc : SourceIPField = '10.10.1.2' (None)", "hwdst : MACField = '00:06:07:08:09:00' ('00:00:00:00:00:00')", "pdst : IPField = '10.10.1.3' ('0.0.0.0')", "--", "load : StrField = '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')", "--", "0000 00 06 07 08 09 00 6C B9 C5 15 BE 55 08 06 00 01 ......l....U....", "0010 08 00 06 04 00 02 6C B9 C5 15 BE 55 0A 0A 01 02 ......l....U....", "0020 00 06 07 08 09 00 0A 0A 01 03 00 00 00 00 00 00 ................", "0030 00 00 00 00 00 00 00 00 00 00 00 00 ............", "========== RECEIVED ==========", "1 total packets. Displaying most recent 1 packets:", "------------------------------", "dst : DestMACField = '00:06:07:08:09:00' (None)", "src : SourceMACField = '6c:b9:c5:15:be:55' (None)", "type : XShortEnumField = 2054 (0)", "--", "hwtype : XShortField = 1 (1)", "ptype : XShortEnumField = 2048 (2048)", "hwlen : ByteField = 6 (6)", "plen : ByteField = 4 (4)", "op : ShortEnumField = 2 (1)", "hwsrc : ARPSourceMACField = '6c:b9:c5:15:be:55' (None)", "psrc : SourceIPField = '10.10.1.2' (None)", "hwdst : MACField = '00:06:07:08:09:00' ('00:00:00:00:00:00')", "pdst : IPField = '10.10.1.3' ('0.0.0.0')", "--", "load : StrField = '\xbd\xf6\xd4a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' ('')", "--", "0000 00 06 07 08 09 00 6C B9 C5 15 BE 55 08 06 00 01 ......l....U....", "0010 08 00 06 04 00 02 6C B9 C5 15 BE 55 0A 0A 01 02 ......l....U....", "0020 00 06 07 08 09 00 0A 0A 01 03 BD F6 D4 61 00 00 .............a..", "0030 00 00 00 00 00 00 00 00 00 00 00 00 ............", "==============================",

robert1030 commented 4 years ago

After change profile to msdc_ipv4_profile, the issue still be reproduced.

New syslog [https://drive.google.com/open?id=1XMSLndgPx1p-RmEB3l-wOSC6IihqmJQwav05r7ipHp8] ()

robert1030 commented 4 years ago

Hi akokhan:

After change profile to msdc_ipv4_profile, the issue still be reproduced.

New syslog

Profile link check is below: drwxr-xr-x 1 root root 4096 Nov 13 02:38 . drwxr-xr-x 1 root root 4096 Nov 10 05:34 .. lrwxrwxrwx 1 root root 25 Nov 13 02:38 install -> install_msdc_ipv4_profile

tcpdump capture arp-vlan tagged packets

This issue seems similar to what happened to the Cisco N9K. Troubleshoot the Corrupted Ethernet Packet on Cisco Nexus 9000