search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
741 stars 1.43k forks source link

[static route] No arp entries for nexthops in the static route after config reload #7968

Closed lolyu closed 3 years ago

lolyu commented 3 years ago

Description

Steps to reproduce the issue:

  1. increase the IO packet count from 1 to 10000
  2. run test_static_route.py::test_static_route_ecmp

Describe the results you received:

  1. after config reload, verify the static route existence 
    root@str2-7050cx3-acs-02:~# redis-cli -n 4 keys STATIC*
1) "STATIC_ROUTE|2.2.2.0/24"
root@str2-7050cx3-acs-02:~# ip route show match 2.2.2.2
default via 10.3.146.1 dev eth0 proto 196 metric 20
2.2.2.0/24 proto 196 metric 20
    nexthop via 192.168.0.21 dev Vlan1000 weight 1
    nexthop via 192.168.0.22 dev Vlan1000 weight 1
    nexthop via 192.168.0.25 dev Vlan1000 weight 1
  2. verify the ARP entries, no arp entries for those nexthops(192.168.0.21, 192.168.0.22, 192.168.0.25) 
    root@str2-7050cx3-acs-02:~# show arp
Address     MacAddress         Iface            Vlan
----------  -----------------  ---------------  ------
10.0.0.57   52:54:00:78:cd:47  PortChannel0001  -
10.0.0.59   52:54:00:44:83:48  PortChannel0002  -
10.0.0.61   52:54:00:8e:3f:97  PortChannel0003  -
10.0.0.63   52:54:00:82:d3:cd  PortChannel0004  -
10.3.146.1  00:00:5e:00:01:64  eth0             -
Total number of entries 5
  3. the IO test failed because the packets sent to 2.2.2.1 are dropped. 
    root@str2-7050cx3-acs-02:~# portstat
Last cached time was 2021-06-24 10:24:18.835688
  IFACE    STATE    RX_OK       RX_BPS    RX_UTIL    RX_ERR    RX_DRP    RX_OVR    TX_OK      TX_BPS    TX_UTIL    TX_ERR    TX_DRP    TX_OVR
-----------  -------  -------  -----------  ---------  --------  --------  --------  -------  ----------  ---------  --------  --------  --------
Ethernet0        X        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet4        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet8        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet12        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet16        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet20        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet24        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet28        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet32        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet36        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet40        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet44        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet48        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet52        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet56        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet60        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet64        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet68        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet72        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet76        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet80        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet84        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet88        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet92        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet96        U        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet100        X        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet104        X        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet108        X        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet112        U   10,008  201.01 KB/s      0.00%         0    10,000         0        8  138.29 B/s      0.00%         0         0         0
Ethernet116        U        8   138.29 B/s      0.00%         0         0         0        6  103.52 B/s      0.00%         0         0         0
Ethernet120        U        8   138.29 B/s      0.00%         0         0         0        6  111.25 B/s      0.00%         0         0         0
Ethernet124        U        8   138.29 B/s      0.00%         0         0         0        8  138.29 B/s      0.00%         0         0         0
Ethernet128        X        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
Ethernet132        X        0     0.00 B/s      0.00%         0         0         0        0    0.00 B/s      0.00%         0         0         0
    
>       raise self.failureException(msg)
E       AssertionError: Did not receive expected packet on any of ports [24, 20, 21] for device 0.
E       ========== EXPECTED ==========
E       Mask:
E       0000   94 8E D3 04 BF D8 98 03  9B 03 22 00 08 00 45 00   .........."...E.
E       0010   00 56 00 01 00 00 40 06  74 9D 01 01 01 01 02 02   .V....@.t.......
E       0020   02 01 04 D2 10 E1 00 00  00 00 00 00 00 00 50 02   ..............P.
E       0030   20 00 82 ED 00 00 74 65  73 74 5F 73 74 61 74 69    .....test_stati
E       0040   63 5F 72 6F 75 74 65 20  74 65 73 74 5F 73 74 61   c_route test_sta
E       0050   74 69 63 5F 72 6F 75 74  65 20 74 65 73 74 5F 73   tic_route test_s
E       0060   74 61 74 69                                        tati
E       mask = 00 00 00 00 00 00 00 00  00 00 00 00 ff ff ff ff
E       0010   ff ff ff ff ff ff 00 ff  00 00 ff ff ff ff ff ff
E       0020   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0050   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0060   ff ff ff ff
E
E       ========== RECEIVED ==========
E       6744 total packets. Displaying most recent 3 packets:
E       ------------------------------
E       0000   52 54 00 82 D3 CD 94 8E  D3 04 BF D8 86 DD 6C 0C   RT............l.
E       0010   82 33 00 20 06 01 FC 00  00 00 00 00 00 00 00 00   .3. ............
E       0020   00 00 00 00 00 7D FC 00  00 00 00 00 00 00 00 00   .....}..........
E       0030   00 00 00 00 00 7E 00 B3  D5 DD D0 48 BF 30 42 3F   .....~.....H.0B?
E       0040   58 FF 80 10 06 6F 51 F9  00 00 01 01 08 0A 0E 0C   X....oQ.........
E       0050   F8 F8 12 AE 0A 5D                                  .....]
E       ------------------------------
E       0000   52 54 00 8E 3F 97 94 8E  D3 04 BF D8 08 00 45 C0   RT..?.........E.
E       0010   00 47 A5 E4 40 00 01 06  BE 94 0A 00 00 3C 0A 00   .G..@........<..
E       0020   00 3D 00 B3 D5 7C 53 49  53 84 45 99 3A AB 80 18   .=...|SIS.E.:...
E       0030   01 BA A1 19 00 00 01 01  08 0A CA 4C 69 89 12 AD   ...........Li...
E       0040   77 7E FF FF FF FF FF FF  FF FF FF FF FF FF FF FF   w~..............
E       0050   FF FF 00 13 04                                     .....
E       ------------------------------
E       0000   52 54 00 8E 3F 97 94 8E  D3 04 BF D8 08 00 45 C0   RT..?.........E.
E       0010   00 34 A5 E5 40 00 01 06  BE A6 0A 00 00 3C 0A 00   .4..@........<..
E       0020   00 3D 00 B3 D5 7C 53 49  53 97 45 99 3A BE 80 10   .=...|SIS.E.:...
E       0030   01 BA A2 2E 00 00 01 01  08 0A CA 4C 69 8F 12 AD   ...........Li...
E       0040   7A 6B                                              zk
E       ==============================

/usr/lib/python2.7/unittest/case.py:410: AssertionError ===================================================================================== 1 failed in 410.43 seconds =====================================================================================

#### Describe the results you expected:
* ARP entries should be restored for those nexthops listed in the added static route.

#### Output of `show version`:

root@str2-7050cx3-acs-02:~# show version

SONiC Software Version: SONiC.master.21023-f8eb5b095 Distribution: Debian 10.10 Kernel: 4.19.0-12-2-amd64 Build commit: f8eb5b095 Build date: Wed Jun 23 13:24:19 UTC 2021 Built by: AzDevOps@sonic-build-workers-000ETK

Platform: x86_64-arista_7050cx3_32s HwSKU: Arista-7050CX3-32S-C32 ASIC: broadcom ASIC Count: 1 Serial Number: JPE20424995 Model Number: DCS-7050CX3-32S-SSD Hardware Revision: N/A Uptime: 10:26:15 up 50 min, 2 users, load average: 1.98, 3.14, 3.10

Docker images: REPOSITORY TAG IMAGE ID SIZE docker-platform-monitor latest 50dba5a7f17e 627MB docker-platform-monitor master.21023-f8eb5b095 50dba5a7f17e 627MB docker-snmp latest 6f83a92b6a13 454MB docker-snmp master.21023-f8eb5b095 6f83a92b6a13 454MB docker-macsec latest 1b16d5d81457 427MB docker-macsec master.21023-f8eb5b095 1b16d5d81457 427MB docker-orchagent latest 4c686f33e270 442MB docker-orchagent master.21023-f8eb5b095 4c686f33e270 442MB docker-fpm-frr latest 6ca4795ce810 442MB docker-fpm-frr master.21023-f8eb5b095 6ca4795ce810 442MB docker-sonic-telemetry latest d6a34ee25aeb 501MB docker-sonic-telemetry master.21023-f8eb5b095 d6a34ee25aeb 501MB docker-sflow latest e3245977ec05 425MB docker-sflow master.21023-f8eb5b095 e3245977ec05 425MB docker-syncd-brcm latest c04f5f5eb087 705MB docker-syncd-brcm master.21023-f8eb5b095 c04f5f5eb087 705MB docker-dhcp-relay latest 1a37e33d2f3b 420MB docker-dhcp-relay master.21023-f8eb5b095 1a37e33d2f3b 420MB docker-sonic-mgmt-framework latest a2ef9bfb5599 570MB docker-sonic-mgmt-framework master.21023-f8eb5b095 a2ef9bfb5599 570MB docker-teamd latest 75acb5c42dde 424MB docker-teamd master.21023-f8eb5b095 75acb5c42dde 424MB docker-nat latest 647bc062aeee 427MB docker-nat master.21023-f8eb5b095 647bc062aeee 427MB docker-router-advertiser latest ba7987ff640d 413MB docker-router-advertiser master.21023-f8eb5b095 ba7987ff640d 413MB docker-lldp latest 4ea83826a506 453MB docker-lldp master.21023-f8eb5b095 4ea83826a506 453MB docker-database latest c685468049a8 413MB docker-database master.21023-f8eb5b095 c685468049a8 413MB

lolyu commented 3 years ago

@prsunny, could you please check this issue? Thanks!

bingwang-ms commented 3 years ago

The issue is reopened since same issue is detected again on SONiC.20201231.11 on Mellanox-SN4600C. Static routes are configured, but neighbors are not resolved after config reload even after 10 minutes.

admin@str-msn4600c-acs-02:~$ redis-cli -n 4 hgetall "STATIC_ROUTE|2.2.2.0/24"
1) "nexthop"
2) "192.168.0.28,192.168.0.36,192.168.0.10"
admin@str-msn4600c-acs-02:~$ show arp
Address       MacAddress         Iface            Vlan
------------  -----------------  ---------------  ------
10.0.0.1      52:54:00:82:1d:7a  PortChannel0001  -
10.0.0.5      52:54:00:c3:fe:06  PortChannel0002  -
10.0.0.9      52:54:00:e6:e9:37  PortChannel0003  -
10.0.0.13     52:54:00:4b:b9:56  PortChannel0004  -
10.3.146.1    00:00:5e:00:01:64  eth0             -
10.3.146.148  74:83:ef:d3:67:b8  eth0             -
10.3.146.149  74:83:ef:bb:25:e0  eth0             -
10.3.146.151  98:5d:82:14:f7:40  eth0             -
Total number of entries 8 
admin@str-msn4600c-acs-02:~$
prsunny commented 3 years ago

@shi-su , is it due to arp_responder?