search

sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
195 stars 708 forks source link

[multi-asic] "show ip interfaces" displays N/A for neighbor name and ip on asic1 #9051

Open ysmanman opened 1 year ago

ysmanman commented 1 year ago

Description

We noticed the issue in t2 testing with latest 202205 image. This was seen on SKU with two asics: 

admin@cmp227-4:~$ show platform summary
Platform: x86_64-arista_7800r3ak_36dm2_lc
HwSKU: Arista-7800R3AK-36DM2-C36
ASIC: broadcom
ASIC Count: 2
Serial Number: SSN21060156
Model Number: 7800R3AK-36DM2-LC
Hardware Revision: 29.00
Switch Type: voq

This issue will cause tests like qos to fail because of the following error: 

Exception('Invalid IPv4 address N/A',)
Traceback (most recent call last):
  File "/data/tests/common/plugins/log_section_start/__init__.py", line 84, in _fixture_generator_decorator
    res = next(it)
  File "/data/tests/qos/qos_sai_base.py", line 844, in dutConfig
    active_ips = dst_asic.get_active_ip_interfaces(tbinfo)
  File "/data/tests/common/devices/sonic_asic.py", line 293, in get_active_ip_interfaces
    ip_ifs, tbinfo, self.namespace
  File "/data/tests/common/devices/sonic.py", line 2068, in active_ip_interfaces
    self.ping_v4(v["peer_ipv4"], count=3, ns_arg=ns_arg)):
  File "/data/tests/common/devices/sonic.py", line 2026, in ping_v4
    raise Exception("Invalid IPv4 address {}".format(ipv4))
Exception: Invalid IPv4 address N/A

The issue was only seen on asic1, but not on asic0:

admin@cmp227-4:~$ show ip interfaces  -n asic0
Interface        Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
---------------  --------  -------------------  ------------  --------------  -------------
Ethernet96                 10.0.0.32/31         up/up         ARISTA13T3      10.0.0.33
Ethernet120                10.0.0.34/31         up/up         ARISTA17T3      10.0.0.35
Loopback0                  10.1.0.1/32          up/up         N/A             N/A
PortChannel102             10.0.0.0/31          up/up         ARISTA01T3      10.0.0.1                                  
PortChannel104             10.0.0.4/31          up/up         ARISTA03T3      10.0.0.5
PortChannel106             10.0.0.8/31          up/up         ARISTA05T3      10.0.0.9
PortChannel108             10.0.0.12/31         up/up         ARISTA07T3      10.0.0.13
PortChannel1010            10.0.0.16/31         up/up         ARISTA09T3      10.0.0.17
PortChannel1012            10.0.0.20/31         up/up         ARISTA11T3      10.0.0.21
PortChannel1016            10.0.0.28/31         up/up         ARISTA15T3      10.0.0.29
PortChannel1020            10.0.0.24/31         up/up         ARISTA18T3      10.0.0.25
docker0                    240.127.1.1/24       up/up         N/A             N/A
eth0                       10.240.86.56/25      up/up         N/A             N/A
eth1                       127.100.4.10/16      up/up         N/A             N/A
eth1-midplane              127.100.4.1/16       up/up         N/A             N/A
lo                         127.0.0.1/16         up/up         N/A             N/A
admin@cmp227-4:~$ show ip interfaces  -n asic1
Interface      Master    IPv4 address/mask    Admin/Oper    BGP Neighbor    Neighbor IP
-------------  --------  -------------------  ------------  --------------  -------------
Ethernet144              10.0.0.36/31         up/up         N/A             N/A
Ethernet152              10.0.0.38/31         up/up         N/A             N/A
Ethernet160              10.0.0.40/31         up/up         N/A             N/A
Ethernet168              10.0.0.42/31         up/up         N/A             N/A
Ethernet176              10.0.0.44/31         up/up         N/A             N/A
Ethernet184              10.0.0.46/31         up/up         N/A             N/A
Ethernet192              10.0.0.48/31         up/up         N/A             N/A
Ethernet200              10.0.0.50/31         up/up         N/A             N/A
Ethernet208              10.0.0.52/31         up/up         N/A             N/A
Ethernet216              10.0.0.54/31         up/up         N/A             N/A
Ethernet224              10.0.0.56/31         up/up         N/A             N/A
Ethernet232              10.0.0.58/31         up/up         N/A             N/A
Ethernet240              10.0.0.60/31         up/up         N/A             N/A
Ethernet248              10.0.0.62/31         up/up         N/A             N/A
Loopback0                10.1.0.1/32          up/up         N/A             N/A
docker0                  240.127.1.1/24       up/up         N/A             N/A
eth0                     10.240.86.56/25      up/up         N/A             N/A
eth1                     127.100.4.11/16      up/up         N/A             N/A
eth1-midplane            127.100.4.1/16       up/up         N/A             N/A
lo                       127.0.0.1/16         up/up         N/A             N/A
admin@cmp227-4:~$

Steps to reproduce the issue:

  1. Run qos test with t2 topo and also latest 202205 images.

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

ysmanman commented 1 year ago

Add @arlakshm @kenneth-arista for visibility.

ysmanman commented 1 year ago

The neighbor information was correct in asic config_db. For example, take Ethernet144 as an example:

admin@cmp227-4:~$ sudo sonic-db-cli -n asic1 CONFIG_DB hgetall "BGP_NEIGHBOR|10.0.0.37"
{'admin_status': 'up', 'asn': '65200', 'holdtime': '10', 'keepalive': '3', 'local_addr': '10.0.0.36', 'name': 'ARISTA19T3', 'nhopself': '0', 'rrclient': '0'}
admin@cmp227-4:~$

However, only BPG neighbor of asic0 was added in the config_db of default namespace. This is why the neighbor name and ip of asic1 were missing in the show cmd.

admin@cmp227-4:~$ sudo sonic-db-cli CONFIG_DB keys "BGP_NEIGHBOR|*"                                                    
BGP_NEIGHBOR|fc00::22
BGP_NEIGHBOR|fc00::a
BGP_NEIGHBOR|fc00::3a
BGP_NEIGHBOR|10.0.0.13
BGP_NEIGHBOR|fc00::32
BGP_NEIGHBOR|10.0.0.21
BGP_NEIGHBOR|10.0.0.1
BGP_NEIGHBOR|fc00::42
BGP_NEIGHBOR|10.0.0.5
BGP_NEIGHBOR|10.0.0.29
BGP_NEIGHBOR|fc00::46
BGP_NEIGHBOR|10.0.0.25
BGP_NEIGHBOR|10.0.0.35
BGP_NEIGHBOR|10.0.0.17
BGP_NEIGHBOR|fc00::1a
BGP_NEIGHBOR|10.0.0.9
BGP_NEIGHBOR|fc00::2
BGP_NEIGHBOR|10.0.0.33
BGP_NEIGHBOR|fc00::12
BGP_NEIGHBOR|fc00::2a
admin@cmp227-4:~$
arlakshm commented 1 year ago

This PR https://github.com/sonic-net/sonic-mgmt/pull/8884 is causing this issue