sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
196 stars 718 forks source link

[bgp_traffic_shift] no neighbors check after restart bgp service | dependency of execution tests order #4337

Closed AntonHryshchuk closed 2 years ago

AntonHryshchuk commented 3 years ago

Description

The test _bgp_traffic_shift::test_TSA_B_C_with_noneighbors doing restart to BGP process in finally step. After that only the BGP docker is checked. No neighbors check.

The test itself passed, but if after it run another BGP test, for example _test_bgp_allowlist or _bgp_traffic_shift::testTSA, it will fail with error message:

_bgp_traffic_shift::testTSA:

  pytest_assert(not res['failed'], "Failed to retrieve routes from VM {}".format(hostname))
  E           Failed: Failed to retrieve routes from VM ARISTA11T0

_test_bgp_allowlist:

  'Unexpected routes on neighbors, failed_results={}'.format(json.dumps(failed_results, indent=2))) 
  E       Failed: Unexpected routes on neighbors, failed_results={ 
  E         "ARISTA03T2": [ 
  E           { 
  E             "reasons": [ 
  E               "Route 172.16.30.0/24 not found on ARISTA03T2" 
  E             ], 
  E             "failed": true, 
  E             "prefix": "172.16.30.0/24" 
  E           }, 
  E           { 
  E             "reasons": [ 
  E               "Route 2000:172:16:50::/64 not found on ARISTA03T2" 
  E             ], 
  E             "failed": true, 
  E             "prefix": "2000:172:16:50::/64" 
  E           }, 
  E    .........

The easy way to fix it is to add checker _verify_all_routes_announce_toneighs for ipv4 and ipv6 at the end of test _bgp_traffic_shift::test_TSA_B_C_with_noneighbors , but it will take many time ~15 minutes for t1-lag topology. Need to add a more faster way to do it. In my debug tries I got, that the time of routes bring up on neighbors is ~3-4 minutes.

Steps to reproduce the issue:

  1. run test test _bgp_traffic_shift::test_TSA_B_C_with_noneighbors on t1-lag topology
  2. run test _bgp_traffic_shift::testTSA or _test_bgp_allowlist

Describe the results you received: The second test always fails due to the fact that on neighbors not all routes are learned yet.

Describe the results you expected: the tests can passed without dependency of execution order

Additional information you deem important:

**Output of `show version`:**

```
root@r-leopard-58:/home/admin# show version 

SONiC Software Version: SONiC.202012.164-f6f4c7f4b_Internal
Distribution: Debian 10.10
Kernel: 4.19.0-12-2-amd64
Build commit: f6f4c7f4b
Build date: Tue Sep 21 09:17:09 UTC 2021
Built by: sw-r2d2-bot@r-build-sonic-ci02-244

Platform: x86_64-mlnx_msn4700-r0
HwSKU: ACS-MSN4700
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2119X03331
Uptime: 06:05:15 up  2:35,  1 user,  load average: 0.92, 0.90, 0.99

Docker images:
REPOSITORY                    TAG                                     IMAGE ID            SIZE
docker-teamd                  202012.164-f6f4c7f4b_Internal           259cca844294        412MB
docker-teamd                  latest                                  259cca844294        412MB
docker-orchagent              202012.164-f6f4c7f4b_Internal           61fe9ee7219a        431MB
docker-orchagent              latest                                  61fe9ee7219a        431MB
docker-nat                    202012.164-f6f4c7f4b_Internal           e3c39e6aa253        415MB
docker-nat                    latest                                  e3c39e6aa253        415MB
docker-fpm-frr                202012.164-f6f4c7f4b_Internal           7eb0ed9d1484        431MB
docker-fpm-frr                latest                                  7eb0ed9d1484        431MB
docker-sflow                  202012.164-f6f4c7f4b_Internal           7e5f5be0fe65        413MB
docker-sflow                  latest                                  7e5f5be0fe65        413MB
docker-syncd-mlnx             202012.164-f6f4c7f4b_Internal           cc71fc9848e8        980MB
docker-syncd-mlnx             latest                                  cc71fc9848e8        980MB
docker-platform-monitor       202012.164-f6f4c7f4b_Internal           d6a2f4889c74        725MB
docker-platform-monitor       latest                                  d6a2f4889c74        725MB
docker-lldp                   202012.164-f6f4c7f4b_Internal           cedf87ed955b        442MB
docker-lldp                   latest                                  cedf87ed955b        442MB
docker-snmp                   202012.164-f6f4c7f4b_Internal           bd834f2d6214        444MB
docker-snmp                   latest                                  bd834f2d6214        444MB
docker-router-advertiser      202012.164-f6f4c7f4b_Internal           a743086d1cb1        402MB
docker-router-advertiser      latest                                  a743086d1cb1        402MB
docker-dhcp-relay             202012.164-f6f4c7f4b_Internal           322ff5dbaa0f        409MB
docker-dhcp-relay             latest                                  322ff5dbaa0f        409MB
docker-database               202012.164-f6f4c7f4b_Internal           e30c78a04899        402MB
docker-database               latest                                  e30c78a04899        402MB
docker-sonic-telemetry        202012.164-f6f4c7f4b_Internal           65dbd84419ec        491MB
docker-sonic-telemetry        latest                                  65dbd84419ec        491MB
docker-sonic-mgmt-framework   202012.164-f6f4c7f4b_Internal           de8407f69e30        816MB
docker-sonic-mgmt-framework   latest                                  de8407f69e30        816MB
docker-wjh                    202012.202012.0-dirty-20210819.160721   7c03b2d081cc        518MB
docker-wjh                    latest 
```
yxieca commented 3 years ago

@lolyu can you look into this issue?