Closed ihorchekh closed 2 years ago
@prsunny Can you please assign it to relevant team to handle?
@rraghav-cisco , would you please check on this?
Pls assign to me.
@dgsudharsan : Pls let me know where to access the script run logs.
@ihorchekh Can you please attach the script run logs in this bug?
@dgsudharsan @rraghav-cisco I've attached three log files:
I ran the script internally attempting to reproduce this problem. What I observed is that the vnets are all removed from the DUT, however they remain attached to the portchannel interfaces. I am not sure if this is cause of the problem listed in this bug.
show run all | less output:
"PORTCHANNEL_INTERFACE": {
"PortChannel0002": {
"vnet_name": "Vnet_v4_in_v6-0"
},
}
So I tried to remove the entries using the following CLI: redis-cli -n 4 hdel "PORTCHANNEL_INTERFACE|PortChannel0019" vnet_name in case of T1-64-lag or redis-cli -n 4 hdel "INTERFACE|Ethernet4" vnet_name in case of T1.
This removes the vnets from the interfaces. However, even after this, the script is failing the second attempt to run it. The reason I found was that the vnet routes are not present in the ASIC DB, even though they are showing up in the "show vnet route all" output.
root@mth64-m5-2:/home/cisco# show vnet route all
vnet name prefix nexthop interface
----------- -------- --------- -----------
vnet name prefix endpoint mac address vni
--------------- ----------------------- -------------------- ------------- -----
Vnet_v4_in_v4-0 150.0.3.1/32 100.0.1.10
Vnet_v4_in_v4-0 150.0.4.1/32 100.0.2.10
Vnet_v4_in_v4-0 fddd:a150:a0::a13:1/128 100.0.11.10
Vnet_v4_in_v4-0 fddd:a150:a0::a14:1/128 100.0.12.10
Vnet_v4_in_v6-0 150.0.8.1/32 fddd:a100:a0::a6:10
Vnet_v4_in_v6-0 150.0.9.1/32 fddd:a100:a0::a7:10
Vnet_v4_in_v6-0 fddd:a150:a0::a18:1/128 fddd:a100:a0::a16:10
Vnet_v4_in_v6-0 fddd:a150:a0::a19:1/128 fddd:a100:a0::a17:10
root@mth64-m5-2:/home/cisco# redis-dump -y -d0 | grep 150.0.3.1
"VNET_ROUTE_TUNNEL_TABLE:Vnet_v4_in_v4-0:150.0.3.1/32": {
root@mth64-m5-2:/home/cisco# redis-dump -y -d1 | grep 150.0.3.1
root@mth64-m5-2:/home/cisco# redis-dump -y -d2 | grep 150.0.3.1
root@mth64-m5-2:/home/cisco#
One way to ensure complete cleanup is to do config reload. But I decided to use the vxlan clean up code from the previous test_vnet_vxlan.py. @prsunny: pls advice.
@ihorchekh , @prsunny , @dgsudharsan , @mattetti : Pls review, and if possible, pls verify the fix in the PR:https://github.com/Azure/sonic-mgmt/pull/5190 and confirm it works for you.
Description The test_vxlan_ecmp.py doesn't do cleanup properly. During the test it creates Vnet_v4_in_v4-0 interface and it remains after the test finished. Moreover, running test in the loop causing failure, because of it.
Steps to reproduce the issue:
Describe the results you received: Leftover after the test finished. Test fails after 2-4 runs
Describe the results you expected: No leftovers. Test pass constantly
Additional information you deem important:
Output of
show version
:Attach debug file
sudo generate_dump
:sonic_dump_r-tigris-13_20220207_081510.tar.gz vxlan_ecmp_run1.txt vxlan_ecmp_run2.txt vxlan_ecmp_run3.txt