search

sonic-net / sonic-mgmt

Configuration management examples for SONiC
Other
201 stars 727 forks source link

[Bug]: [Dualtor-AA] test_vlan_ping.py would failed when randomly choose lower tor as duthost #15061

Open echuawu opened 3 weeks ago

echuawu commented 3 weeks ago

Issue Description

When the Lower tor was choosen as duthost, the test_vlan_ping.py would failed due to packet could not be received.

Results you see

06:23:35 init.pytest_runtest_call L0040 ERROR | Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 1788, in runtest self.ihook.pytest_pyfunc_call(pyfuncitem=self) File "/usr/local/lib/python3.8/dist-packages/pluggy/_hooks.py", line 513, in call return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) File "/usr/local/lib/python3.8/dist-packages/pluggy/_manager.py", line 120, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 139, in _multicall raise exception.with_traceback(exception.traceback) File "/usr/local/lib/python3.8/dist-packages/pluggy/_callers.py", line 103, in _multicall res = hook_impl.function(*args) File "/usr/local/lib/python3.8/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call result = testfunction(**testargs) File "/root/mars/workspace/sonic-mgmt/tests/vlan/test_vlan_ping.py", line 250, in test_vlan_ping verify_icmp_packet(duthost.facts['router_mac'], ptfhost_info[member], File "/root/mars/workspace/sonic-mgmt/tests/vlan/test_vlan_ping.py", line 219, in verify_icmp_packet raise e # If it fails on the last attempt, raise the exception File "/root/mars/workspace/sonic-mgmt/tests/vlan/test_vlan_ping.py", line 214, in verify_icmp_packet testutils.verify_packet_any_port(ptfadapter, exptd_pkt, dst_port['port_index_list']) File "/usr/local/lib/python3.8/dist-packages/ptf/testutils.py", line 3441, in verify_packet_any_port test.fail( File "/usr/lib/python3.8/unittest/case.py", line 753, in fail raise self.failureException(msg) AssertionError: Did not receive expected packet on any of ports [66, 64, 67, 65, 70, 68, 71, 69, 74, 72, 75, 73, 78, 76, 79, 77] for device 0. ========== EXPECTED ========== Mask:

packet status: OK packet: 0000 5A EF 86 8B 24 F5 9C 05 91 30 F8 00 08 00 45 00 Z...$....0....E. 0010 00 2E 00 01 00 00 3F 01 A7 EF C0 A8 07 FE 0A 00 ......?......... 0020 01 39 08 00 46 4E 00 00 00 00 30 30 30 30 30 30 .9..FN....000000 0030 30 30 30 30 30 30 30 30 30 30 30 30 000000000000

packet's mask: 0000 FF FF FF FF FF FF 00 00 00 00 00 00 FF FF FF FF ................ 0010 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................ 0020 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................ 0030 FF FF FF FF FF FF FF FF FF FF FF FF ............

========== RECEIVED ========== 143 total packets. Displaying most recent 3 packets:

0000 00 AA BB CC DD EE 04 27 28 7A 00 28 08 00 45 B8 .......'(z.(..E. 0010 00 37 CD 7C 00 00 40 01 E1 B7 C0 A8 00 0D 0A 01 .7.|..@......... 0020 00 24 00 00 E1 C2 00 28 1C 37 47 65 6D 69 00 00 .$.....(.7Gemi.. 0030 00 00 72 21 98 D7 F5 02 4E 12 00 00 00 00 00 00 ..r!....N....... 0040 00 00 FF 00 00 .....

0000 32 10 C1 A8 E5 22 9C 05 91 E7 49 00 08 00 45 C0 2...."....I...E. 0010 00 34 D4 9E 40 00 01 06 8F ED 0A 00 00 3C 0A 00 .4..@........<.. 0020 00 3D 00 B3 96 1D 42 96 8B 17 33 08 CD FE 80 10 .=....B...3..... 0030 01 D0 85 7B 00 00 01 01 08 0A 80 45 F0 D0 15 BE ...{.......E.... 0040 EE 9F ..

0000 5A EF 86 8B 24 F5 9C 05 91 E7 49 00 08 00 45 C0 Z...$.....I...E. 0010 00 34 C2 59 40 00 01 06 A2 3A 0A 00 00 38 0A 00 .4.Y@....:...8.. 0020 00 39 B0 22 00 B3 32 9B F4 17 18 2F ED 41 80 10 .9."..2..../.A.. 0030 02 EB A2 DD 00 00 01 01 08 0A 06 56 87 4E 54 83 ...........V.NT. 0040 FD 62 .b

FAILED

Results you expected to see

Packet would be forwarded correctly

Is it platform specific

generic

Relevant log output

MSN4700 with hwsku Mellanox-SN4700-V64 and deployed with dualtor-aa-64-breakout topology.

Output of show version

202405

Attach files (if any)

No response

bingwang-ms commented 1 week ago

Is the issue addressed by PR https://github.com/sonic-net/sonic-mgmt/pull/12306/files?

echuawu commented 1 week ago

Is the issue addressed by PR https://github.com/sonic-net/sonic-mgmt/pull/12306/files?

Hi @bingwang-ms , we have this PR, but the case still failed from time to time.

bingwang-ms commented 6 days ago

@echuawu What the reason for the packet not received?

echuawu commented 5 days ago

@echuawu What the reason for the packet not received?

According to the test log, it seems like the nic simulator had been set to drop state.