[acl] TestAclWithReboot fails because expected packets were not received

[theasianpianist]

Description

Steps to reproduce the issue:

1. Run nightly test cases
2. Check failures

Describe the results you received:

self = <test_acl.TestAclWithReboot object at 0x7fe0904b1490>
setup = {'acl_table_ports': ['Ethernet108', 'Ethernet100', 'Ethernet104', 'Ethernet96', 'Ethernet124', 'Ethernet92', ...], 'dst_ip_spine': '192.168.128.1', 'dst_ip_spine_blocked': '192.168.144.1', 'dst_ip_spine_forwarded': '192.168.136.1', ...}
direction = 'spine->tor'
ptfadapter = <tests.common.plugins.ptfadapter.ptfadapter.PtfTestAdapter testMethod=runTest>
counters_sanity_check = []

    def test_source_ip_match_forwarded(self, setup, direction, ptfadapter, counters_sanity_check):
        """ test source IP matched packet is forwarded """

        pkt = self.tcp_packet(setup, direction, ptfadapter)
        pkt['IP'].src = '20.0.0.2'
        exp_pkt = self.expected_mask_routed_packet(pkt)

        ptfadapter.dataplane.flush()
        testutils.send(ptfadapter, self.get_src_port(setup, direction), pkt)
>       testutils.verify_packet_any_port(ptfadapter, exp_pkt, ports=self.get_dst_ports(setup, direction))

counters_sanity_check = []
direction  = 'spine->tor'
exp_pkt    = <ptf.mask.Mask instance at 0x7fe08ad794b0>
pkt        = <Ether  dst=c0:d6:82:b8:ca:51 src=9a:5f:d0:cf:0a:00 type=0x800 |<IP  ihl=None ...t=81 flags=S |<Raw  load='test_acl test_acl test_acl test_acl test_acl t' |>>>>
ptfadapter = <tests.common.plugins.ptfadapter.ptfadapter.PtfTestAdapter testMethod=runTest>
self       = <test_acl.TestAclWithReboot object at 0x7fe0904b1490>
setup      = {'acl_table_ports': ['Ethernet108', 'Ethernet100', 'Ethernet104', 'Ethernet96', 'Ethernet124', 'Ethernet92', ...], 'dst_ip_spine': '192.168.128.1', 'dst_ip_spine_blocked': '192.168.144.1', 'dst_ip_spine_forwarded': '192.168.136.1', ...}

acl/test_acl.py:442: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/dist-packages/ptf/testutils.py:2538: in verify_packet_any_port
    % (ports, device_number, result.format()))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.common.plugins.ptfadapter.ptfadapter.PtfTestAdapter testMethod=runTest>
msg = 'Did not receive expected packet on any of ports [27, 25, 26, 24, 31, 23, 30, 19, 18, 22, 29, 20, 28, 21, 16, 17] for ...  .>.....H........\n0050   EF 51 04 69 66 08                                  .Q.if.\n==============================\n'

    def fail(self, msg=None):
        """Fail immediately, with the given message."""
>       raise self.failureException(msg)
E       AssertionError: Did not receive expected packet on any of ports [27, 25, 26, 24, 31, 23, 30, 19, 18, 22, 29, 20, 28, 21, 16, 17] for device 0.
E       ========== EXPECTED ==========
E       Mask:
E       0000   C0 D6 82 B8 CA 51 9A 5F  D0 CF 0A 00 08 00 45 00   .....Q._......E.
E       0010   00 56 00 01 00 00 3F 06  A6 F6 14 00 00 02 C0 A8   .V....?.........
E       0020   00 01 43 21 00 51 00 00  00 00 00 00 00 00 50 02   ..C!.Q........P.
E       0030   20 00 C7 96 00 00 74 65  73 74 5F 61 63 6C 20 74    .....test_acl t
E       0040   65 73 74 5F 61 63 6C 20  74 65 73 74 5F 61 63 6C   est_acl test_acl
E       0050   20 74 65 73 74 5F 61 63  6C 20 74 65 73 74 5F 61    test_acl test_a
E       0060   63 6C 20 74                                        cl t
E       mask = 00 00 00 00 00 00 00 00  00 00 00 00 ff ff ff ff
E       0010   ff ff ff ff ff ff ff ff  00 00 ff ff ff ff ff ff
E       0020   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0030   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0040   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0050   ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
E       0060   ff ff ff ff  
E       
E       ========== RECEIVED ==========
E       16 total packets. Displaying most recent 3 packets:
E       ------------------------------
E       0000   52 54 00 91 C9 2A C0 D6  82 B8 CA 51 86 DD 6C 04   RT...*.....Q..l.
E       0010   CF B4 00 20 06 01 FC 00  00 00 00 00 00 00 00 00   ... ............
E       0020   00 00 00 00 00 51 FC 00  00 00 00 00 00 00 00 00   .....Q..........
E       0030   00 00 00 00 00 52 81 0A  00 B3 04 AA EA B3 EF 93   .....R..........
E       0040   B4 5B 80 10 01 EC F4 92  00 00 01 01 08 0A 70 67   .[............pg
E       0050   BB D0 04 69 41 EE                                  ...iA.
E       ------------------------------
E       0000   52 54 00 BC 9D 8A C0 D6  82 B8 CA 51 86 DD 6C 00   RT.........Q..l.
E       0010   61 32 00 33 06 01 FC 00  00 00 00 00 00 00 00 00   a2.3............
E       0020   00 00 00 00 00 45 FC 00  00 00 00 00 00 00 00 00   .....E..........
E       0030   00 00 00 00 00 46 B9 10  00 B3 DF 21 85 28 9B 1C   .....F.....!.(..
E       0040   B5 2B 80 18 01 EC BC 47  00 00 01 01 08 0A F6 BC   .+.....G........
E       0050   EF 38 04 69 63 1A FF FF  FF FF FF FF FF FF FF FF   .8.ic...........
E       0060   FF FF FF FF FF FF 00 13  04                        .........
E       ------------------------------
E       0000   52 54 00 BC 9D 8A C0 D6  82 B8 CA 51 86 DD 6C 00   RT.........Q..l.
E       0010   61 32 00 20 06 01 FC 00  00 00 00 00 00 00 00 00   a2. ............
E       0020   00 00 00 00 00 45 FC 00  00 00 00 00 00 00 00 00   .....E..........
E       0030   00 00 00 00 00 46 B9 10  00 B3 DF 21 85 3B 9B 1C   .....F.....!.;..
E       0040   B5 3E 80 10 01 EC BD 48  00 00 01 01 08 0A F6 BC   .>.....H........
E       0050   EF 51 04 69 66 08                                  .Q.if.
E       ==============================

msg        = 'Did not receive expected packet on any of ports [27, 25, 26, 24, 31, 23, 30, 19, 18, 22, 29, 20, 28, 21, 16, 17] for ...  .>.....H........\n0050   EF 51 04 69 66 08                                  .Q.if.\n==============================\n'
self       = <tests.common.plugins.ptfadapter.ptfadapter.PtfTestAdapter testMethod=runTest>

/usr/lib/python2.7/unittest/case.py:410: AssertionError

Describe the results you expected: Test does not fail

Additional information you deem important:

**Output of `show version`:**

admin@str2-7050cx3-acs-03:~$ show ver

SONiC Software Version: SONiC.master.415-d12e9cbb
Distribution: Debian 10.5
Kernel: 4.19.0-9-2-amd64
Build commit: d12e9cbb
Build date: Thu Sep 17 08:28:24 UTC 2020
Built by: johnar@jenkins-worker-8

Platform: x86_64-arista_7050cx3_32s
HwSKU: Arista-7050CX3-32S-C32
ASIC: broadcom
Serial Number: JPE20181528
Uptime: 21:51:21 up 14:21,  1 user,  load average: 0.20, 0.30, 0.29

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-teamd                  latest                f6cb57e45936        390MB
docker-teamd                  master.415-d12e9cbb   f6cb57e45936        390MB
docker-sonic-mgmt-framework   latest                e1d291e35af6        486MB
docker-sonic-mgmt-framework   master.415-d12e9cbb   e1d291e35af6        486MB
docker-router-advertiser      latest                d9a7fc7484e8        359MB
docker-router-advertiser      master.415-d12e9cbb   d9a7fc7484e8        359MB
docker-platform-monitor       latest                46eb51c7c3d5        433MB
docker-platform-monitor       master.415-d12e9cbb   46eb51c7c3d5        433MB
docker-lldp                   latest                6f242ddb4e69        387MB
docker-lldp                   master.415-d12e9cbb   6f242ddb4e69        387MB
docker-dhcp-relay             latest                88d9f48d315d        366MB
docker-dhcp-relay             master.415-d12e9cbb   88d9f48d315d        366MB
docker-database               latest                d9f261bb35a0        359MB
docker-database               master.415-d12e9cbb   d9f261bb35a0        359MB
docker-orchagent              latest                a50b6f7b9898        404MB
docker-orchagent              master.415-d12e9cbb   a50b6f7b9898        404MB
docker-nat                    latest                9df96e22eae5        393MB
docker-nat                    master.415-d12e9cbb   9df96e22eae5        393MB
docker-sonic-telemetry        latest                c43ff6838f29        429MB
docker-sonic-telemetry        master.415-d12e9cbb   c43ff6838f29        429MB
docker-fpm-frr                latest                c2a581d96906        406MB
docker-fpm-frr                master.415-d12e9cbb   c2a581d96906        406MB
docker-sflow                  latest                bfa8b0694a13        394MB
docker-sflow                  master.415-d12e9cbb   bfa8b0694a13        394MB
docker-snmp                   latest                aebcffb1d55f        399MB
docker-snmp                   master.415-d12e9cbb   aebcffb1d55f        399MB
docker-syncd-brcm             latest                aca27d898d0f        447MB
docker-syncd-brcm             master.415-d12e9cbb   aca27d898d0f        447MB

[daall]

this is the same issue as #2294, closing

[Prakash-Sanchela]

Though this issue is old, I am still getting the same issue while running test_acl.py::test_source_ip_match_forwarded[ipv4-ingress-downlink->uplink-default-Vlan1000]. All the test case related to forward specifically for upstream devices are getting failed and all the test case related to drop is passed. Currently using sonic-mgmt 202305 and also dut is running on latest SONiC version, platform currently in use is broadcom.

While debugging this issue, on dut aclshow commands I am getting PKT_COUNTER value as per number of sent packets from .send() api from test case and similarly show interfaces counters also showing the number of packet is getting increased on RX_OK also RX_DRP count is also increased based on number of packet sent. Also I have checked ACL rules looks file.