search

sonic-net / sonic-buildimage

Scripts which perform an installable binary image build for SONiC
Other
741 stars 1.44k forks source link

[Bug][202405]: `test/test_acl.py` Fail with `AssertionError: Rule counters should be ready!` #20872

Open arista-nwolfe opened 4 days ago

arista-nwolfe commented 4 days ago

Issue Description

test/test_acl.py fails on all tests with:

    def set_up_acl_rules_single_dut(self, acl_table,
                                    conn_graph_facts, dut_to_analyzer_map, duthost, # noqa F811
                                    ip_version, localhost,
                                    populate_vlan_arp_entries, tbinfo):
        logger.info("{}: ACL rule application started".format(duthost.hostname))
        if duthost.is_supervisor_node():
            return
        loganalyzer = LogAnalyzer(ansible_host=duthost, marker_prefix="acl_rules")
        loganalyzer.load_common_config()
        dut_to_analyzer_map[duthost] = loganalyzer
        try:
            loganalyzer.expect_regex = [LOG_EXPECT_ACL_RULE_CREATE_RE]
            # Ignore any other errors to reduce noise
            loganalyzer.ignore_regex = [r".*"]
            with loganalyzer:
                self.setup_rules(duthost, acl_table, ip_version)
                # Give the dut some time for the ACL rules to be applied and LOG message generated
                wait_until(300, 20, 0, check_msg_in_syslog,
                           duthost, LOG_EXPECT_ACL_RULE_CREATE_RE)

            self.post_setup_hook(duthost, localhost, populate_vlan_arp_entries, tbinfo, conn_graph_facts)

>           assert self.check_rule_counters(duthost), "Rule counters should be ready!"
E           AssertionError: Rule counters should be ready!

This appears to be due to the counters showing N/A for the ACL rules:

root@nfc405-7:~# aclshow -a
RULE NAME     TABLE NAME                PRIO  PACKETS COUNT    BYTES COUNT
------------  ----------------------  ------  ---------------  -------------
RULE_1        DATA_INGRESS_IPV4_TEST    9999  N/A              N/A
RULE_2        DATA_INGRESS_IPV4_TEST    9998  N/A              N/A
RULE_3        DATA_INGRESS_IPV4_TEST    9997  N/A              N/A
RULE_4        DATA_INGRESS_IPV4_TEST    9996  N/A              N/A
RULE_5        DATA_INGRESS_IPV4_TEST    9995  N/A              N/A
RULE_6        DATA_INGRESS_IPV4_TEST    9994  N/A              N/A
RULE_7        DATA_INGRESS_IPV4_TEST    9993  N/A              N/A
RULE_8        DATA_INGRESS_IPV4_TEST    9992  N/A              N/A
RULE_9        DATA_INGRESS_IPV4_TEST    9991  N/A              N/A
RULE_10       DATA_INGRESS_IPV4_TEST    9990  N/A              N/A
RULE_11       DATA_INGRESS_IPV4_TEST    9989  N/A              N/A
RULE_12       DATA_INGRESS_IPV4_TEST    9988  N/A              N/A
RULE_13       DATA_INGRESS_IPV4_TEST    9987  N/A              N/A
RULE_14       DATA_INGRESS_IPV4_TEST    9986  N/A              N/A
RULE_15       DATA_INGRESS_IPV4_TEST    9985  N/A              N/A
RULE_16       DATA_INGRESS_IPV4_TEST    9984  N/A              N/A
RULE_17       DATA_INGRESS_IPV4_TEST    9983  N/A              N/A
RULE_18       DATA_INGRESS_IPV4_TEST    9982  N/A              N/A
RULE_19       DATA_INGRESS_IPV4_TEST    9981  N/A              N/A
RULE_20       DATA_INGRESS_IPV4_TEST    9980  N/A              N/A
RULE_21       DATA_INGRESS_IPV4_TEST    9979  N/A              N/A
RULE_22       DATA_INGRESS_IPV4_TEST    9978  N/A              N/A
RULE_23       DATA_INGRESS_IPV4_TEST    9977  N/A              N/A
RULE_24       DATA_INGRESS_IPV4_TEST    9976  N/A              N/A
RULE_25       DATA_INGRESS_IPV4_TEST    9975  N/A              N/A
RULE_26       DATA_INGRESS_IPV4_TEST    9974  N/A              N/A
RULE_27       DATA_INGRESS_IPV4_TEST    9973  N/A              N/A
RULE_28       DATA_INGRESS_IPV4_TEST    9972  N/A              N/A
RULE_29       DATA_INGRESS_IPV4_TEST    9971  N/A              N/A
RULE_30       DATA_INGRESS_IPV4_TEST    9970  N/A              N/A
RULE_31       DATA_INGRESS_IPV4_TEST    9969  N/A              N/A
RULE_32       DATA_INGRESS_IPV4_TEST    9968  N/A              N/A
RULE_33       DATA_INGRESS_IPV4_TEST    9967  N/A              N/A
DEFAULT_RULE  DATA_INGRESS_IPV4_TEST       1  N/A              N/A

We confirmed the issue appears to have been introduced by https://github.com/sonic-net/sonic-buildimage/pull/20555 After reverting this change we no longer see test/test_acl.py failing:

acl/test_acl.py::TestBasicAcl::test_ingress_unmatched_blocked[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                        [  0%]
acl/test_acl.py::TestBasicAcl::test_egress_unmatched_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] SKIPPED (Only run for eg...) [  1%]
acl/test_acl.py::TestBasicAcl::test_source_ip_match_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                        [  1%]
acl/test_acl.py::TestBasicAcl::test_rules_priority_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                         [  2%]
acl/test_acl.py::TestBasicAcl::test_rules_priority_dropped[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                           [  2%]
acl/test_acl.py::TestBasicAcl::test_dest_ip_match_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                          [  3%]
...
acl/test_acl.py::TestBasicAcl::test_tcp_flags_match_dropped[ipv6-ingress-uplink->downlink-default-no_vlan] PASSED                          [ 99%]
acl/test_acl.py::TestBasicAcl::test_icmp_match_forwarded[ipv6-ingress-uplink->downlink-default-no_vlan] PASSED                             [100%]

Results you see

test/test_acl.py is seeing aclshow -a display N/A for the rule counters

Results you expected to see

When test/test_acl.py creates ACL rules aclshow -a show display an integer for the rule counter

Is it platform specific

generic

Relevant log output

No response

Output of show version

No response

Attach files (if any)

No response

Javier-Tan commented 3 days ago

Investigated and found the same thing yesterday

For more reference, this is exactly where aclshow -a fails to pull the counters and delivers N/A

admin@sonic:~$ sonic-db-cli -n asic0 COUNTERS_DB HGETALL "ACL_COUNTER_RULE_MAP"
{'DATA_INGRESS_IPV4_TEST:DEFAULT_RULE': 'oid:0x9000000000947', 'DATA_INGRESS_IPV4_TEST:RULE_1': 'oid:0x9000000000949', 'DATA_INGRESS_IPV4_TEST:RULE_10': 'oid:0x900000000094b', 'DATA_INGRESS_IPV4_TEST:RULE_11': 'oid:0x900000000094e', 'DATA_INGRESS_IPV4_TEST:RULE_12': 'oid:0x9000000000951', 'DATA_INGRESS_IPV4_TEST:RULE_13': 'oid:0x9000000000953', 'DATA_INGRESS_IPV4_TEST:RULE_14': 'oid:0x9000000000955', ...

admin@sonic:~$ sonic-db-cli -n asic0 COUNTERS_DB keys COUNTERS:oid:90*
(nothing returned -> `aclshow -a` returns N/A for each counter it can't find)
rlhui commented 2 days ago

@stepanblyschak could you please help check this? Thanks.

bingwang-ms commented 2 days ago

Hi @Javier-Tan Is there any other counter issue with this change #20555?

stepanblyschak commented 2 days ago

@bingwang-ms This change should come together with https://github.com/sonic-net/sonic-buildimage/pull/20554 and https://github.com/sonic-net/sonic-swss/pull/3326