sonic-net / sonic-swss

SONiC Switch State Service (SwSS)
https://azure.github.io/SONiC
Other
170 stars 512 forks source link

[VS] All subport tests are consistently failing #2365

Open prsunny opened 2 years ago

prsunny commented 2 years ago

All recent PRs have subport tests consistently failing

test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_creation FAILED [ 82%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_add_ip_addrs FAILED [ 82%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_appl_db_proc_seq FAILED [ 82%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_admin_status_change FAILED [ 82%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_remove_ip_addrs FAILED [ 83%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_removal FAILED [ 83%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_mtu FAILED    [ 83%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_nhg_accel FAILED [ 83%]
test_sub_port_intf.py::TestSubPortIntf::test_sub_port_intf_oper_down_with_pending_neigh_route_tasks FAILED [ 83%]
    def get_default_vrf_oid(self):
        oids = self.get_oids(ASIC_VIRTUAL_ROUTER_TABLE)
>       assert len(oids) == 1, "Wrong # of default vrfs: %d, expected #: 1." % (len(oids))
E       AssertionError: Wrong # of default vrfs: 2, expected #: 1.
E       assert 2 == 1
E         -2
E         +1

More logs at - https://dev.azure.com/mssonic/build/_build/results?buildId=116747&view=logs&j=3f6395b2-1619-5ebe-f305-2aedcf353cb5&t=f9db2383-9349-50ba-61c3-4bde15cec59e

An urgent fix is required to unblock the PRs.

prsunny commented 2 years ago

@prabhataravind looking into this issue

prsunny commented 2 years ago

@liuh-80 for viz

prsunny commented 2 years ago

Update: Tests started failing from 6/24

prabhataravind commented 2 years ago

Manual script run passed several times locally. Issue seems to be due to a lack of cleanup of vrfs from prior test cases. I'm working on identifying that.

~/sonic-swss/tests$ sudo pytest test_sub_port_intf.py ==================================================================== test session starts ==================================================================== platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/paravind/sonic-swss/tests plugins: flaky-3.7.0 collected 10 items

test_sub_port_intf.py .......... [100%]

prsunny commented 2 years ago

@neethajohn for viz

prsunny commented 2 years ago

https://github.com/Azure/sonic-swss/pull/2384 identifies the failing commit

prsunny commented 2 years ago

@liorghub is looking into this issue