Test Case execution FAILS with Pretest in CEOS - T1 Hardware Topology

[mithun2498]

Description

Test case execution gets interrupted and FAILS with Pretest using CEOS image in T1 hardware topology.

Steps to reproduce the issue:

1. Deploy T1 topology using ceos.
2. Execute T1 test cases with pretest.

Describe the results you received: Case getting Failed with pretest.

Describe the results you expected: The test case should run with pretest and produce a valid result.

Additional information you deem important:

**Output of `show version`:**

SONiC Software Version: SONiC.202205.264312-5d9658f50 SONiC OS Version: 11 Distribution: Debian 11.6 Kernel: 5.10.0-18-2-amd64 Build commit: 5d9658f50 Build date: Mon May 1 14:04:52 UTC 2023 Built by: AzDevOps@vmss-soni000Z8B Platform: x86_64-accton_as7716_32x-r0 HwSKU: Accton-AS7716-32X ASIC: broadcom ASIC Count: 1 Serial Number: N/A Model Number: N/A Hardware Revision: N/A Uptime: 16:50:26 up 50 min, 1 user, load average: 0.26, 0.40, 0.43

• Tried executing (bgp/test_bgp_bounce.py) test case.

• CLI used - ./run_tests.sh -n vms-sn2700-t1 -d dev-msn2700-01 -c bgp/test_bgp_bounce.py -f ../ansible/testbed.yaml -i ../ansible/lab,../ansible/veos

• generated xml file: /data/sonic-mgmt/tests/logs/30-06-2023/test_bgp_bounce.py/pretest.xml - ---------------------------- live log sessionfinish ---------------------------- INFO root:init.py:64 Can not get Allure report URL. Please check logs =========================== short test summary info ============================ SKIPPED [1] /data/sonic-mgmt/tests/test_pretest.py:301: No URL specified for python saithrift package FAILED test_pretest.py::test_disable_rsyslog_rate_limit[dev-msn2700-01] - Run... = 1 failed, 8 passed, 1 skipped, 2978 deselected, 33 warnings in 671.04 seconds = !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!! Prepare DUT failed, skip testing !!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

bgp_bounce.txt

[mithun2498]

The issue faced is same in vsonic image also.

[mithun2498]

Hello @yxieca @wangxin

Kindly suggest a solution for the issue seen. Thanks

[mithun2498]

Hello @yxieca @wangxin

The above issue is now not seen in "VSONIC" and the results for T0 and T1 hardware topology is stable. However in CEOS I am still facing the above issue (DUT Failure) when the test case is executed with pretest.

[yxieca]

@mithun2498 this looks like an image issue. The eventd docker is not running?

{"changed": false, "cmds": ["docker exec -i eventd sed -i 's/^\$SystemLogRateLimit/#\$SystemLogRateLimit/g' /etc/rsyslog.conf", "docker exec -i eventd supervisorctl restart rsyslogd"], "delta": "0:00:00.083163", "end": "2022-12-26 16:07:11.816900", "failed": true, "msg": "At least running one of the commands failed", "results": [{"cmd": "docker exec -i eventd sed -i 's/^\$SystemLogRateLimit/#\$SystemLogRateLimit/g' /etc/rsyslog.conf", "rc": 1, "stderr": "Error: No such container: eventd\n", "stderr_lines": ["Error: No such container: eventd"], "stdout": "", "stdout_lines": []}, {"cmd": "docker exec -i eventd supervisorctl restart rsyslogd", "rc": 1, "stderr": "Error: No such container: eventd\n", "stderr_lines": ["Error: No such container: eventd"], "stdout": "", "stdout_lines": []}], "start": "2022-12-26 16:07:11.733737"}

[mithun2498]

Hi @yxieca

I tried with our latest build image and also verified that the eventd docker is running. During execution I could see that all the pretests failed.

Output of Show Version:

root@dev-msn2700-01:~# show version

SONiC Software Version: SONiC.202211.0-dirty-20230820.133115 SONiC OS Version: 11 Distribution: Debian 11.7 Kernel: 5.10.0-18-2-amd64 Build commit: e0ef5b980 Build date: Sun Aug 20 13:00:41 UTC 2023 Built by: gitlab-runner@labuser-Super-Server

Platform: x86_64-accton_as7716_32x-r0 HwSKU: Accton-AS7716-32X ASIC: broadcom ASIC Count: 1 Serial Number: N/A Model Number: N/A Hardware Revision: N/A Uptime: 21:40:07 up 6:06, 1 user, load average: 0.15, 0.22, 0.20 Date: Fri 23 Jun 2023 21:40:07