search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
38 stars 23 forks source link

Failure of 'Python Testing Suite - Legacy' Test Cases When Executed Alongside 'Python Testing Suite' Cases #211

Open Survensa opened 7 months ago

Survensa commented 7 months ago

Describe the bug

Test cases in the "Python Testing Suite - Legacy" fail when executed alongside test cases from both the "Python Testing Suite" and "Python Testing Suite - Legacy." However, there are no issues when multiple test cases are executed within their respective suites, either "Python Testing Suite" or "Python Testing Suite - Legacy."

Steps to reproduce the behavior

  1. Select one test case from Python Testing Suite.
  2. Select one test case from Python Testing Suite - Legacy.
  3. Run the test case.

For example :

image

Expected behavior

The test case should execute without any issues irrespective of the suite from which it is selected.

Log files

UI_Test_Run_2024_03_06_16_07_41.log

Screenshots

image

Environment

Version: 2.10-beta3+spring2024 Sha: 1fd1429

rquidute commented 7 months ago

Hi @Survensa, I performed the execution of TC-ACL-2.2 along TC-SM-1.1 it worked fine.

Screenshot 2024-03-07 at 08 58 09

Regarding the execution of TC-ACE-1.2, when I performed the test execution, it is always failing even running alone. I'm still investigating the reason.