search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
38 stars 23 forks source link

[Feature] Common Failures Documents Presentation in UI #230

Open antonio-amjr opened 6 months ago

antonio-amjr commented 6 months ago

Feature description

This new feature consists of giving the user easy access to common failures causes for specific failed test cases. The idea is allow the user access that information from the documents without even leaving the Test Harness application.

A folder is maintained in the matter-test-scritps repository with the common failures documents. Also, this repository is already mapped inside backend's file structure.

The following images illustrates how the feature could be done in the current state of the Test Harness:

Test Results with failed test TC-IDM-10.2:

Test_Result_badge_icon2

After clicking the info badge of TC-IDM-10.2:

Test_Result_with_popup

Use Cases

[Use Case 1] Verify the common failures information for the failed TC-RR-1.1:

  1. Execute the test case TC-RR-1.1
  2. The test results in failure
  3. Verify the common failure information about the TC-RR-1.1 test

Test Cases

[Test Case 1] Verify that the common failure button does not appear in case the TC-RR-1.1 passes with no error.

[Test Case 2] Enforcing some error in the TC-RR-1.1 test, verify that the the common failures button is available, and the information is accessible upon click, showing the same information of the matter-test-scripts document.

Additional Information

There is a discussion thread about another interesting approach, where one could have hints from the test scripts themselves. Maybe it's worth to discuss this option further.

rquidute commented 5 months ago

Another possible suggestion: At the end of the test run execution, the TH shows a prompt message stating the failed tests and the links to the respective common failure page for each test case.