project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
42 stars 23 forks source link

[Feature] Don't require recommissioning the DUT for every test #456

Open cecille opened 1 month ago

cecille commented 1 month ago

Feature description

With the time required to factory reset the device, delay starting the commissioning, delay starting the test, and re-running the failed test, this add hours to the testing process even for small devices.

Use Cases

Commission device once, run all tests.

Test Cases

just run through multiple tests in a row

Additional Information

No response

fabiowmm commented 4 weeks ago

@cecille The current logic in TH for running python tests is as follows:

The inclusion of test scripts in each category above is determined by the value of the is_commissioning variable defined in the script code (True means type 1, False means type 2 and undefined/none means type 3).

So, except for the Old script format category, TH does not try to commission before each test.

cecille commented 2 weeks ago

This is a slightly different problem - I think I mean re-commissioning at each run. The TH often fails to start tests, resulting in test runs with partial failures. To re-run these tests, you need to create a new test run with only the failing tests etc. until all the tests are run. The back and forth of creating new test runs with only the failing tests is challenging, as is identifying which run holds the pass result for a given test. As a result the ATLs seem to have developed some workarounds. One such workaround is their recommendation that if you're going to use the UI that test be run one at a time, one per run, so you can just re-run the entire run and see the passing result.

This makes it easier to determine when the correct set of tests has been run, but it also means that the device needs to be recommissioned before every time. The first thing the TH seems to do before each test run is to ask you to put the device in commissioning mode.

Storing the admin kvs between runs would allow folks to run multiple test runs without re-commissioning.