Open cecille opened 1 month ago
@cecille The current logic in TH for running python tests is as follows:
The inclusion of test scripts in each category above is determined by the value of the is_commissioning
variable defined in the script code (True means type 1, False means type 2 and undefined/none means type 3).
So, except for the Old script format category, TH does not try to commission before each test.
This is a slightly different problem - I think I mean re-commissioning at each run. The TH often fails to start tests, resulting in test runs with partial failures. To re-run these tests, you need to create a new test run with only the failing tests etc. until all the tests are run. The back and forth of creating new test runs with only the failing tests is challenging, as is identifying which run holds the pass result for a given test. As a result the ATLs seem to have developed some workarounds. One such workaround is their recommendation that if you're going to use the UI that test be run one at a time, one per run, so you can just re-run the entire run and see the passing result.
This makes it easier to determine when the correct set of tests has been run, but it also means that the device needs to be recommissioned before every time. The first thing the TH seems to do before each test run is to ask you to put the device in commissioning mode.
Storing the admin kvs between runs would allow folks to run multiple test runs without re-commissioning.
Feature description
With the time required to factory reset the device, delay starting the commissioning, delay starting the test, and re-running the failed test, this add hours to the testing process even for small devices.
Use Cases
Commission device once, run all tests.
Test Cases
just run through multiple tests in a row
Additional Information
No response