project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
42 stars 24 forks source link

[Bug] Documentation of the exact semantics and usage of `test_parameters` config section missing #357

Open tcarmelveilleux opened 3 months ago

tcarmelveilleux commented 3 months ago

Describe the bug

test_parameters is critical to python scripts but is not explained in detail in the manual.

It is covered by weak examples in sections:

9.5.2. Run Tests on the TH User Interface

To configure specific/custom parameters, please edit the project configuration to include the parameters in the session (test_parameters).

and

image

This is completely insufficient to understand the rules/usage, and does not offer any guidance for how to deal with large numbers of tests that have different parameters.

For example, different tests may require different arguments but the config is global and all arguments are given to all tests.

Furthermore, there is no validation that some tests need particular arguments (MANDATORY) to run (e.g. TC-IDM-10.2 needs --qr-code or --manal-code) and some tests require PIXIT.xxxx. The format of the parameters is also not validated. This makes running large numbers of automated Python tests error-prone.

Steps to reproduce the behavior

Read the docs as explained above

Expected behavior

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

andy31415 commented 3 months ago

A UI that simplifies/completes the json config would be great.