search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
36 stars 22 forks source link

[Feature] Structured Templates for Test Harness Project Configuration Parameters #361

Open ZainZhongUL opened 1 month ago

ZainZhongUL commented 1 month ago

Feature description

Currently, the documentation for test configuration parameters, particularly the test_parameters section, is limited. This makes it challenging for users to determine the required parameters for different tests and can lead to errors and time-consuming debugging.

This feature request proposes adding structured templates for the test_parameters section of common automated and Python-UI tests. These templates would pre-populate the required parameters with clear descriptions and examples, allowing users to quickly and accurately configure their test projects.

See the detailed discussion in Slack: https://csamembers.slack.com/archives/C07DZVBQ9S5/p1723476212001589

Use Cases

TC-IDM-10.2: The required network parameters for this test are not clearly documented, leading to potential configuration errors. TC-SMOKECO-2.3: The TEST_EVENT_TRIGGER_KEY parameter seems to require manual adjustment in the config, as the value from PICS is not being read correctly by the Test Harness. A template could ensure this parameter is correctly set. TC-WNCV-3.5: This test requires fastMotionDuration and fullMotionDuration parameters specific to the DUT, which are not consistently documented. A template could guide users in providing these values. Other tests: There are likely other tests with similar issues where structured templates would improve the user experience.

Test Cases

No response

Additional Information

No response

fabiowmm commented 4 weeks ago

@raju-apple I don't think detailed test case information belongs to the tool user guide. My proposal is the same as Sander's: Add a section on the user guide referring to the verification steps document as the proper place to find TC guidance and ask the team in charge of VS to add TH steps in addition to manual steps.

fabiowmm commented 1 week ago

This is the same as #333. Look there for updates.