search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
42 stars 24 forks source link

[Bug] Testharness 1.4 selects too many not needed tests #437

Closed LarsOesten closed 1 month ago

LarsOesten commented 1 month ago

Describe the bug

The test plan generator in the Testharness selects far to many tests. Most of these tests are even not applicable to the selected xml file. The online pics tool prints out the correct tests according to the selected xml file

Steps to reproduce the behavior

  1. Select a single xml file, e.g. "Access Control Cluster Test Plan.xml", and create a new project in the testharness
  2. approximately 75 tests are selected

Expected behavior

The online pics tools selects 11 TC-ACL tests, if "Access Control Cluster Test Plan.xml" is used. The testharness should have the same amount of tests.

Log files

No response

PICS file

default "Access Control Cluster Test Plan.xml" for Matter 1.4. All pics are set to be supported.

Screenshots

No response

Environment

No response

Additional Information

No response

fabiowmm commented 1 month ago

@raju-apple again this was a explicit design decision from the PICS V2 feature as can be seen in the excerpt below:

Automatic test case selection: TH will try to run ALL TCs that don't have top-level PICS annotation (the pics_ method). The applicability of those TCs to a specific DUT will be decided by the TC script itself. For those which do have annotations we keep deciding based in the PICS file.

We need to revisit this with the SDK team.

fabiowmm commented 1 month ago

Regarding the issues faced with the TH Certification Mode, our recommendation is for users to initially run test cycles against their DUTS in certification mode and then afterwards repeat the failing ones in the regular (non-certification) mode. The test logs from the regular mode are no different from the ones generated by certification mode and can also be used for certification. Certification mode is an experiment on providing some DUT-driven test case selection in which all test scripts are launched by TH and each script itself figures out whether it should run or not. We learned from the SVE that, due to the current latency when running multiple python tests from the UI, this is not yet very practical.