search

project-chip / matter-test-scripts

Test scripts related to Matter Certification
https://csa-iot.org/
Apache License 2.0
9 stars 4 forks source link

TC-CNET-4.11 YAML Step 7 has incorrect PICS #452

Open simonhmorris1 opened 2 days ago

simonhmorris1 commented 2 days ago

TC-CNET-4.11 YAML Step 7 has incorrect PICS

R1.3 Test Harness (Version: 2.10+spring2024Sha: 5f28838) references incorrect PICS. This causes Step 7 not to be run, therefore failing the test as the second SSID is not connected to.

Description:

In File ~/src/app/tests/suites/certification/Test_TC_CNET_4_11.yaml (below) PICS: CNET.C.C06.TX is referenced, however according to the R1.3 Matter Test Plan, Section 19.2.9. [TC-CNET-4.11] [Wi-Fi] Verification for ConnectNetwork Command [DUT-Server], step 7 should reference CNET.S.C06.Rsp(ConnectNetwork)

           "Step 7: TH sends ConnectNetwork command to the DUT with NetworkID
           field set to PIXIT.CNET.WIFI_2ND_ACCESSPOINT_SSID and Breadcrumb field
           set to 2"
-      PICS: CNET.C.C06.Tx

Steps to reproduce:

Run the TC-CNET-4.11 Test from the GUI, without the DUT CNET,C (Network Commissioning Cluster as a client)

Fix

Update the referenced PICS in Step7 of the YAML file

cecille commented 2 days ago

Suggest just removing the PICS so we stop getting bugs like this. The test is already gated on the WI feature - every command in that test is mandatory based on that feature.

simonhmorris1 commented 1 day ago

If the PICS for that step is removed then it diverges from the Matter Test Plan, making the YAML the definitive source rather than the test plan. So for now changing the YAML to meet the Test plan seems the way forward unless we update the test plan, however updating the test plan would be for R1.5.

In the meantime people need to certify on R1.3 and R1.4 and this YAML test will always fail, so an urgent change for that is needed. Hence make this small change to meet that need, and it will also cover subsequent releases.

I have raised PR https://github.com/project-chip/connectedhomeip/pull/36701 if you are happy with the proposed change.