search

project-chip / matter-test-scripts

Test scripts related to Matter Certification
https://csa-iot.org/
Apache License 2.0
9 stars 3 forks source link

TC-DGGEN-1.1 Matter 1.3 TE#2 #138

Open SamCullen-Element opened 8 months ago

SamCullen-Element commented 8 months ago

Please follow the below template to file issues/concerns/feature enhancement against Test Scripts:

Summary Title:

Assessment of PICS usage by the test script not identifying configuration issues

Description:

Test case does not seem to properly assess the PICS vs expectations during execution. What it looks to do is check that the DUT sends what is expected but it does not seem to be able to flag any inconsistencies between the PICS and the subject reports. Subsequently we are getting a PASS for somethings that should fail.

In this example, the uploader is stating no support for DGGEN.S.C01.Rsp or DGGEN.S.A0003, however both are in the lists when read back from the DUT.

Further to this, a mandatory events list item was not checked by the test script due to PICS_EVENT_LIST_ENABLED not been set. Expectation is that this should not need to be present.

Steps to reproduce:

Configure PICS and leave PICS_EVENT_LIST_ENABLED unset, discrepancies do not seem to be flagged.

Logs:

Not available, result was submitted in Matter 1.3 TE#2

Additional Info:

Please include any other additional or relevant information here. Example: Spec version, PICS version, Test plan version, PICS XML files used for testing PICS, Configuration of the project, related issues,  etc.
cecille commented 8 months ago

As with most 1.1 tests, this test does not properly verify the negative test cases. This is the reason we have added an explicit pics checker test.

1.1 tests are being slated for removal post TE2 in favour of the conformance checker and pics. Suggest low priority for this issue as the expectation is that this test case will be removed.

cecille commented 5 months ago

IMO this issue should be close. PICS validation is performed in IDM-10.4, this test, as with ALL tests in the test suite, assume the pics is correct. This test is slated for deletion in 1.4