search

project-chip / connectedhomeip

Matter (formerly Project CHIP) creates more connections between more objects, simplifying development for manufacturers and increasing compatibility for consumers, guided by the Connectivity Standards Alliance.
https://buildwithmatter.com
Apache License 2.0
7.48k stars 2.01k forks source link

[CERT-TEST-FAILURE] [TC-IDM-10.2][TC-IDM-10.3] - python script Failure #31315

Open Rajashreekalmane opened 10 months ago

Rajashreekalmane commented 10 months ago

Feature Area

Other

Test Case

[TC-IDM-10.2][TC-IDM-10.3]

Reproduction steps

Command to execute the [TC-IDM-10.2][TC-IDM-10.3] : python3 src/python_testing/TC_DeviceConformance.py --discriminator 3840 --passcode 20202021 --commissioning-method on-network --storage-path admin_storage.json

Bug prevalence

Whenever I do this

GitHub hash of the SDK that was being used

e241e32083a1b21dfdef438f17ffdf834b5d1153

Platform

raspi

Anything else?

TC_DeviceConformance.py are failed with the error

TestFailure: Details=Require either --qr-code or --manual-code., Extras=None

Below the traceback error i got in the Execution:

[MatterTest] 01-09 12:22:23.426 INFO [Test] test_run_commissioning PASS [MatterTest] 01-09 12:22:23.431 INFO Summary for test class CommissionDeviceTest: Error 0, Executed 1, Failed 0, Passed 1, Requested 1, Skipped 0 [MatterTest] 01-09 12:22:23.436 INFO ==========> TC_DeviceConformance <========== [MatterTest] 01-09 12:22:23.440 ERROR Error in TC_DeviceConformance#setup_class. Traceback (most recent call last): File "/home/ubuntu/CNTR/connectedhomeip/build-env/lib/python3.10/site-packages/mobly/base_test.py", line 418, in _setup_class self.setup_class() File "/home/ubuntu/CNTR/connectedhomeip/src/python_testing/matter_testing_support.py", line 1317, in async_runner return asyncio.run(runner_with_timeout) File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for return fut.result() File "/home/ubuntu/CNTR/connectedhomeip/src/python_testing/TC_DeviceConformance.py", line 222, in setup_class await self.setup_class_helper() File "/home/ubuntu/CNTR/connectedhomeip/src/python_testing/TC_DeviceConformance.py", line 32, in setup_class_helper await super().setup_class_helper() File "/home/ubuntu/CNTR/connectedhomeip/src/python_testing/basic_composition_support.py", line 108, in setup_class_helper info = self.get_setup_payload_info() File "/home/ubuntu/CNTR/connectedhomeip/src/python_testing/matter_testing_support.py", line 881, in get_setup_payload_info asserts.fail("Require either --qr-code or --manual-code.") File "/home/ubuntu/CNTR/connectedhomeip/build-env/lib/python3.10/site-packages/mobly/asserts.py", line 475, in fail raise signals.TestFailure(msg, extras) mobly.signals.TestFailure: Details=Require either --qr-code or --manual-code., Extras=None [MatterTest] 01-09 12:22:23.456 INFO ***** Test Failure : [MatterTest] 01-09 12:22:23.457 INFO Finished test in 17ms [MatterTest] 01-09 12:22:23.487 INFO Summary for test class TC_DeviceConformance: Error 1, Executed 0, Failed 0, Passed 0, Requested 2, Skipped 2 [MatterTest] 01-09 12:22:23.489 INFO Summary for test run MatterTest@01-09-2024_12-22-22-772: Total time elapsed 0.7144900680000035s

Reference log: TC-IDM-10.xx.txt

cecille commented 9 months ago

This test defaults to being run over PASE, which requires the QR or manual code. If you're running with a commissioned device, you can use the flag --bool-arg use_pase_only:false to run over CASE.

This test will also fail on all clusters because the app really does have conformance issues.

Rajashreekalmane commented 9 months ago

@cecille We used manual code pair to overcome this issue but in current SDK it's failing by throwing a below mentioned error.

Problem: ProblemSeverity.WARNING test_name: Spec XML parsing location: Endpoint: 0, Cluster: 70 (0x46) IcdManagement problem: ConformanceException(Conformance specifies feature not in feature table: LITS) spec_location: [MatterTest] 01-24 12:06:19.969 INFO Problem: ProblemSeverity.WARNING test_name: Spec XML parsing location: Endpoint: 0, Cluster: 40 (0x28) BasicInformation, Event: 3 (0x03) problem: Unable to find conformance element spec_location: [MatterTest] 01-24 12:06:19.970 INFO Problem: ProblemSeverity.WARNING test_name: Spec XML parsing location: Endpoint: 0, Cluster: HERE IS THE PROBLEM, Attribute:0 (0x00) problem: Unable to find access element

PFA log below : TC-IDM-10.2.log TC-IDM-10.3.log

cecille commented 9 months ago

So these actually are legitimate conformance errors found on the all clusters app. So really, it's good that these tests are failing. I've been chasing conformance problems in the sdk for some time now, but the only one I've managed to get fully under control in the CI is door lock. All clusters will likely never pass the conformance test since it's really just a QA thing and it has a million clusters randomly stuck on endpoints in a nonsensical way.

IDM-1.2 right now passes on door lock in the CI. It MAY also pass for the following examples (or it did the last time I checked, but the groups never enabled these in the CI, so they're kind of on their own to keep these up to date):

I don't believe TC-IDM-10-3 will pass for any example right now - none of the app developers kept up with the spec changes as far as i can tell. None the less, I do think we need to keep this test in for TE2, especially in light of this, because failing at TE2 is better than failing at SVE or cert.

Do we have a category for "failed correctly" in the spreadsheet :)?