search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
40 stars 23 forks source link

[Feature] SDK Python testcases needed to be commissioned again. #146

Closed Saravana-kr22 closed 3 weeks ago

Saravana-kr22 commented 9 months ago

Feature description

As per the latest image : Version: 2.10-beta2+spring2024 SDK Yaml testcases and SDK python testcases need to commissioned seperately for the successful execution.

Here the Example if the both are executed in a single test , the python tet cases is fails with the timeout error

Failed to establish secure session to device: src/controller/python/ChipDeviceController-ScriptPairingDeviceDiscoveryDelegate.h:59: CHIP Error 0x00000032: Timeout Failed to commission: src/controller/python/ChipDeviceController-ScriptPairingDeviceDiscoveryDelegate.h:59: CHIP Error 0x00000032: Timeout [MatterTest] 01-12 11:27:07.710 INFO [Test] test_run_commissioning FAIL

image

Reference log: yaml&py.log

Use Cases

By implementing this feature the whole test suite canbe run in the single commissioning process and does not require the interaction for advertising the dut.

Note: if this feature is not vaild please feel free toclose this issue

ccruzagralopes commented 9 months ago

Hi @Saravana-kr22,

I’m not 100% sure, but I believe that you are suggesting to keep the commissioning between YAML and Python tests. If that’s the case, then I can say that it’s something that we have considered but, since the YAML and Python tests use different controllers, that’s not possible. The YAML tests use the chip-tool and chip-app binaries in interactive mode and the Python test scripts are executed directly, not using this binaries. The ways that the binaries and the Pyhton scripts store the commissioning information are not the same, so the pairing are not shared between them. Maybe we can suppress the need for the user prompt by opening a commissioning window at the end of a suite execution and then performing the new commissioning automatically, but we’re not 100% sure if or how that would work, we’ll have to look into that.

If the suggestion was related to running multiple Python tests one after the other without having to re-commission the device, then that’s what we aimed to do by diving those tests in separate test suites. However, some Python tests expects the DUT to be commissioned and others expect it to not be commissioned and TH needs to know which are which. In order for that to happen, TH expects the scripts to follow a certain template that indicates whether the DUT should be commissioned or not. So unfortunately, until all Python tests are updated to follow this template, we must ask the user which test needs commissioning and which one doesn’t.

ccruzagralopes commented 9 months ago

Also, regarding the timeout error: that can be avoided by reseting the DUT once you see the "Make sure the DUT is in Commissioning Mode" pop-up. Only after it has been reset do you select DONE and submit the answer.

ccruzagralopes commented 9 months ago

@Saravana-kr22 Can you please confirm if you were able to run the tests by reseting the DUT between the test suites (YAML and Python) in the same test run? We'd just like to make sure that there's no bug when doing the current approach of manually resetting the DUT when the pop-up appears.

Saravana-kr22 commented 3 weeks ago

This issue is resolved in latest TH image.