[Bug] Manual operation in the python testcases need to be implemented in the TH.

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.

https://csa-iot.org/

Apache License 2.0

38 stars 23 forks source link

[Bug] Manual operation in the python testcases need to be implemented in the TH. #163

Closed Saravana-kr22 closed 1 month ago

Saravana-kr22 commented 8 months ago

Describe the bug

The SDK python testcases which has the input() are executing for long time without an end.

For exapmle: while running the TC-RVCRUNM-2.1 , The testcase is neither passed or failed it is in execution for long time.

Expected behavior

The UI prompt needs to be added for these testcases in their respective test-steps to perform the manual operation in the DUT, and then continue the execution.

rquidute commented 8 months ago

HI @cecille, It seems this was related what was discussed in the last meeting

Saravana-kr22 commented 7 months ago

As per the lastest TH version: Version: 2.10-beta3+spring2024 Sha: 1fd1429 , Below are the list of test cases that requires this implemntation to validate in the TH.

These testcases are execpting to provide input to the script. but TH didn't have any method to provide input to these scripts

[ ] TC-OPSTATE-2.1
[ ] TC-OPSTATE-2.2
[ ] TC-OPSTATE-2.3
[ ] TC-OPSTATE-2.4
[ ] TC-OPSTATE-2.5
[ ] TC-OVENOPSTATE-2.1
[ ] TC-OVENOPSTATE-2.2
[ ] TC-OVENOPSTATE-2.3
[ ] TC-OVENOPSTATE-2.4
[ ] TC-OVENOPSTATE-2.5
[ ] TC-RVCCLEANM-2.1
[ ] TC-RVCCLEANM-2.2
[ ] TC-RVCOPSTATE-2.1
[ ] TC-RVCOPSTATE-2.3
[ ] TC-RVCOPSTATE-2.4
[ ] TC-RVCRUNM-2.1
[ ] TC-RVCRUNM-2.2

ccruzagralopes commented 7 months ago

Hi @Saravana-kr22, @cecille,

We've just opened 2 PRs to handle this issue:

TH backend PR: https://github.com/project-chip/certification-tool-backend/pull/73
SDK PR: https://github.com/project-chip/connectedhomeip/pull/32299

These PRs add the support for getting the input, but the python tests still need to be updated. The SDK PR updates one of the tests as an example.

Ashwinigrl commented 2 months ago

@cecille Still this issue is not resolved in TH Version: v2.11-beta2+fall2024 Sha: bbeda31.

For Example: TC-RVCRUNM-2.2 Is still failing. TC-RVCRUNM-2.2.log

hiltonlima commented 1 month ago

Hello @Ashwinigrl ,

Running here the TC-RVCRUNM-2.2 we are able to see the prompt request on UI. Please see the image attached

Is the error you are seeing related to the prompt display? Or is it an error in the normal test execution?

If it is the prompt display, could you please share both the project config and the pics files you are using so we can replicate it here?

Ashwinigrl commented 1 month ago

@hiltonlima Re verified the Test case in TH Version: v2.11-beta2+fall2024 Sha: bbeda31 I am able to see the prompt request on UI. Hence closing the issue. Thanks!