project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
40 stars 23 forks source link

[Bug] Test steps are not being shows for all python tests #457

Open cecille opened 3 weeks ago

cecille commented 3 weeks ago

Describe the bug

There are a bunch of tests where we have steps_ functions defined, but the individual steps are not displayed in the GUI

Steps to reproduce the behavior

This happens for opstate tests for example, but I assume also others.

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

fabiowmm commented 2 weeks ago

@antonio-amjr can you try to reproduce this with TC_OPSTATE_2_1.py?

antonio-amjr commented 2 weeks ago

@antonio-amjr can you try to reproduce this with TC_OPSTATE_2_1.py?

Sure, I'll take a look

antonio-amjr commented 2 weeks ago

@cecille and @fabiowmm ,

The OPSTATE and OVENOPSTATE tests are falling into the legacy python test category. That is happening because the Python Test Parser is expecting a ast.List type from the "steps_" method, but instead is receiving a ast.Call.

So the problem is that the OPSTATE/OVENOPSTATE tests are the only ones, I believe, that are using another method to retrieve the steps, like shown below:

Screenshot 2024-10-11 at 09 15 25
The python_test_parser.py method verifying the test steps type.

Screenshot 2024-10-11 at 09 11 58
The original TC_OPSTATE_2_1 steps method that triggers the error.

Screenshot 2024-10-11 at 09 22 48
Changing the TC_OPSTATE_2_1 steps method successfully adds to the correct suite and show the steps in execution.


I think that the steps are specific to tests and shouldn't be inside the TC_OpstateCommon.py at all. And to solve this would be just moving the steps to each related file.

What is your take on this?

antonio-amjr commented 2 weeks ago

[!NOTE]
It seems that there's a feature already in progress that solves this issue: