search

project-chip / certification-tool

A test harness and tooling designed to simplify development, testing, and certification for devices, guided by the Connectivity Standards Alliance.
https://csa-iot.org/
Apache License 2.0
42 stars 24 forks source link

[Bug] [Frontend] TH Test run UI TC mismatch #292

Open rquidute opened 5 months ago

rquidute commented 5 months ago

Describe the bug

The test run UI in the left panel shows a Test Case that is not the same that is running according to log panel in the right Screenshot 2024-06-20 at 11 03 32

Important: TH UI shows the tests in the left panel according to the order presented at test selection screen, but backend returns a specific test list order in the following endpoint responses: POST /api/v1/test_run_executions/ and POST /api/v1/test_run_executions/{id}/start.

In addition, the endpoint /{id}/applicable_test_cases, is also returning the test cases in a different order than the displayed in Test selection screen.

Steps to reproduce the behavior

Expected behavior

Test Run UI in the left panel must reflect the log panel in the right, so it should consider the backend response for POST /api/v1/test_run_executions/{id}/start and also POST /api/v1/test_run_executions/{id}/repeat

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response

raul-marquez-csa commented 2 months ago

@rquidute

From trying to reproduce the issue I noticed that the scenario in the screenshot can happen if we expand one of the upcoming tests (tests not in execution yet) making it appear like a mismatch.

You can notice from the screenshot that the left hand side isn't displaying a test starting or in execution (there are not icons indicating status or progress).

When there's a test under execution it shows the progress icons to the left.

Might this be a false positive?

If not can you please provide a short video or more details showing the behavior described?

Or might this be a regression? was this not happening on earlier versions of the TH?

rquidute commented 2 months ago

Hi @raul-marquez-csa, We are right, the test under execution is not displayed in the screenshot and there is only one progress icons that is present for the current test in progress. I think, since the first test displayed in the left panel is not the first to be executed, might lead user to think it is the wrong test case. The mandatories test cases should be the first ones to be executed.

I'm attaching a video where you can see the whole process. Please also notice the screen right before the test execution (The screen where you are able to select the test cases), the right panel shows the test case in the same order as the execution screen.

https://github.com/user-attachments/assets/7b3b407c-58d5-413f-9dc3-6a10a22a8d0d

raul-marquez-csa commented 2 months ago

PRs in progress Frontend Backend

raul-marquez-csa commented 3 weeks ago

Blocked for now

High complexity, many moving parts, and potential major re-design/refactor of both frontend and backend, deeper discussion needed.

I see no easy or straightforward way to fix this, latest progress and attempts are in my previous comment (PR links).

Any other approach or pointers in the correct direction appreciated.

antonio-amjr commented 1 week ago

Hi @raul-marquez-csa,

I can try to help you on this issue. I'll take a look on the PRs you've created and if I can think on another approach.

Can you give a summary of complexity you discovered?