Show list of errors from testing farm

[abitrolly]

Description

When build fails, there is are no error, only logs. GitHub and GitLab are able to parse error logs, and provide info with links that directly show what happened. With packit even reaching the logs is like 4 clicks away and then scrolling scrolling scrolling.

For example, this test failure in tmt.

1. https://github.com/teemtee/tmt/pull/3039#issuecomment-2205966416

See logs in various failed checks.

:sob:

2. Parse statuses, click red one, go https://github.com/teemtee/tmt/pull/3039/checks?check_run_id=26991071043
3. Parse table, go https://dashboard.packit.dev/results/testing-farm/515717
4. Parse page, go to build results

5. See everything is green https://dashboard.packit.dev/results/copr-builds/1696226

WHAT???

6. Go back an click red oval
7. Finally land on https://artifacts.dev.testing-farm.io/42e7b5c6-c720-4fcf-b0d9-043b66ee6325/#artifacts-/plans/features/core

8. fail with (1 passed, 0 failed, 0 error)

WHAT???

Now the only way to find out the error is to read through all the logs in these tiny scrolls.

Benefit

1. Save precious developer's time.
2. Reduce cognitive load from visual log parsing, and thus make more active brain cells available for the rest of the day.

Importance

Very important.

Workaround

• [ ] There is an existing workaround that can be used until this feature is implemented.

Participation

• [X] I am willing to submit a pull request for this issue. (Packit team is happy to help!)

[lachmanfrantisek]

Hi @abitrolly !

Sorry for not getting to this sooner. You are touching multiple coupled (and relevant) points here so I'll try to split this into multiple (hopefully doable) tasks:

1. Clicks on GitHub

Sadly, we can't avoid the extra click to get from the pull request to the Check Run UI (this is a design of GitHub). But, we can use this markdown view to provide more information, but, this is a GitHub-specific feature and we try to share as much as possible => not saying we can't put more info there but we don't want to put all the info from the dashboard into this markdown.

2. Quickly get the summary of the error

@mfocko mentioned we might be able to provide GitHub with a standardised form of test results. (We need to check this.) Also, we can try to get this from TF and use it either in a status name (a little space) or in GitHub Check Run Markdown (more space, but GitHub specific as mentioned above) and/or in the Packit's dashboard.

3. Navigation through the dashboard

Definitely can be improved -- I can see a few small changes that might help here:

• Providing more of a pipeline-centric view on the tasks (WIP).
• When showing Copr builds in the test result page on the Packit dashboard, provide the build's overall status to ensure the build succeeded. Showing it green should always be the case since the tests are run only when the build is finished (if we don't skip the builds entirely).
• Make it clear that the red oval is meant to be used to get to the logs. (At least a tooltip. Maybe some log-like icon? Not sure now but definitely improbable.)

4. TF result view

Can definitely be improved, but is more on the TF developers. This specific occurrence of "failure" is very confusing. For me as well...;) We can ask on their issue tracker to improve this.

One thing @Venefilyn is trying to achieve is to have one shared dashboard for all the related tools (basically to have one dashboard both for Packit and TF). This is still not clear how to approach (and sustainably manage) but something we are thinking about.

---

I hope I haven't forgotten about any crucial issue -- please, let me know what you think about these and we can create a separate task for each item. Thanks for providing us with the whole story of you going through this! This is really helpful since we are a bit biased.

[abitrolly]

Thanks for the reply. The "first experience" UI/UX issues are still actual. Now that I've got some answers, I've also a bit biased, so let me concentrate on the problem I am trying to solve right now.

4. TF result view

Can definitely be improved, but is more on the TF developers. This specific occurrence of "failure" is very confusing. For me as well...;) We can ask on their issue tracker to improve this.

I asked here https://gitlab.com/testing-farm/oculus/-/issues/24 but the Web UI only parses results.xml (https://gitlab.com/testing-farm/oculus/-/merge_requests/65/diffs) that is provided by TMT (I guess), and the results.xml from TMT just doesn't provide sufficient details.

In https://github.com/teemtee/tmt/pull/3039#issuecomment-2221169944 we've traced the incomplete results.xml to beakerlib test result processor. There are still some missing pieces connecting results.xml to JUnit format. Web UI lists both, but it is not clear if it uses both.

One thing @Venefilyn is trying to achieve is to have one shared dashboard for all the related tools (basically to have one dashboard both for Packit and TF). This is still not clear how to approach (and sustainably manage) but something we are thinking about.

I like static web app approach that TF Web UI (https://gitlab.com/testing-farm/oculus/-/merge_requests/64/diffs) is using.

If the results.xml format is documented and well engineered, people could use that to also render GitLab etc. Maybe it is impossible to have a perfect dashboard for everything, but it definitely possible to make a dashboard that could be customized for specific workflow.

In conclusion I must absolutely add that

graph LR
  this --> needs --> diagrams

[lachmanfrantisek]

Thanks for all the info, @abitrolly ! And thanks for your work on the related tools -- this shows quite well how is the current state a bit misleading to the user when trying to get to the responsible service...;)

Just a small update that we are starting a small research about the shared dashboard. (How do people want to use it, and what information do they need.) I am linking this issue since it has a couple of interesting points. If you are interested, we can even include you in our interview round. (But issue(s) works fine as well, so no push...;)