Show Test Report Summary in CI

exercism / csharp

Exercism exercises in C#.

https://exercism.org/tracks/csharp

MIT License

344 stars 346 forks source link

Show Test Report Summary in CI #2222

Closed putquo closed 8 months ago

putquo commented 8 months ago

Description

This PR aims to resolve #1949.

Changes

Added test-summary/action to the Test workflow
Added JunitXml.TestLogger as a dependency to all existing exercises
Added JunitXml.TestLogger as a dependency to new exercises via the add-new-exercise.ps1 script
Updated the test.ps1 script to use the test logger when the CI environment variable evaluates to true

Notes

Since the tests do not specify a namespace, the logger defaults to UnknownNamespace. Here is an example of what this looks like.

github-actions[bot] commented 8 months ago

Hello. Thanks for opening a PR on Exercism 🙂

We ask that all changes to Exercism are discussed on our Community Forum before being opened on GitHub. To enforce this, we automatically close all PRs that are submitted. That doesn't mean your PR is rejected but that we want the initial discussion about it to happen on our forum where a wide range of key contributors across the Exercism ecosystem can weigh in.

You can use this link to copy this into a new topic on the forum. If we decide the PR is appropriate, we'll reopen it and continue with it, so please don't delete your local branch.

If you're interested in learning more about this auto-responder, please read this blog post.

Note: If this PR has been pre-approved, please link back to this PR on the forum thread and a maintainer or staff member will reopen it.

putquo commented 8 months ago

Thanks for working on this! I'd like to suggest a slightly different approach. Right now, students will have to download the JunitXml.TestLogger package even if they're just working locally, which doesn't really offer them anything. What I would suggest to do instead is to modify the test.ps1 script to dynamically add the package (via dotnet add package JunitXml.TestLogger) when it detects that it runs in CI.

Good point! I have made the change as you suggested. This does increase the run time of the workflow quite a bit (if that is of any concern), but I managed to shave off about a couple of minutes with the --no-restore option since it will run during dotnet test anyway. Maybe you have some ideas if you think it may be problematic?

ErikSchierboom commented 8 months ago

Maybe you have some ideas if you think it may be problematic?

I think this is fine!

ErikSchierboom commented 8 months ago

Maybe a silly question, but where can I see the summary?

putquo commented 8 months ago

Maybe a silly question, but where can I see the summary?

I wondered that at first too :) but you can view it over here. In general, you’ll need to go to the “summary” section of a workflow run (top left corner on web) and scroll to the bottom of the page to see it. It seems like you also have to be signed in to see it.

ErikSchierboom commented 8 months ago

Right! It's unimpressive right now because everything is passing, but reading up on the docs, it will be very useful when tests are failing:

show: the test results to summarize in a table (optional) This controls whether a test summary table is created or not, as well as what tests are included. It could be all, none, pass, skip, or fail. The default is fail - that is, the summary table will only show the failed tests. For example, if you wanted to show failed and skipped tests:

Right now finding failing tests involves some annoying text searching.

ErikSchierboom commented 8 months ago

Thanks a lot! If you care for it, you'd be welcome to do the same thing for exercism/vbnet or exercism/fsharp.