exercism / julia-test-runner

GNU Affero General Public License v3.0
2 stars 8 forks source link

adding verbose testing #111

Closed depial closed 6 months ago

depial commented 6 months ago

This PR aims to do two things:

  1. Add a test case with the new runtests.jl structure, as requested here. In this aim, I've added a new folder with two files (verbose_testing/expected_results.json and verbose_testing/runtests.jl).
  2. Modify the way test names are displayed on the website by excluding the top-level testset description when testing uses the new verbose format.

I haven't contributed to this part of the project yet, so please review carefully.

github-actions[bot] commented 6 months ago

Hello. Thanks for opening a PR on Exercism 🙂

We ask that all changes to Exercism are discussed on our Community Forum before being opened on GitHub. To enforce this, we automatically close all PRs that are submitted. That doesn't mean your PR is rejected but that we want the initial discussion about it to happen on our forum where a wide range of key contributors across the Exercism ecosystem can weigh in.

You can use this link to copy this into a new topic on the forum. If we decide the PR is appropriate, we'll reopen it and continue with it, so please don't delete your local branch.

If you're interested in learning more about this auto-responder, please read this blog post.


Note: If this PR has been pre-approved, please link back to this PR on the forum thread and a maintainer or staff member will reopen it.

ErikSchierboom commented 6 months ago

This is looking great! There is one issue though where the Dockerfile isn't built correctly. Could you look at https://github.com/exercism/julia-test-runner/actions/runs/8843674200/job/24284240727?pr=111?

depial commented 6 months ago

I found the first error (line 1046) was because the struct ReportingTestSet did not have a verbose field, so I've tried to define this field and adjust the other methods to work in concert. Hopefully this resolves the issue.

As an aside: at present, any time verbose = true in testing, the test name will be suppressed in output on the website (i.e. even if/when verbose = true is used at lower level testsets). For example:

@testset verbose = true "tests" begin
    @testset verbose = true "first real testset" begin
        @test 1 ==1
    end
end

Would not show first real testset in the test name on the website. However,

@testset verbose = true "tests" begin
    @testset "first real testset" begin
        @test 1 ==1
    end
end

will show first real testset in the name on the website. In other words, we will likely have to enforce the use of only the wrapper testset in the new format having verbose = true.

This side effect seems like a reasonable compromise to still be able to accept both formats, since nobody has used this in other tests before and it could be caught in the review process for new exercises if someone misused verbose = true. However, let me know if this is unacceptable.

The only other option I can think of is to use the standardized testset name tests as an additional condition to suppress testset name output (which has its own drawbacks). This would allow the first example above to show first real testset in the name, but then the standard naming of the wrapper testset would have to be enforced.

ErikSchierboom commented 6 months ago

Just needed a little tweaking: https://github.com/exercism/julia-test-runner/pull/111/commits/f27c85feca4bb80d10086a8321f4d2ea73f05608