JuliaDocs / Documenter.jl

A documentation generator for Julia.
https://documenter.juliadocs.org
MIT License
822 stars 479 forks source link

Add number of passed and failed doctests to test summary #1376

Open benzwick opened 4 years ago

benzwick commented 4 years ago

It would be nice to show the number of doctests that passed or failed in the test summary as in other tests, e.g. (https://docs.julialang.org/en/v1/stdlib/Test/#Working-with-Test-Sets-1)

Test Summary: | Pass  Fail  Total
Foo Tests     |    3     1      4
  Animals     |    2            2
  Arrays      |    1     1      2

Currently the output I get is similar to this when all doctests pass:

Test Summary:                             | Pass   Total
  doctest                                 |    1       1
    Doctests: MyPackage                   |    1       1

and this when one or more (in this case two) doctests fail:

Test Summary:                                                          | Pass  Fail  Error  Total
  doctest                                                              |          1             1
    Doctests: MyPackage                                                |          1             1

so it is not possible to see how many doctests have failed. Conversely, I would also like to know how many of my doctests were successful.

The doctest for this example is in runtests.jl as follows:

@time @testset "MyPackage" begin
    @testset "doctest" begin
        Documenter.DocMeta.setdocmeta!(MyPackage, :DocTestSetup, :(using MyPackage); recursive=true)
        Documenter.doctest(MyPackage)
    end
end
fingolfin commented 2 weeks ago

Perhaps this would be resolved (or at least made a bit better) by adding verbose=true to the @testset generated by Documenter.doctest, and perhaps also your one wrapping that?