More comprehensive test coverage

[timholy]

Updated Feb 23, 2015.

This would make a whole bunch of good "first contribution" projects, and does not require deep insider knowledge of the language.

The basic idea is to expand the test suite to make sure that julia's base code works as promised. Here is one recommended way to contribute toward this goal:

The easy way

1. Go visit https://coveralls.io/r/JuliaLang/julia. (At the time of this writing, coverage numbers are in the 70s%. If you see numbers that are much lower than this, don't trust the reports; there is likely a problem in measuring coverage.)
2. Browse through the source files and find some untested functionality (highlighted in red) that you think you might be able to write a test for.
3. Write a test that exercises this functionality---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

   The manual method

4. Make sure you have an up-to-date git checkout and are on the master branch. Build julia with make
5. Copy/paste the following script into an editor window and save it. Here I'll use the filename /tmp/coverage_tests.jl:

using Base.Test
testnames = ["core", "keywordargs", "numbers", "strings", "dates",
    "hashing", "remote", "iobuffer", "staged", "arrayops",
    "subarray", "reduce", "reducedim", "random", "intfuncs",
    "simdloop", "blas", "fft", "dsp", "sparse", "bitarray", "copy", "math",
    "functional", "bigint", "sorting", "statistics", "spawn",
    "backtrace", "priorityqueue", "arpack", "file", "suitesparse", "version",
    "pollfd", "mpfr", "broadcast", "complex", "socket",
    "floatapprox", "readdlm", "reflection", "regex", "float16", "combinatorics",
    "sysinfo", "rounding", "ranges", "mod2pi", "euler", "show",
    "lineedit", "replcompletions", "repl", "test", "goto",
    "llvmcall", "grisu", "nullable", "meta", "profile",
    "libgit2", "docs", "base64", "pkg", "linalg1", "linalg2",
    "linalg3", "linalg4", "linalg/lapack", "linalg/triangular", "linalg/tridiag", 
    "linalg/pinv", "linalg/cholmod", "linalg/umfpack", "linalg/givens"
]
for tst in testnames
    println(tst)
    include(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia","test","$tst.jl"))
end

(This is the list of tests currently in test/runtests.jl, with 3 omissions (resolve, reflection, and meta) and one addition (pkg). The omitted tests explicitly test inlining, which we're going to disable, or are problematic when inlining is disabled.)

1. From the top-level directory, type rm usr/lib/julia/sys.so. Deleting sys.so will prevent julia from using any pre-compiled functions, increasing the accuracy of the results. (This also makes startup a bit slower, but that's fine for this test---and once you're done, simply typing make will cause this file to be rebuilt).
2. Via the command prompt, navigate to julia's test/ directory.
3. Start julia with julia --code-coverage=all --inline=no. This turns on code-coverage and prevents inlining (inlining makes it difficult to accurately assess whether a function has been tested).
4. At the julia prompt, type include("/tmp/coverage_tests.jl").
5. Once the tests complete successfully, quit julia
6. Navigate to the base/ directory.
7. Browse through the *.jl.cov files, and look for functions (or branches within functions) that either have 0 in front of them (indicating that they were run 0 times), or have - in front of them (indicating that they were never compiled).
8. Write a test that exercises some function that is not adequately covered---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

[timholy]

@chrisvoncsefalvay, I need to update this page. There's now a much easier way: go visit https://coveralls.io/r/timholy/julia and start browsing through the files. It's built nightly (momentarily by me executing a script, but someday I'll figure out the bug in my cron script...).

[timholy]

Updated, see revised top post.

[chrisvoncsefalvay]

@timholy Ah-hah! Thank you ;)

[timholy]

Ah, thanks for the revision @chrisvoncsefalvay. Updated that, too.

[hayd]

@timholy Is this a line highlighting bug?

Also should some lines (sparingly) by pragma no covered ? e.g. https://coveralls.io/files/422931392#L35 though I guess that specific example is about catching specific error types... #7026.

Edit: another bug? https://coveralls.io/files/422931266#L75

[timholy]

In your first example, which line are you referring to on that page? Number 37?

I've seen occasional oddities myself. One of the most likely explanations is #1334. Also see #9663. Any bug in line numbering is guaranteed to translate into bugs in coverage analysis.

[chrisvoncsefalvay]

Have now started covering (no pun intended) Base.graphics (just thought I'll put it here so we don't accidentally multiply our efforts - we could use a greenie board or a Trello checking who does what so we don't accidentally overlap!)

[timholy]

Good plan!

[chrisvoncsefalvay]

I created a Trello for us to keep stuff coordinated. Happy to add any of you as users, email me your username.

[chrisvoncsefalvay]

@timholy Can you add adding the test to tests/runtests.jl to your original post, so that the new test would be run by the CI suites? Thanks ;)

[timholy]

Done.

[timholy]

Satisfying observation: coverage has gone up every single day since I started regularly measuring it (on Jan 11). The net increase is from 70.4% to 71.6%, but when you consider the size of julia's code base, that's pretty good. At this rate we'll hit 100% :smile: in about 7 months.

And those increases omit early test-coverage commits from @kshyatt and @hayd that occurred before I had set up the necessary infrastructure.

[hayd]

@timholy one annoying thing with coveralls is that it reports 0/0 as 0% e.g. https://coveralls.io/files/433294481 so this will remain first in the sorted list of files.

I wonder if it's worth flagging one of those lines (perhaps module or include) so that it reports instead as 100% (yes, that'll inflate the numbers a little, but it'll also make it easier to find the next least covered file candidate).

80% by Spring. :)

[timholy]

We can't control how coveralls chooses to interpret 0/0; all we do is submit a string containing the code and a single value per line containing either null (javascript analog of nothing) or the number of times that line ran.

One option is that we could just exclude files that have no runnable lines (see https://github.com/IainNZ/Coverage.jl/issues/45). As @garborg correctly pointed out, whatever we choose to do with files that have no runnable lines will have no impact on the final percentage.

Perhaps it would be worth asking them whether such files could be moved to the end of the list, and scored as NA rather than 0%?

[timholy]

Oh, I read your strategy more carefully; we could lie that way, I guess, but I like the idea of either excluding the files or contacting Coveralls.io about this better.

[garborg]

Contacting Coveralls would be great. (I was going to say I was for the change to NA and ambivalent about the move to the end, but for us, at least, it seems like a clear improvement, too.)

I do like keeping the 0/0 files in, though (second best option being doing nothing, imo). They're worth showing to people (at least leaving available, no one has to click on them) -- both because the tooling has TODOS (functions defined via macros, etc.) and may have regressions, and because those files are often filled with types, and the moment someone is scanning through Coveralls results is a moment it's likely to register that a subtype is left out completely or from a chunk of the tests.

[timholy]

https://github.com/lemurheavy/coveralls-public/issues/218

[dhoegh]

@timholy The changes I made in #9485 you merged more than a week ago does not seem to be included in the coverage result in https://coveralls.io/files/440827056. So I think there is something that does not seem to work with the daily updates of the results.

[timholy]

Indeed, sorry about that. I was debugging my cron script and commented out the git pull; make cleanall && make part. Oops. Can you check again?

[dhoegh]

https://github.com/JuliaLang/julia/blob/master/base/replutil.jl#L175 still seems to different from https://coveralls.io/files/440930474#L175

[dhoegh]

When you click at GitHub in coveralls it redirects you to your https://github.com/timholy/julia, which also seems to be behind.

[timholy]

Good debugging. I had convinced myself that it only used the source that we packaged up, but clearly it needs GitHub repo to be updated, too.

This will be a little hard to automate, because of the need to use the SSH key when pushing to https://github.com/timholy/julia; in the short term I'll try to remember to update it daily.

[dhoegh]

Thank you, the results also looks more correct now:) I can see that one code path I introduced in #9485 is untested, I thought I had covered it and but apparently not and I did not check it locally. I am going to submit a pull.

[prcastro]

How can I check if a function has "raised" a Warning?

[timholy]

I'm not sure I understand what you're asking.

[johnmyleswhite]

Warnings aren't thrown (just printed), so you can't check them without monitoring STDOUT. You can check errors with test_throws because errors are thrown.

[prcastro]

Thanks

[dhoegh]

The coveralls test has not run the last two days.

[quinnj]

I believe @timholy is running it manually at night, so let's not beat him up too much :)

On Mon, Feb 9, 2015 at 8:30 AM, Daniel Høegh notifications@github.com wrote:

The coveralls test has not run the last two days.

— Reply to this email directly or view it on GitHub https://github.com/JuliaLang/julia/issues/9493#issuecomment-73527671.

[staticfloat]

My automatic runs are held up by #10027

[timholy]

I appreciate the support @quinnj, but in this case @timholy frankly appreciates that (1) somebody is looking and (2) the same somebody is willing to ping him when something goes wrong.

I assumed you meant that it just wasn't running, so I ran a new one. But looking I now see that the coverage status is just borked. This is probably because of https://github.com/IainNZ/Coverage.jl/issues/48. Let me see what I can do about that.

[catawbasam]

EDIT: I'm no longer seeing this. Now getting the source listing.

On Coveralls TIMHOLY / JULIA / 41, clicking a file name to view source yields the following message:

SOURCE NOT AVAILABLE
The owner of this repo needs to re-authorize with Github; their OAuth credentials are no longer valid so the file cannot be pulled from the Github API.

[timholy]

We seem to be back in business now. Let me know of any further troubles.

[catawbasam]

Coveralls coverage % dropped roughly in half with JULIA / 49. More accurate, or more trouble?

[timholy]

Presumably it's trouble. But it seems to be a base-julia problem.

[timholy]

Update: things are back to working with the local run, but as of today I'm having issues with GnuTLS.

[timholy]

And we're back in business. Up to 74%!

[sbromberger]

@timholy glad to hear. The GnuTLS failure was related to https://github.com/JuliaLang/julia/commit/2af73dd7723c91b0e9fcf5af7662b40c8fb97067.

[timholy]

Looks like @staticfloat has gotten the buildbots working! If you're feeling ready to make that the official version, OK if I update the link above?

(Looks like my days of running things manually can thankfully come to a close....)

[staticfloat]

I still haven't figured out how to get the Git information to propagate properly, but other than that things seem to be working just fine.

[prcastro]

Coveralls isnt's working since march 7th. Anyone know why?

[staticfloat]

There have been a couple bugs that killed things, one of which was https://github.com/jakebolewski/JuliaParser.jl/issues/18 which was just fixed. The latest failure is due to a segfault.

[kshyatt]

I looked at Coveralls today and it seems to be behaving weirdly. For example, let's look at https://coveralls.io/builds/2460415/source?filename=base%2Fstatistics.jl. Coveralls claims this has 0% test coverage for the latest build. Yet, when I actually look in test/, I see that test/statistics.jl is there and has lots of great tests. There are a LOT of files that Coveralls says have no test coverage but have files full of tests in test/. Are they just not being run? It's a bit frustrating for those who want to write more tests if the coverage results are totally wrong.

[hayd]

Has the coverage of Julia dropped to 26%?? https://coveralls.io/r/JuliaLang/julia

(The coverall's page shows 73% at the top level, but 26% on each build.)

[ihnorton]

No; see https://github.com/JuliaLang/julia/issues/10804

[ihnorton]

Now fixed. I'm not sure what magic wand must be waved to regenerate those numbers, but presumably they should go up if @staticfloat's report was the only problem.

[timholy]

Thanks, @ihnorton!

[tkelman]

Better, but still some bugs apparently http://buildbot.e.ip.saba.us:8010/builders/coverage_ubuntu14.04-x64

[kshyatt]

Some more Coveralls builds ran last night and the file coverage stats are still bad. Is there a backlog, or more bug-hunting to do?

[tkelman]

More bug-hunting. Reflection http://buildbot.e.ip.saba.us:8010/builders/coverage_ubuntu14.04-x64/builds/397/steps/Run%20non-inlined%20tests/logs/stdio, and linalg/symmetric http://buildbot.e.ip.saba.us:8010/builders/coverage_ubuntu14.04-x64/builds/397/steps/Run%20inlined%20tests/logs/stdio have issues in the buildbot job that's uploading to coveralls