More comprehensive test coverage

[timholy]

Updated Feb 23, 2015.

This would make a whole bunch of good "first contribution" projects, and does not require deep insider knowledge of the language.

The basic idea is to expand the test suite to make sure that julia's base code works as promised. Here is one recommended way to contribute toward this goal:

The easy way

1. Go visit https://coveralls.io/r/JuliaLang/julia. (At the time of this writing, coverage numbers are in the 70s%. If you see numbers that are much lower than this, don't trust the reports; there is likely a problem in measuring coverage.)
2. Browse through the source files and find some untested functionality (highlighted in red) that you think you might be able to write a test for.
3. Write a test that exercises this functionality---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

   The manual method

4. Make sure you have an up-to-date git checkout and are on the master branch. Build julia with make
5. Copy/paste the following script into an editor window and save it. Here I'll use the filename /tmp/coverage_tests.jl:

using Base.Test
testnames = ["core", "keywordargs", "numbers", "strings", "dates",
    "hashing", "remote", "iobuffer", "staged", "arrayops",
    "subarray", "reduce", "reducedim", "random", "intfuncs",
    "simdloop", "blas", "fft", "dsp", "sparse", "bitarray", "copy", "math",
    "functional", "bigint", "sorting", "statistics", "spawn",
    "backtrace", "priorityqueue", "arpack", "file", "suitesparse", "version",
    "pollfd", "mpfr", "broadcast", "complex", "socket",
    "floatapprox", "readdlm", "reflection", "regex", "float16", "combinatorics",
    "sysinfo", "rounding", "ranges", "mod2pi", "euler", "show",
    "lineedit", "replcompletions", "repl", "test", "goto",
    "llvmcall", "grisu", "nullable", "meta", "profile",
    "libgit2", "docs", "base64", "pkg", "linalg1", "linalg2",
    "linalg3", "linalg4", "linalg/lapack", "linalg/triangular", "linalg/tridiag", 
    "linalg/pinv", "linalg/cholmod", "linalg/umfpack", "linalg/givens"
]
for tst in testnames
    println(tst)
    include(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia","test","$tst.jl"))
end

(This is the list of tests currently in test/runtests.jl, with 3 omissions (resolve, reflection, and meta) and one addition (pkg). The omitted tests explicitly test inlining, which we're going to disable, or are problematic when inlining is disabled.)

1. From the top-level directory, type rm usr/lib/julia/sys.so. Deleting sys.so will prevent julia from using any pre-compiled functions, increasing the accuracy of the results. (This also makes startup a bit slower, but that's fine for this test---and once you're done, simply typing make will cause this file to be rebuilt).
2. Via the command prompt, navigate to julia's test/ directory.
3. Start julia with julia --code-coverage=all --inline=no. This turns on code-coverage and prevents inlining (inlining makes it difficult to accurately assess whether a function has been tested).
4. At the julia prompt, type include("/tmp/coverage_tests.jl").
5. Once the tests complete successfully, quit julia
6. Navigate to the base/ directory.
7. Browse through the *.jl.cov files, and look for functions (or branches within functions) that either have 0 in front of them (indicating that they were run 0 times), or have - in front of them (indicating that they were never compiled).
8. Write a test that exercises some function that is not adequately covered---you can add your test to one of the existing files, or start a new one, whichever seems most appropriate to you. If you're adding a new test file, make sure you include it in the list of tests in test/runtests.jl. http://julia.readthedocs.org/en/latest/stdlib/test/ may be helpful in explaining how the testing infrastructure works. Submit the test as a pull request (see CONTRIBUTING.md).

[JeffBezanson]

:100:

[sbromberger]

This sounds like a fantastic idea, but I think I'm missing something: I fire up julia* from test/ with --code-coverage=all and follow the instructions (the tests all appear to run correctly; that is, I get SUCCESS at the end), but then when I go to base/, I don't see any .cov files.

*I've tried using /usr/local/bin/julia which was installed via make install, as well as the one in the source tree under usr/bin.

julia> VERSION
v"0.4.0-dev+2251"

[IainNZ]

I just did it, and it worked for me. Are you definitely using the julia you are building from source?

[sbromberger]

Pretty sure. See https://gist.github.com/sbromberger/282ed8d69695b34d4279 for the log.

ETA: I do see a runtests.jl.cov in test/:

-rw-r--r--  1 seth  staff  2675 Dec 30 01:51 runtests.jl.cov

[IainNZ]

Why are you doing ../usr/bin/julia, or is that symlinked to /Users/seth/dev/julia/julia/julia?

Anyway, I thought I'd try to give this a go, but I'm not getting sensible results from the coverage. For example, I got that intersect and setdiff weren't run - sure enough they're in the tests. I added some printlns just to be sure, and they do get run, but I see

        - function intersect(s::Set, sets::Set...)
        1     println(:intersect, s, sets)
        0     i = similar(s)
        0     for x in s
        0         inall = true
        0         for t in sets
        0             if !in(x,t)
        0                 inall = false
        0                 break
        -             end
        -         end
        0         inall && push!(i, x)
        -     end
        0     return i
        - end

which is even harder to explain ¯\_(ツ)_/¯.

[sbromberger]

Why are you doing ../usr/bin/julia, or is that symlinked to /Users/seth/dev/julia/julia/julia?

Because that's where the compiled julia (pre-make install) lives. There is no /Users/seth/dev/julia/julia/julia. It's

-rwxr-xr-x  1 seth  staff  20404 Dec 24 01:19 /Users/seth/dev/julia/julia/usr/bin/julia*

(On my system, $SRCROOT/julia was a broken symlink to $SRCROOT/usr/bin/julia-debug, which broke when I moved the directory. Even after fixing the broken symlink, I did not get the expected results.)

[sbromberger]

OK, mystery solved. Because I moved the source root after the build, I have broken symlinks all over the place. Specifically, $SRCROOT/usr/share/julia/base is a symlink that is not relative (that is, it points to a full path starting with /), which ties it to the filesystem at the time of the build. Since I moved the source root, that absolute symlink now points to an invalid directory, which (I guess) forces $SRCROOT/usr/bin/julia to try to find some other base/ (though I don't know where).

Why the symlinks within the build directory are absolute-pathed (at least on OSX) is a mystery to me. Far better, instead of having

lrwxr-xr-x  1 seth  staff  26 Dec 24 00:43 base@ -> /Users/seth/dev/julia/base

in $SRCROOT/usr/share/julia, we should have

lrwxr-xr-x  1 seth  staff  26 Dec 24 00:43 base@ -> ../../../base

so that the build directory is independent of the overall filesystem and the source root (/Users/seth/dev/julia) can be moved without problem.

[waldyrious]

@timholy, step 7 appears to be less detailed than the other ones. Perhaps it should at least link to http://julia.readthedocs.org/en/latest/stdlib/test/ for those new to unit testing. (The rest of the list is crystal clear, thanks for doing that!)

Additionally, CONTRIBUTING.md mentions "Make sure you test your code as described here", but the "here" is kinda ambiguous. Does it means the list that follows shortly afterwards, or was it meant to be a link, perhaps to the docs I pointed to above?

[timholy]

@IainNZ: wow. OK, I'll take a peek and see if I can figure out what's happening.

@waldyrious: very good suggestions. I've edited the post up top to include your link, and will look into editing CONTRIBUTING.md.

[timholy]

@IainNZ, the following test script works for me:

using Base.Test
s1 = Set(1,2,3)
s2 = Set(2,5,7)
s = intersect(s1, s2)
@test s == Set(2)

I get useful coverage results with or without #9354 and --inline=no:

        - function intersect(s::Set, sets::Set...)
        1     i = similar(s)
        1     for x in s
        3         inall = true
        3         for t in sets
        3             if !in(x,t)
        2                 inall = false
        2                 break
        -             end
        -         end
        3         inall && push!(i, x)
        -     end
        1     return i
        - end

Does that work for you? If not, what platform and commit are you on?

[svaksha]

Hi, @timholy, I ran the tests for commit: 52193154ec1abc60f2bf6736d38665aab286b306, which threw this error:

julia> include("runtests.jl") From worker 3: * linalg2 From worker 2: * linalg1 From worker 2: * linalg3 From worker 3: * linalg4 From worker 2: * linalg/lapack From worker 2: * linalg/triangular exception on 3: ERROR: test failed: transpose(qrfact(randn(3,3))) did not throw ErrorException in expression: transpose(qrfact(randn(3,3))) in error at error.jl:21 in default_handler at test.jl:27 in do_test_throws at test.jl:69 in runtests at /home/mom/julia/test/testdefs.jl:5 in anonymous at multi.jl:852 in run_work_thunk at multi.jl:603 in anonymous at task.jl:852 while loading linalg4.jl, in expression starting on line 205 ERROR: test failed: transpose(qrfact(randn(3,3))) did not throw ErrorException in expression: transpose(qrfact(randn(3,3))) in anonymous at task.jl:1616 while loading linalg4.jl, in expression starting on line 205 while loading /home/mom/julia/test/runtests.jl, in expression starting on line 42

I had another question regarding running these tests: Currently, for personal devel stuff I prefer to use the nightly julia build from Elliot/staticfloat for Ubuntu-14.04-LTS. But, for the test coverage, is it kosher to use and build julia in the /home/user/julia directory? I figured that since the nightlies run off a different space, this arrangement will give me a sandbox (not exactly, but then julia does not have an Anaconda / Canopy/ virtualenv env like python does) space. What are the drawbacks of doing this, if any? TIA.

[IainNZ]

@timholy everything seems to be working fine, not sure what was happening before. Woops!

[timholy]

@svaksha, thanks for giving this a spin. Does make test-linalg4 fail reliably for you?

is it kosher to use and build julia in the /home/user/julia directory?

I am not anything remotely resembling a build guru, so don't listen to me :smile:. I'm not even sure where @staticfloat's binary installs. But, given those caveats: it should be OK to run code_coverage even from a binary install. However, when you add to files in test/, it's better to use a git checkout, just to make it easier to contribute. I don't see any problem with having two julia installs---I have one 0.4 and one 0.3 on my machine, and except for history and some package symlinks I've set up deliberately, the two don't interact at all.

[timholy]

@IainNZ, very glad to hear that! It is weird that it was giving you something so strange; if you can replicate, please do file an issue.

[staticfloat]

@svaksha I routinely keep a "stable" version installed via package manager, (whether it be Homebrew on OSX, or aptitude on Ubuntu) and an "unstable" version just sitting in ~/src/julia. Absolute paths shouldn't make a difference here, just be sure to invoke the proper julia executable when playing around in the unstable julia directory. (I can't tell you how many times I've scratched my head before realizing that I needed to ./julia to use the unstable version, rather than just julia to use the stable version)

[svaksha]

On Wed, Dec 31, 2014 at 12:56 AM, Tim Holy notifications@github.com wrote:

@svaksha, thanks for giving this a spin. Does make test-linalg4 fail reliably for you?

Hi and Thanks for the reply @timholy! If "fail reliably" means it ran successfully, then it did (with a small linker warning) : https://gist.github.com/svaksha/aa25f53bf540e3fd52d6

is it kosher to use and build julia in the /home/user/julia directory?

I am not anything remotely resembling a build guru, so don't listen to me . I'm not even sure where @staticfloat's binary installs.

Hah, same here - my ppa install gets called via /etc/apt/sources.list.d is the limit of my interest in system level internals, unless an OS issue crops up breaking stuff that I have to fix so that I can get back to work :-)

However, when you add to files in test/, it's better to use a git checkout, just to make it easier to contribute.

Good point - will keep this in mind.

I don't see any problem with having two julia installs---I have one 0.4 and one 0.3 on my machine, and except for history and some package symlinks I've set up deliberately, the two don't interact at all.

True, but I'm a huge fan of sandboxed testing/development which makes it easier to catch the bug source. </thinking out loud>.

PS: @timholy, Would you mind sharing (in a blog post or a gist) how you installed, maintain and update the two julia installs? I fail at this for code that was written for older julia versions.

Thanks, SVAKSHA ॥ http://about.me/svaksha ॥

[svaksha]

Hi and Thanks for the reply @staticfloat.

On Wed, Dec 31, 2014 at 1:00 AM, Elliot Saba notifications@github.com wrote:

@svaksha I routinely keep a "stable" version installed via package manager, (whether it be Homebrew on OSX, or aptitude on Ubuntu) and an "unstable" version just sitting in ~/src/julia.

Got that. What isnt clear is how (if) the bash modifications affect the 'unstable' version on Ubuntu-LTS.

Absolute paths shouldn't make a difference here, just be sure to invoke the proper julia executable when playing around in the unstable julia directory. (I can't tell you how many times I've scratched my head before realizing that I needed to ./julia to use the unstable version, rather than just julia to use the stable version)

Could you elaborate a little more on what you mean by needing "./julia to use the unstable version, rather than just julia to use the stable version"? I'm not sure I understand this entirely. This may not be entirely relevant to this testing thread so feel free to reply in a gist/email.

On an unrelated note, I recall seeing a thread about running Julia within Anaconda - are you involved with that or do you happen to know its progress?

@staticfloat, Also wanted to thank you for the nightlies - they make life so simple that I've forgotten the harsh world of compile failures :-). Just last week I was caught trying to figure out why the latest git co was failing and yet showing a 3-day old version on the REPL.

Thanks, SVAKSHA ॥ http://about.me/svaksha ॥

[tkelman]

On an unrelated note, I recall seeing a thread about running Julia within Anaconda - are you involved with that or do you happen to know its progress?

I don't know that anyone has specifically tried this. It's been mentioned a few times, such as https://github.com/JuliaLang/julia/issues/4853#issuecomment-29118252 or a few times in relation to making the installation of IJulia easier, but I don't know if anyone has done any work in this direction. Maybe some people more familiar with the SciPy packaging ecosystem would know if it's happened yet from that side. I did once try using their BinStar binary packaging/building service, but it was (is?) very young and not too well documented and I couldn't figure it out.

[staticfloat]

What I mean is this: My setup is to have a stable version in /usr/local/bin (that's my Homebrew installation of OSX, using the Homebrew tap) or /usr/bin (that's the apt installation on Ubuntu using the stable PPA). Those directories are both listed in the default PATH environment variable, so if I type julia at the command prompt, it will look in the folders that are on my PATH, and when it finds a julia executable, it will run that. However, if I type ./julia, it only will execute a file called julia in the current directory. (I'm providing an unambiguous path to a file in this case; . represents the current directory, so ./julia means "the file named julia in the current directory")

My setup is most useful if you do most of your work in a stable version of julia, but want to try something out on master every now and then. Running julia anywhere will always start the stable version of Julia, but running, for instance, ~/src/julia/julia will run the Julia executable sitting in ~/src/julia, which happens to be the master checkout. (Alternatively, if you're already sitting in ~/src/julia, you can just run ./julia)

Does that make more sense?

On an unrelated note, I recall seeing a thread about running Julia within Anaconda - are you involved with that or do you happen to know its progress?

I haven't heard anything about that recently, so if people are working on that right now, our paths haven't crossed.

I'm a huge fan of sandboxed testing/development which makes it easier to catch the bug source.

I agree. Julia (and I'm only talking about things in this git repository; packages are another story) doesn't touch anything outside of ~/julia/X.Y, where X.Y is the major/minor version of julia. Two versions of Julia can coeexist very peacefully, and if you'd like to have control over where your Julia stores its package state, you can do so via the JULIA_PKGDIR environment variable. This allows for a (somewhat labor-intensive) method of pretending like we have a virtualenv for Julia; you can put the Julia executable, and all its packages in a directory, and then just call Julia with the correct environment variables setup to have your own little packaged, isolated Julia installation.

Packages can do whatever they want, however. For instance, every time I run Pkg.build() on 0.3, the IJulia.jl package sets up an iPython notebook profile for IJulia, which gets overwritten every time I run Pkg.build() on 0.4. So those two fight every now and then, but that's something that is more of a problem in IPython (and which they are addressing in IPython 3.0 I believe) than it is a problem in Julia.

[svaksha]

On Wed, Dec 31, 2014 at 5:56 AM, Elliot Saba notifications@github.com wrote:

My setup is most useful if you do most of your work in a stable version of julia, but want to try something out on master every now and then. Running julia anywhere will always start the stable version of Julia, but running, for instance, ~/src/julia/julia will run the Julia executable sitting in ~/src/julia, which happens to be the master checkout. (Alternatively, if you're already sitting in ~/src/julia, you can just run ./julia)

Does that make more sense?

Thanks for explaining that. Its what I'd like to (and was trying to) do but for some reason, its not working as I imagined it would. In /home/user/julia, when I try Tim's instructions it complains about being unable to find the directory:

~/julia/test$ ./julia --code-coverage=all
bash: ./julia: No such file or directory

If I run the above command without the ./ it runs off your nightlies, which isnt what I want. I'd like to keep the packaged nightlies disjunct from the git master, the latter being the sandbox to break stuff.

I agree. Julia (and I'm only talking about things in this git repository; packages are another story) doesn't touch anything outside of ~/julia/X.Y, where X.Y is the major/minor version of julia. Two versions of Julia can coeexist very peacefully, and if you'd like to have control over where your Julia stores its package state, you can do so via the JULIA_PKGDIR environment variable. This allows for a (somewhat labor-intensive) method of pretending like we have a virtualenv for Julia; you can put the Julia executable, and all its packages in a directory, and then just call Julia with the correct environment variables setup to have your own little packaged, isolated Julia installation.

Is there any documentation on how one can setup such an isolated environment? I grepped http://docs.julialang.org/en/latest/ for JULIA_PKGDIR but it does not explain how to setup this env.

SVAKSHA ॥ http://about.me/svaksha ॥

[svaksha]

On Wed, Dec 31, 2014 at 5:38 AM, Tony Kelman notifications@github.com wrote:

I don't know that anyone has specifically tried this. It's been mentioned a few times, such as #4853 (comment) or a few times in relation to making the installation of IJulia easier, but I don't know if anyone has done any work in this direction. Maybe some people more familiar with the SciPy packaging ecosystem would know if it's happened yet from that side. I did once try using their BinStar binary packaging/building service, but it was (is?) very young and not too well documented and I couldn't figure it out.

Thanks for the reply. Found the SO thread: http://stackoverflow.com/questions/25373577/how-to-install-julia-in-an-anaconda-environment I prefer the Anaconda env's more (than say, virtualenv) as it acts like an OS inside an OS (for lack of a better term) for Python, making it easier to sanitize the development work. As for pure julia development, the current method of running off other directories with a path defined works well but while developing for a mixed codebase it would be nicer if Julia would run off the Anaconda env too.

SVAKSHA ॥ http://about.me/svaksha ॥

[timholy]

@svaksha, from inside test/ just say ../julia --code-coverage=all. Note there are two dots in front of julia, so you go up a directory.

What I do is have julia refer to master, and julia-old be a symlink (on my PATH) to a v0.3 checkout. Most users will want to flip that, and make julia be for stable and julia-next for master. As long as both are on your PATH, you're pretty much done.

[timholy]

@svaksha, I remembered your linalg4 issue; that gist you posted showed success, so clearly it's not "failing reliably." You might want to try a second time running the whole test suite as described above.

If you experience the same transpose(qrfact(... error again, it would be great to open a separate issue.

[timholy]

OK, I've substantially edited the instructions above; with these changes, I think we get reasonably-accurate results, as long as a human parses the *.jl.cov files.

However, I'm running into #9536, so (for me) currently this doesn't work.

[kshyatt]

@timholy I am having the same problem as @IainNZ. I tried your example with intersect, which worked perfectly. But when I follow the guide in the top post (including --inline=no) I'm getting dashes on lines which are definitely called. I'm on commit b06873f on OSX 10.10. I tried your fixed instructions in the top post (deleting sys.dylib) as well but I'm still having the problem.

Edit: my problem was that I wasn't running with just one Julia core. Once I did that, even without deleting sys.dylib, I get the correct counts in the .cov files.

[svaksha]

On Thu, Jan 1, 2015 at 3:15 PM, Tim Holy notifications@github.com wrote:

OK, I've substantially edited the instructions above; with these changes, I think we get reasonably-accurate results, as long as a human parses the *.jl.cov files.

However, I'm currently running into #9536, so (for me) currently this doesn't work.

Thanks for the update. On Ubuntu-14.04, with commit 3f3998e408aa1aec0af6d767ff7cf42ccf9c50fd, make failed with this error after deleting rm usr/lib/julia/sys.so

~/julia$ make CC src/jltypes.o CC src/gf.o CC src/support/hashing.o CC src/support/timefuncs.o CC src/support/ptrhash.o CC src/support/operators.o CC src/support/utf8.o CC src/support/ios.o CC src/support/htable.o CC src/support/bitvector.o CC src/support/int2str.o CC src/support/libsupportinit.o CC src/support/arraylist.o CC src/support/strtod.o LINK src/support/libsupport.a CC src/flisp/flisp.o CC src/flisp/builtins.o CC src/flisp/string.o CC src/flisp/equalhash.o CC src/flisp/table.o CC src/flisp/iostream.o CC src/flisp/julia_extensions.o LINK src/flisp/libflisp.a CC src/flisp/flmain.o LINK src/flisp/flisp FLISP src/julia_flisp.boot FLISP src/julia_flisp.boot.inc CC src/ast.o CC src/builtins.o CC src/module.o CC src/codegen.o CC src/disasm.o CC src/debuginfo.o CC src/interpreter.o CC src/alloc.o CC src/dlload.o CC src/sys.o CC src/init.o CC src/task.o CC src/array.o CC src/dump.o CC src/toplevel.o CC src/jl_uv.o CC src/jlapi.o CC src/profile.o CC src/llvm-simdloop.o CC src/gc.o LINK usr/lib/libjulia.so /home/mom/julia/usr/lib/libunwind.a(Los-linux.o): In function `_ULx86_local_resume': Los-linux.c:(.text+0x75b): warning: sigreturn is not implemented and will always fail

Furthermore, runtests.jl (with JULIA_CPU_CORES=1 julia --code-coverage=all --inline=no) threw a core exception: https://gist.github.com/svaksha/76ee078ae00e694a9e35

And, since yesterday, @staticfloat's nightlies are not working - strangely, julia (anywhere) and ./julia (git master) both show the same version:

Version 0.4.0-dev+2396 (2015-01-02 07:26 UTC)
master/3f3998e (fork: 140 commits, 8 days)

Any idea what is the bug, if any? tia, SVAKSHA ॥ http://about.me/svaksha ॥

[tkelman]

@svaksha it looks like that's a warning which you've gotten before? e.g. https://github.com/JuliaLang/julia/issues/7676 - if make test doesn't succeed, then please open a separate issue.

On test coverage, I just found that interestingly our tests all pass even without a doc/helpdb.jl file present, you get a could not open file error the first time you try calling help on something. That would seem to indicate help mode has no (or very few?) tests.

[timholy]

@kshyatt, glad that fixed it for you. @IainNZ, for you it was probably also the multiprocess issue. Sorry I was slow to realize that one needs to add JULIA_CPU_CORES=1 to that launch command.

[svaksha]

On Thu, Jan 1, 2015 at 1:25 PM, Tim Holy notifications@github.com wrote:

@svaksha, I remembered your linalg4 issue; that gist you posted showed success, so clearly it's not "failing reliably." You might want to try a second time running the whole test suite as described above again.

@timholy, I am unable to reproduce the same error - ran it 3 times. Seems like a fluke.

SVAKSHA ॥ http://about.me/svaksha ॥

[svaksha]

On Fri, Jan 2, 2015 at 10:43 AM, Tony Kelman notifications@github.com wrote:

@svaksha it looks like that's a warning which you've gotten before? e.g.

7676 - if make test doesn't succeed, then please open a separate issue.

@tkelman, thanks. #9555

[chrisvoncsefalvay]

--inline=no gives me an error:

ERROR: unknown option: --inline=no
 in process_options at /Applications/Julia-0.4.0-dev-d066c3f061.app/Contents/Resources/julia/lib/julia/sys.dylib
 in _start at /Applications/Julia-0.4.0-dev-d066c3f061.app/Contents/Resources/julia/lib/julia/sys.dylib

Runs just fine without it.

[timholy]

You have to be running a very recent julia for that to work.

[chrisvoncsefalvay]

I.e. more recent than d066c3f?

[garborg]

Yeah, searching for '--inline', looks like a commit @timholy got merged 3 days ago, and I believe your commit is just a hair older.

[hayd]

Just braindumping here: I think there is something to be said (along with coverage) for "organisation" of tests, that is to say locating tests in consistent files (since we don't have cleverer ways to do it in base)... for the most part:

• foo defined in base/bar.jl
• foo is tested in test/bar.jl

(I would say this is often the case right now...)

A rough heuristic of how well a function is tested might be to count how many times that function was called in test dir (or the specific test file)... you could also do the number of lines vs lines in its definition (or the calls on specific types if several methods are defined)...

I think this would give an interesting "hitlist" of functions to test better (in the above PR I just looked up those methods which aren't called at all), but it would be interesting to see what's the next "poorest covered" method to test next.

I guess in someways this is independent of coverage but I thought I'd share my thought.

[timholy]

Interesting ideas. Regarding organization of test files, I don't have strong feelings and will let others comment, but certainly there seems to be some logic to your proposal.

Regarding measurements of coverage, I think a lot of what you're proposing could be done by expanding the analysis that's done in Coverage.jl. You've obviously been reading through the *.cov files, so you know that line counts are already available. Coverage.jl now (thanks to @jakebolewski's excellent JuliaParser package) also now "understands" methods. So one could do some analysis on a method-by-method basis, sort methods by how many times they've been called, etc.

[ViralBShah]

I think it would be useful to capture what is discussed here in the documentation, along with some guidelines on writing tests. That will perhaps get a lot more people going. We can even advertise this on the website - as a great way to get started with julia.

[quinnj]

@hayd, FWIW, your suggestion is how the Dates module is organized (see base/dates/ and test/dates). I agree that it was a very natural way to stay organized and improve coverage (with the recent coverage changes, Dates.jl coverage went from 99.73% to 83.42%, so I have a little more work to do!)

[nalimilan]

@quinnj Pretty impressive rates, still!

[hayd]

@quinnj and +1 for the Dates tests :)

I guess I was hoping to distinguish coverage between functions called "in anger" (i.e. in Base in a function which is tested) vs tests specifically against that method - looking for edge cases etc. (in a way you can only do if testing that method separately.

In python with unittest, for method foo, you might have a class TestFoo and all the tests are test_this_foo_thing methods... and you can quickly look over TestFoo, you may also want to check out the coverage from just running the TestFoo class. (i.e. for each function foo lookup it's test class and check its coverage).

IMO this flexibility is one of the benefits with a testing framework (that danluu was griping about in his blogpost)... I don't think you can do it with tests as scripts. :walking:

[StefanKarpinski]

It's somewhat embarrassing to admit, but a very long time ago there was a bug that made it problematic to have a base file and a test file with the same name, so we gave them different names intentionally :-\

[staticfloat]

It shocks me that Stefan can talk about things "a very long time ago" that I remember.

[IainNZ]

Re the test file - base file thing, I'm doing something like that in this PR: https://github.com/JuliaLang/julia/pull/9679

[chrisvoncsefalvay]

I'm trying to do this but with the current master, I get:

julia> include("/tmp/coverage_tests.jl")
linalg
ERROR: LoadError: could not open file /tmp/linalg.jl
 in include at ./boot.jl:248
 in include_from_node1 at ./loading.jl:128
 in anonymous at no file:17
 in include at ./boot.jl:248
 in include_from_node1 at ./loading.jl:128
while loading /tmp/coverage_tests.jl, in expression starting on line 15

Why's that? I feel like I'm missing something ;)

[tkelman]

try replacing include("$tst.jl") with include(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia","test","$tst.jl"))

You'll also likely need using Base.Test

[chrisvoncsefalvay]

I just checked, the problem seems to be that there is indeed no linalg.jl. Hmmm... bug or feature? :)

[tkelman]

Oh, right, linalg is a group of tests, it gets replaced by ["linalg1", "linalg2", "linalg3", "linalg4", "linalg/lapack", "linalg/triangular", "linalg/tridiag", "linalg/pinv", "linalg/cholmod", "linalg/umfpack", "linalg/givens"] in that list

[chrisvoncsefalvay]

That's one problem solved ;) Now, however, it seems it cannot find @test:

ERROR: LoadError: LoadError: UndefVarError: @test not defined
 in include at ./boot.jl:248
 in include_from_node1 at ./loading.jl:128
 in anonymous at no file:18
 in include at ./boot.jl:248
 in include_from_node1 at ./loading.jl:128
 in eval at no file
while loading /Users/mywittyusername/Developer/julia/usr/bin/../share/julia/test/core.jl, in expression starting on line 7
while loading /tmp/coverage_tests.jl, in expression starting on line 16

[tkelman]

using Base.Test should fix that, or include("testdefs.jl")

[chrisvoncsefalvay]

Excellent. Worky working now. Thanks for shepherding this testing-noob ;)

@timholy can we upd the coverage_tests.jl you provided to the following:

using Base.Test
testnames = ["core", "keywordargs", "numbers", "strings", "dates",
    "hashing", "remote", "iobuffer", "staged", "arrayops",
    "subarray", "reduce", "reducedim", "random", "intfuncs",
    "simdloop", "blas", "fft", "dsp", "sparse", "bitarray", "copy", "math",
    "functional", "bigint", "sorting", "statistics", "spawn",
    "backtrace", "priorityqueue", "arpack", "file", "suitesparse", "version",
    "pollfd", "mpfr", "broadcast", "complex", "socket",
    "floatapprox", "readdlm", "reflection", "regex", "float16", "combinatorics",
    "sysinfo", "rounding", "ranges", "mod2pi", "euler", "show",
    "lineedit", "replcompletions", "repl", "test", "goto",
    "llvmcall", "grisu", "nullable", "meta", "profile",
    "libgit2", "docs", "base64", "pkg", "linalg1", "linalg2",
    "linalg3", "linalg4", "linalg/lapack", "linalg/triangular", "linalg/tridiag", 
    "linalg/pinv", "linalg/cholmod", "linalg/umfpack", "linalg/givens"
]
for tst in testnames
    println(tst)
    include(joinpath(JULIA_HOME,Base.DATAROOTDIR,"julia","test","$tst.jl"))
end