search

JuliaCI / julia-buildbot

Buildbot configuration for build.julialang.org
MIT License
19 stars 14 forks source link

Coverage failing #15

Closed staticfloat closed 9 years ago

staticfloat commented 9 years ago

We're trying to run CoverageBase.jl after every commit that makes it through testing and packaging (for Linux x86_64). Unfortunately, it looks like I'm not doing something quite right.

First off, here is the code that's getting run, here is the latest run with all the steps and their logs. There are two things I have questions about. First off, there's the "skipped" messages during the .cov parsing stages. Here's an example. Secondly, after running all these steps, I try to submit to Coveralls, and I get a method undefined error. I'm thinking this might have something to do with all the "skipped" messages.

@timholy any tips here would be appreciated. I ran a Pkg.clone() on your CoverageBase repository and am currently sitting on commit c77855df2. If you need to login and mess around with the buildbot I'm running this on, just let me know, I'll email you the login details.

staticfloat commented 9 years ago

@IainNZ I just looked through the source, and realized that it's Coverage.jl that spits out the "skipped" messages while parsing .cov files. Do you have any idea why it might be doing that here? Here are the .cov files that are created by the second (non-inlined) run from CoverageBase.jl. taking a look at them, I'd have to say that they look like something is wrong, but I don't know what. Any ideas?

timholy commented 9 years ago

First, nice that you're getting this set up!

Yes, indeed it's Coverage.jl that emits the "skipped" messages, and what you're seeing is normal behavior. Coverage.jl is set up to try to process every file in the directory, and emits "skipped" messages when it looks like the file is not a "julia source file" with a corresponding *.cov file. Since there is no file.jl.cov.cov, it skips file.jl.cov (but doesn't skip file.jl since there is a corresponding file.jl.cov). There is a proposal to skip any file ending in .cov. That coverage file you linked to looks normal, so I don't think there is anything wrong.

The only actual error is the "method undefined" error, and it's in HDF5/JLD. I needed to merge https://github.com/timholy/HDF5.jl/pull/206; I just did so, and tagged a new version. You should be good to go now.

staticfloat commented 9 years ago

Rejoice. It worked! Next question; how can I link in git information? It isn't getting the commit/committer information in Coveralls, which is kind of a bummer.

timholy commented 9 years ago

No clue how to do that, but I only skimmed the Coveralls API. https://coveralls.zendesk.com/hc/en-us

Why is your coverage number (58.2%) so much lower than mine (71.1%)? https://coveralls.io/r/timholy/julia

staticfloat commented 9 years ago

I have no idea why my coverage number is so much lower. I'll try to take a look at it this weekend to see what's going on. Here's to hoping that it's a problem with my invocations and not yours!!!!

staticfloat commented 9 years ago

I wonder if backtrace issues are causing this difference. What kind of environment are you running your tests on? This is using the generic linux binaries, so perhaps something is causing those binaries to give less detail in their backtraces.

timholy commented 9 years ago 
julia> versioninfo()
Julia Version 0.4.0-dev+2651
Commit dfbcf50* (2015-01-12 19:02 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

It's a Kubuntu 14.04.1 system (GNU/Linux 3.13.0-35-generic x86_64).

Just checking, are you deleting (or running without) the sys.so file? That contains a lot of precompiled functions, and anything inlined during precompilation will not get the credit it is due.

In my script in CoverageBase.jl I delete that file, but later I learned from Jameson this trick (now in our .travis.yml):

cp usr/lib/julia/sys.ji local.ji && ./julia -J local.ji <testscript> && rm local.ji
staticfloat commented 9 years ago

Well. Removing sys.so seems to have done the trick! I guess I missed that when I was writing up the buildbot script. I guess I'll work on adding commit data so this can actually be tracked per-commit*, and then we're good to push this over to the main JuliaLang repository.

Caveats: * per commit that is applied to master, makes its way through the buildbot testing suite and is packaged on linux successfully.

timholy commented 9 years ago

Nice!

One last small issue: on coveralls, clicking the link for a particular file gives a "source not found" error. If I remember correctly, I needed to pay close attention to the directory from which Coverage was being run (I think in julia's top-level directory, but you can check the script).

staticfloat commented 9 years ago

Yes, my paths are all screwed up. Thanks for the reminder.

On Wed, Jan 21, 2015, 13:07 Tim Holy notifications@github.com wrote:

Nice!

One last small issue: on coveralls, clicking the link for a particular file gives a "source not found" error. If I remember correctly, I needed to pay close attention to the directory from which Coverage was being run (I think in julia's top-level directory, but you can check the script).

— Reply to this email directly or view it on GitHub https://github.com/staticfloat/julia-buildbot/issues/15#issuecomment-70921717 .

tkelman commented 9 years ago

This passes for now. Tantalizingly close to 80%, largely thanks to @kshyatt really raising the bar.