Branch Coverage Metrics? (Ruby 2.5 is here!)

[Mah-D]

Is there any way to collect branch coverage metrics additional to statement/line coverage?

[xaviershay]

No, the underlying Ruby coverage feature does not provide this.

http://ruby-doc.org/stdlib-2.1.0/libdoc/coverage/rdoc/Coverage.html

[bf4]

I'll need to add this to the standard responses page should it ever get written. In the meantime, link to the issue for creating that page. https://github.com/colszowka/simplecov/issues/340

[ragesoss]

It's now possible on a recent build of Ruby to get branch and method coverage: https://bugs.ruby-lang.org/issues/13901

It's supposed to be available in Ruby 2.5.

I've started poking at simplecov to see how hard it will be to incorporate the new coverage options...

I think it'll be straightforward to annotate lines with the number of missed branches. Parsing the code to highlight the missed branches may be a little tougher, but seems doable.

[PragTob]

@ragesoss :wave: hi there, thanks for digging this up. I certainly didn't know about this. It's good to know. Also a bit scary to know about all the work coming our way :D Tracking it would somehow work, however I'm not sure it's the best format. I still struggle to see how branch coverage can be visualized properly, but I think a look at other languages would help us mightily there.

Might open a new issue to track this. Would be amazing to have this once 2.5 drops. If they don't release it and change it mightily it would be very sad for our efforts :|

[ragesoss]

Yeah, it's a hard problem to visualize it and I haven't seen a really nice solution, but my project uses Istanbul and it's usable, with inline annotations of where there are untaken if/else branches.https://github.com/gotwarlost/istanbul

[colszowka]

Nice find @ragesoss, thank you for bringing this to our attention!

As far as I understood the Ruby issue correctly however, this is not "branch coverage" in the classical sense, i.e. one || two, but rather adds support to the coverage module to distinguish for inline conditionals foo if bar whether simply the condition was evaluated, or the actual call to foo has happened. Currently, if in all your tests bar is falsy, you'd still get this line marked as covered.

[ragesoss]

Currently, if in all your tests bar is falsy, you'd still get this line marked as covered.

Can you explain a little more the situation you mean?

It shows whether or not the truthy branch was covered in the last line:

#target.rb
def bar
  return false
end

p(:foo) if bar

=>

{"target.rb"=>{:lines=>[1, 1, nil, nil, 1], :branches=>{[:if, 0, 5, 0]=>{[:then, 1, 5, 0]=>0, [:else, 2, 5, 0]=>1}}}

The 'then' branch on line 5 gets 0 hits, while the implicit 'else' branch gets 1.

[PragTob]

I think the true test for branch coverage is if we can correctly get coverage for nested conditionals, like:

if foo
  if bar
    ...
  else
    ...
  end
else
  if baz
    ...
  else
    ...
  end
end

If I'm not mistaken there are 4 separate paths here - for branch coverage iirc we'd have to be able to distinguish all 4 completely and assert that all 4 branches were taken.

Would need to play with it - I don't really see any time upcoming in the next days though :|

[ragesoss]

def foo;end
def bar;end
def baz;end

if foo
  if bar
    puts 'bar'
  else
    puts 'not bar'
  end
else
  if baz
    puts 'baz'
  else
    puts 'not baz'
  end
end

=>

{"target.rb"=>{
  :lines=>[1, 1, 1, nil, 1, 0, 0, nil, 0, nil, nil, 1, 0, nil, 1, nil, nil],
  :branches=>{
    [:if, 0, 6, 2]=>{[:then, 1, 7, 4]=>0, [:else, 2, 9, 4]=>0}, # inner bar conditional
    [:if, 3, 12, 2]=>{[:then, 4, 13, 4]=>0, [:else, 5, 15, 4]=>1}, # inner baz conditional
    [:if, 6, 5, 0]=>{[:then, 7, 6, 2]=>0, [:else, 8, 12, 2]=>1}}, # outer foo conditional
  :methods=>{[:foo, 0, 1]=>1, [:bar, 1, 2]=>0, [:baz, 2, 3]=>1}}
}

There are 4 paths, but 6 branches. The branch coverage feature looks like it does nested conditionals just fine. :)

[PragTob]

Sorry I was tired and my example was wrong :D This would even work with normal line coverage, it nice to see what the format is though :+1:

For real branch coverage we need:

if bar
  puts "bar"
else
  puts "no bar"
end

if foo
  puts "foo"
else
  puts "no foo"
end

that's 4 branches that we'd like to see are coverd, line coverage wise we could cover it with 2 test cases (foo and bar, not foo and not bar for instance)

[ragesoss]

So you mean, you'd want to see that "foo and not bar" was not covered, because the two test cases covered "foo and bar" and "not foo and not bar"?

My understanding is that that's 'path coverage': https://grosser.it/2008/04/04/whats-my-coverage-c0-c1-c2-c3-path-coverage/

[PragTob]

phew... it seems I need to brush up on my vocabulary :D I thought that was branch coverage... busy weekend and days ahead so might take me a while :|

[PragTob]

You are of course right, I mistook branch coverage for path coverage

[PragTob]

I'm reopening this one in favor of #645

[PragTob]

Yes, we definitely want to do this my time is sadly cut short and right now I sadly don't see myself working on this too soon as it's quite the effort. Contributions of course welcome but this is a bigger project :)

[david-a-wheeler]

@PragTob: I totally understand limited time. That said, I'd like to make it clear that I'm definitely interested in this, and I suspect that others are interested as well. It's clear that Ruby 2.5 really did add the ability to measure it.

I should note that the "Best Practices Badge" top level, gold, requires 80%+ branch coverage if there's a FLOSS tool that can measure it. (This work is sponsored by the Linux Foundation Core Infrastructure Initiative). I'm the technical lead of that project, and we'd like to make sure we meet this requirement ourselves... but we can't do that without a tool that can measure it. So there are people who specifically want to be able to measure branch coverage!

Thanks!

[coorasse]

I think this a one of the greatest improvements ruby did recently. I am not exaggerating it: I think it's really fundamental and it was just a missing piece until now.

Said that, I'd be glad to help simplecov implement branch coverage ASAP and start using it in our projects as well.

As others I give my availability to help but I can also start working on it right away if someone else is not doing that already (don't want to waste time). I think it would be good to know from the maintainer the current status of that feature and if some of you guys is already working on it.

[PragTob]

:wave:

First, thanks for you offer and willingness to help :green_heart:

As far as I know no one is working on it. At least I'm not & have heard from no one. My capacity is rather limited due to other projects, conference talks etc. and will be for the upcoming weeks. Please feel free to ask questions here or in some other form.

I'll do my best to answer them but can't always promise a very timely reply (you know, job, personal life et. al.).

I'm also a fan of early PRs - sometimes just to check if the overall approach is fine. I haven't looked into it but I'm imagining we might need some architectural changes as to allow the processing of 2 "coverage modes" - this will likely include some extractions and extensions. If we could get some of those in separately and not all as a big bang PR that'd probably help the whole process tremendously :)

Cheers, Tobi

[grosser]

FYI

• implementation for simple/fast CLI coverage tool https://github.com/grosser/single_cov/pull/7
• merging branch coverage of multiple processes (forking workflow) https://github.com/grosser/forking_test_runner/pull/10

[marcandre]

In case it helps anyone, note that DeepCover implements branch coverage, as well as improved expression coverage.

[som4ik]

@PragTob can anyone review my PR?

[som4ik]

Simplecov is a standard way of coverage check in Ruby. Unfortunately, it doesn't include a branch check yet. This has led to the appearance of new gems, which can cause a community splitting and dispersion of efforts of developers between different libraries. In our project, we had to check branch coverage and we didn't want to abandon simplecov. That's why I have implemented simplecov support, here is the link to my PR. There can be found a short description of everything I have added for support implementation. I understand that the diff is very big and it's challenging to check it, but the task for me wasn't an easy one either. To simplify your work I am ready to help by answering any questions in the course of code review: Skype call, Gitter, Slack, email — whatever you prefer

[PragTob]

Thanks to lots of people a basic version just landed on master. Remaining work for a full release is tracked in #781