Provide a way to customize badge color thresholds

[alechenninger]

It seems like the default color thresholds are awfully high. Even at 70%, coverage is still "red" implying this is bad.

It would be nice if an individual repo could set up their own target thresholds for what they find an acceptable level of unit test coverage, with pretty colors to match :).

High test coverage percentage numbers are not necessarily good: http://martinfowler.com/bliki/TestCoverage.html

And certainly what is considered good is up for debate: http://googletesting.blogspot.com/2014/07/measuring-coverage-at-google.html

Not a huge deal, but would be nice.

Thanks!

---

Update: For what it's worth, while I originally opened this issue, I actually would prefer this not change from current behavior now :). I don't mind the psychological nudge to increase test coverage (even if not necessarily a good thing), and I like that a Coveralls badge of a certain color means the same thing on any repo. It would be a little surprising if you saw a "green" badge with 10% coverage, for example.

[jewzaam]

+1

[brettwooldridge]

:+1: We have >80% coverage, yet our badge is still "orange". At least for our coverage, we would like >80% to be green. If this could be configured per-repo in the settings, that would be more than awesome.

[giggio]

Agreed. This is very much needed.

[ghost]

I dunno guys, I've recently decided to rewrite my test suite from scratch on a certain project I maintain, and when I wrote a null test suite (that is, a test suite that imported every module but didn't run a single function or test a single thing), I got a coverage report of 40%. I now have 100% test coverage on a few modules but still haven't written a single test for most of the modules, and I'm already at an overall test coverage of 65%.

I think coverage numbers are greatly inflated and the (seemingly) high thresholds are probably a good idea overall. 80% coverage probably isn't as impressive as you think it is.

[giggio]

That is not the point. The point is that different projects demand different thresholds. Simple as that.

[xaionaro]

@robru, for example there may be a lot of exceptions that will never be runned (otherwise program will exit). Also for example there may be a lot of mutually exclusive code. So it can be a really huge and pointless work to make an appropriate test suite. IMO, every project (even every file in every project) should have own configurable thresholds.

Personally I refused coveralls just due to that fact, that I cannot configure this thresholds.

[nickmerwin]

@giggio @xaionaro your Coveralls integration libraries should support some sort of :nocov: indication in the code; it's to prevent those lines from being counted as "relevant" and thus not included in overall coverage calculations.

@alechenninger @brettwooldridge we set the thresholds high to encourage you to strive for higher numbers. Of course you can always wrap non-mission-critical code in :nocov:'s to increase your coverage %. But if every line of code was truly mission critical, then achieving 100% coverage should be the goal.

I'm closing this issue for now, but would love to keep the conversation going. Thanks!

[pron]

AFAIK, :nocov: is not available for JaCoCo.

If you don't want to make the thresholds fully configurable, an alternative would be to provide a few presets, like "strict", "lenient" etc. One of the presets should be "non-judgmental", that should just color the badge blue no matter what the coverage percentage is.

You want to encourage better coverage, but you don't want to turn this into a numbers game, and you certainly don't want developers to start "juking the stats" :)

[alechenninger]

The presets are a neat idea. I can appreciate that coveralls wants to keep things simple, though. I'm not sure if there's any infrastructure for per-repository custom configuration like this.

I guess a "red" badge just implies things are "bad" when "bad" is much more complicated than that.

http://junit.org/faq.html#best_2

becomeTimidAndTestEverything
while writingTheSameThingOverAndOverAgain
    becomeMoreAggressive
    writeFewerTests
    writeTestsForMoreInterestingCases
    if getBurnedByStupidDefect
        feelStupid
        becomeTimidAndTestEverything
    end
end

[aquamatthias]

Encouragement is always a good thing - but it has to met reality. The threshold for good/warn/bad really is very project specific and should be configurable. I can't see any good explanation for closing this ticket.

[giggio]

I completely agree! Every project is different. The argument that we should strive to get higher values is moot, as different projects have different characteristics and different achievable thresholds. "Higher" varies per project.

[xaionaro]

@giggio @xaionaro your Coveralls integration libraries should support some sort of :nocov: indication in the code; it's to prevent those lines from being counted as "relevant" and thus not included in overall coverage calculations.

How to use that? Should I add this into the code? Is this compatible with gcc?

[304NotModified]

+1

[automatictester]

+1

[abesto]

+1, this would be quite useful.

[alechenninger]

For what it's worth, while I originally opened this issue, I actually would prefer this not change from current behavior now :). I don't mind the psychological nudge to increase test coverage (even if not necessarily a good thing), and I like that a Coveralls badge of a certain color means the same thing on any repo. It would be a little surprising if you saw a "green" badge with 10% coverage, for example.

[abesto]

That's a good point; after this, I wouldn't press the issue, it's not quite clear cut. Thanks.

[304NotModified]

That is not the point. The point is that different projects demand different thresholds. Simple as that.

QFT.

Please don't decide for others what is good or not. That you're fine with the current implementation is nice for you, but different projects have different goals.

Don't get this why this is closed. IMO it should be not that technical difficult.

[giggio]

Exactly.

[automatictester]

I care about code coverage, therefore having an amber badge when I'm sure coverage is fine was annoying. I switched to codecov, I can configure color thresholds as I see fit. It was sad to split up with coveralls, I hope you will add this feature one day.

[nicpottier]

Note there's no way to exclude lines from coverage in say Go, so I think the argument towards not having this be configurable is weakened, especially given the propensity of orphaned error case switches in Go which are silly to cover entirely.

[304NotModified]

This is still not fixed ;)

Anyway, we moved to Codecov. The do support this feature.

[bergmeister]

Upvote +1

[ElliotNB]

@nickmerwin Would be great to have this feature. Every project is different.

[dstndstn]

"I'm closing this issue for now, but would love to keep the conversation going" is, uh, some kind of stance

[ItalyPaleAle]

Would actually +1 this. I'm working on projects written in Go in which there's very verbose error handling. Those who have written code in Go know that every function call is always followed by an error check:

err := foo()
if err != nil {
    return err
}

The issue is that some of those errors are not critical or are not testable (e.g. checking if a call to os.Create() failed would require simulating an error from the filesystem trying to allocate a file: hard to do and not very usefu). I also have some files in the projects that are auto-generated (e.g. from protobuf) and for which I won't write tests for

Because of that, I'd love to be able to set a lower "green" threshold that makes sense for my projects.

[lukeredpath]

It’s 2021 and this is still not an option? Please let us set our own coverage thresholds. I’m working on an app where certain parts of the codebase (SwiftUI previews) will never have coverage and they can make a big difference to coverage numbers. There’s no way currently to exclude this code from the reports. I just want to se my own red/orange/green percentages. Please.

[afinetooth]

@lukeredpath, thanks.

I've added your +1 to the feature request.

[dapimentel]

I agree this would be a useful feature.

[suculent]

It’s not just the badge. Thresholds should be customizable (to pass at certain %). Compare to other public projects would be interesting (like LGTM/Semmle).

4. 4. 2022 v 23:23, David A. Pimentel @.***>:

 I agree this would be a useful feature.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[afinetooth]

Thanks @dapimentel and @suculent. I have added your +1's and comments to the appropriate card in our backlog.

[azat-io]

+1 for this feature

[afinetooth]

Thanks @azat-io. I have bumped the feature card in our backlog.

[azat-io]

Any progress? It would be very cool to have this feature!

[afinetooth]

Thanks for checking in on this and plus-one'ing it. It gives us here in support an excuse to bump it! 🙏

[azat-io]

Why is this issue closed? It hasn't been implemented yet, has it?

[afinetooth]

Hi @azat-io. No it hasn't. I'm sure it was auto-closed. I've reopened it.