provide a way to get net branch coverage at project level

[anmolbabu]

Currently, the awesome gobco tool provides an awesome view of package level branch coverage. However, since the output of gobco is a string, any attempt to aggregate the individual package level branch coverage into a coverage at the project level, would potentially involve a lot of string magic(undesirable) so is it possible to:

1. Provide an aggregated metric across all packages or
2. Provide outputs as json or any-other parsable format for external integrations like CI systems

[rillig]

Have a look at the -stats option. If there's anything else you need, please reopen this issue.

[anmolbabu]

@rillig I tried the -stats option. I see the code conditionals however I am not sure how to interpret this. Can you please add some kind of documentation explaining what each of the fields mean especially around TrueCount and FalseCount. Also, how they can be interpreted to count the coverage(Apologies in advance for my limited knowledge of using this tool for last 2-3 days and not having iterated through the code.)... Unfortunately, also I don't see an option to re-open this issue .. So, I am hoping you find it else, I'll create a new one later this week if this comment gets lot... Is this ok ?

[rillig]

Hmmm, I thought I had named the fields in a straight-forward way. If you just look at the names TrueCount and FalseCount, what could be their purpose? Please tell me in detail what you already understand by just looking at the JSON output and what needs further clarification, as I don't want to document things that are already obvious.

By the way, I don't expect you to read the source code of gobco. Everything about gobco should be obvious from just running gobco and then looking at the results. If that's not the case, I have to improve it.

[anmolbabu]

hmmm...

Well I thought the terms TrueCount and FalseCount are obvious of what they probably are. However, not expecting someone to have read the code or some documentation about how gobco works, and what these terms actually are, I failed to understand the non-binary values in the brief excerpt below(with no loops for some and not more than 8 test cases but a false count being 19) :

    {
        "Start": "object.go:114:5",
        "Code": "unMarshaller == nil",
        "TrueCount": 0,
        "FalseCount": 3
    },
    {
        "Start": "object.go:119:5",
        "Code": "err != nil",
        "TrueCount": 0,
        "FalseCount": 3
    },
    {
        "Start": "object.go:128:5",
        "Code": "obj == nil",
        "TrueCount": 1,
        "FalseCount": 19
    },
    {
        "Start": "object.go:132:5",
        "Code": "t.Kind() == reflect.Ptr",
        "TrueCount": 0,
        "FalseCount": 19
    },

I also failed to understand how these non-binary TrueCount and FalseCount can be used for effectively calculating branch coverage without realising why they are non-binary and as above

[rillig]

I failed to understand the non-binary values in the brief excerpt below(with no loops for some and not more than 8 test cases but a false count being 19) :

  {
      "Start": "object.go:128:5",
      "Code": "obj == nil",
      "TrueCount": 1,
      "FalseCount": 19
  },

Did you set a breakpoint on this expression to see how often it is actually called during all the tests? Since 19 is a prime number, I'm pretty sure it is accurate.

I also failed to understand how these non-binary TrueCount and FalseCount can be used for effectively calculating branch coverage without realising why they are non-binary and as above

Ah, well, it might be that I did not use the term branch coverage properly. What gobco does is more of a condition coverage. Gobco assumes that each boolean expression can be false or true, and for each of these possible outcomes, gobco creates one "branch to be covered". In the end, gobco just prints this summary (see gobco.printOutput):

cnt := 0
for _, c := range conds {
    if c.TrueCount  > 0 { cnt++ }
    if c.FalseCount > 0 { cnt++ }
}
g.outf("Branch coverage: %d/%d\n", cnt, len(conds)*2)

This means that gobco might count some branches twice or more, which might produce statistics that are a bit off. But since my goal, in the end, is to have all branches covered, this doesn't matter in practice, at least to me. As soon as I add a test for a previously uncovered branch, the count of covered branches goes up, and this is all I expect.

[rillig]

Feedback timeout.