Summary output format

[j-ulrich]

Using LCOV to calculate the coverage has one drawback: the coverage is based on lines. This makes the coverage inaccurate when there are multiple symbols defined on the same line and also when including enum values in the coverage calculation since Doxygen doesn't provide location information for them (see #12).

This PR adds a new output format: "summary". It calculates the coverage per symbol type and in total and prints a table listing the results like this:

Classes    :  90.5% (38/42)
Defines    :   0.0% (0/2)
Enum Values:  12.3% (8/65)
Enums      :  75.0% (3/4)
Files      :   8.3% (2/24)
Functions  :  64.8% (175/270)
Namespaces :  75.0% (6/8)
Pages      : 100.0% (7/7)
Signals    :  83.3% (5/6)
Slots      :  28.6% (2/7)
Structs    :  80.0% (4/5)
Typedefs   :  39.1% (9/23)
Variables  :  20.0% (8/40)
-----------------------------------
Total      :  53.1% (267/503)

This PR also changes the "json" and "json-legacy" output formats slightly: it adds a "kind" property to the symbol information:

[...]
    {
      "symbol": "Tests::ManagerTest",
      "documented": true,
      "kind": "class",
      "line": 18,
      "file": "/path/to/ManagerTest.cpp"
    },
[...]

IMO, that change is backward compatible. However, if someone relies on the exact format of the JSON, then this will break it of course. So if you think the JSON formats should not be changed, then I can filter the "kind" property out, of course. Just let me know.

PS: I will create another PR to add options to exclude/include files using regular expressions to allow more fine grained control over what is included in the coverage calculation.

[coveralls]

Coverage increased (+4.6%) to 70.642% when pulling a5898f20a7139bbb60b830bf9b49b51efa9c209b on j-ulrich:feature/summary into 33bb35fe5b938b2d61d4a5569bd6c4ff254b79c6 on psycofdj:master.

[psycofdj]

@j-ulrich Thank again for the PR

Few remarks

summary output

No problem adding a new format.

In addition, I think that having this accurate coverage by symbol kind is a valuable information that should also be present in the json output. Sadly, current json and json-legacy formats were poorly designed and can't really evolve.

A better format could be:

{
  "summary": { 
    # section giving general information, global coverage, number of symbols... anything relevant 
  },
  "files": [ 
    # content of current json format
  ],
  "kinds" : {
    # informations contained in your new format proposal
  }
}

In order to keep backward compatibility, this new format could be named json-v3. json and json-legacy must be kept and can respectively be aliased to json-v2 and json-v1 in command line options and help.

Let me know your thoughts about this suggestion.

kind field in output

No problem adding a new field in the object structure. Any well-written parser shoud handle an unused unknown field.

incoming PR about filtering

That would be great. From my experience, it's always more comfortable for a user to manipulate globbing instead if regexp when working with files. If it has enough expressiveness for your filtering use-case, then please consider this choice. If not, then feel free to do as planned with regexps.

[j-ulrich]

json formats

I think your suggestion makes sense. I was just wondering if it would be sensible to also add a json-summary format which contains just the "summary" and "kinds" parts. 🤔 Because if a user is just interested in the summary in JSON format, then json-v3 still spits out that (possibly very long) list of files and symbols.

kind field in output

Just to verify: so it is okay to have the new filed in json-v2 and json-v1 and I don't need to filter it out, right?

incoming PR about filtering

I had a second though on this and came to the conclusion that it is actually unnecessary to provide filtering options in coverxygen since Doxygen already provides sufficient filtering options. So one can simply create a doxyfile which generates XML containing only the files and symbols that should be included in the coverage report.

If you think it makes sense to have file filtering options in coverxygen as well, then I can still submit a MR with it. I have it implemented with regular expressions already, just the unit tests are missing. For globbing we could introduce an --extended-regex option which makes the patterns use full regex and otherwise use globbing patterns.

[j-ulrich]

Here is my proposal for the new json-v3 format:

{
  "total": { 
    "symbol_count": 503,
    "documented_symbol_count": 267,
    "coverage_rate": 0.53081510934393638170974155069583
  },
  "files": {
    "/opt/foo/bla.h": [
      {
        "symbol": "Something",
        "documented": true,
        "kind": "class",
        "line": 17,
        "file": "/opt/foo/bla.h"
      },
      # ...
    ],
    # ...
  },
  "kinds" : {
    "class": {
      "symbol_count": 42,
      "documented_symbol_count": 38,
      "coverage_rate": 0.9047619047619047619047619047619
    },
    "define": {
      "symbol_count": 2,
      "documented_symbol_count": 0,
      "coverage_rate": 0.0
    },
    # ...
  }
}

[psycofdj]

json formats

Yeah, json-summary sounds good since the json-v3 can be indeed very verbose.

The proposed json-v3 format is perfect.

kind field in output

Yes, those filed can be included and don't need to be filtered out from json-v1 and json-v2 formats

incoming PR about filtering

If the work is already done with regex I'm fine with this. I believed that the incoming PR was yet to be implemented. I think the feature is still interesting since it will allow to produce documentation on a specific part and rule it out from coverage for some reason.

[psycofdj]

@j-ulrich Great job, thanks again !

What have you decided about the filtering feature ? Should I release on pypi already or wait another PR ?

[j-ulrich]

Should I release on pypi already or wait another PR ?

I'll finish it and create a PR today!

PS: Thanks for adding me to the "Hall of Fame"! 😄