Add testing of doctest coverage with --doc-coverage

[asmeurer]

To run, use --doc-coverage. --doc-coverage automatically enables --build-docs, because the docs must be built to get the Sphinx coverage.

The output so far looks something like this:

SymPy Bot Summary: :white_check_mark: Passed after merging asmeurer/travis-sphinx (2e1817c7a1fb39a0c798d557ad48e54a0cabe5c1) into master (0ec0ba5a953f8c52d971f9bd1aff9c6089b98766). :white_check_mark: Sphinx 1.2b1: pass :red_circle: Doc Coverage: (Failed) 36% (1983 of 5425) of functions have doctests 40% (2221 of 2221) of functions are imported into Sphinx

Still TODO:

• [ ] (Only if someone wants it) allow to customize the args sent to coverage_doctest.py. This would allow testing the coverage of specific modules, or not testing Sphinx coverage.
• [ ] Cleanup display of the output. I was thinking of using some kind of table. HTML tables should be supported. I'm open to suggestions here.
• [x] Add comparative testing against master, so that we can see if the coverage improved or got worse.
• [x] Figure out a good Emoji to use for the above. I was thinking :chart_with_upwards_trend: if the coverage improved and :chart_with_downwards_trend: if it got worse, but those look kind of the same. What I really want is a green arrow pointing up and a red arrow pointing down. Maybe we should just grab some custom images and host them in the sympy-bot repo, and just show those. Again, I'm open to suggestions.
• [x] Should we pass -v to coverage_doctest.py? The answer is probably yes, though the output is very verbose in that case.
• [ ] Test this.

[asmeurer]

Any idea why the doctest coverage script doesn't flush with every newline?

[asmeurer]

OK, I added comparison against master (the formatting is still not nice, but I will fix that). Here is an example for https://github.com/sympy/sympy/pull/1993.

SymPy Bot Summary: :white_check_mark: Passed after merging pernici/nthroot (828e3cc6d35f894794a87ef5e922a6412fd57bb4) into master (0ec0ba5a953f8c52d971f9bd1aff9c6089b98766). :white_check_mark: Sphinx 1.2b1: pass :red_circle: Doc Coverage: (Failed) 36% (1985 of 5427) of functions have doctests 40% (2221 of 2221) of functions are imported into Sphinx For comparison, in master, the coverage is 36% (1983 of 5425) of functions have doctests 40% (2221 of 2221) of functions are imported into Sphinx

[asmeurer]

OK, images added. It now looks like

SymPy Bot Summary: :white_check_mark: Passed after merging pernici/nthroot (828e3cc6d35f894794a87ef5e922a6412fd57bb4) into master (00a0a1e9d27c3f57ed4acd96c20b4c7374078099). :white_check_mark: Sphinx 1.2b1: pass :red_circle: Doc Coverage: (fail) 36.60% of functions have doctests (compared to 36.58% in master) 40.89% of functions are imported into Sphinx (compared to 40.91% in master)

Or if the coverage remains unchanged

SymPy Bot Summary: :white_check_mark: Passed after merging pernici/nthroot (828e3cc6d35f894794a87ef5e922a6412fd57bb4) into master (00a0a1e9d27c3f57ed4acd96c20b4c7374078099). :white_check_mark: Sphinx 1.2b1: pass :red_circle: Doc Coverage: (fail) 36.58% of functions have doctests (compared to 36.58% in master) 40.91% of functions are imported into Sphinx (compared to 40.91% in master)

[asmeurer]

OK, this is now ready for testing, I think. I'm running sympy-bot review mergeable --doc-coverage.

[asmeurer]

One more change before I test this everywhere. Now the summary says if the coverage overall increased, decreased, or remained unchanged:

SymPy Bot Summary: :white_check_mark: Passed after merging pernici/nthroot (828e3cc6d35f894794a87ef5e922a6412fd57bb4) into master (00a0a1e9d27c3f57ed4acd96c20b4c7374078099). :white_check_mark: Sphinx 1.2b1: pass Doc Coverage: decreased 36.60% of functions have doctests (compared to 36.58% in master) 40.95% of functions are imported into Sphinx (compared to 40.97% in master)

SymPy Bot Summary: :white_check_mark: Passed after merging asmeurer/travis-sphinx (2e1817c7a1fb39a0c798d557ad48e54a0cabe5c1) into master (00a0a1e9d27c3f57ed4acd96c20b4c7374078099). :white_check_mark: Sphinx 1.2b1: pass Doc Coverage: unchanged 36.58% of functions have doctests (compared to 36.58% in master) 40.97% of functions are imported into Sphinx (compared to 40.97% in master)

[asmeurer]

I'm wondering if we should change -D from --build-docs to --doc-coverage, so that the default behavior is to also check the coverage.

[asmeurer]

I finished funning the tests. Only one pull request resulted in an increase in both kinds of coverage. Probably about half resulted in unchanged coverage (they did not add or delete functions), and of those that did change, probably about half resulted in an increase in doctest coverage and half in a decrease. Only one pull request resulted in an increase in Sphinx coverage (I could be wrong about that though because there was a bug that made it not notice small changes). So our current coverage percentages should not be surprising.

Some comments on this:

• Adding a new function and not covering it will decrease coverage (because a/(b + 1) < a/b).
• Adding a new function and covering it will increase coverage (because (a + 1)/(b + 1) > a/b).
• Deleting a function that was not covered will increase coverage (because a/(b - 1) > a/b).
• Deleting a function that was covered will decrease coverage (because (a - 1)/(b - 1) < a/b).

Due to this, I think the increase or decrease arrows are good metrics.

The one pull request that resulted in an increase in Sphinx coverage did so by deleting a function. The coverage script is not run when Sphinx fails, so that could also be part of the issue (whenever something changes Sphinx, there is a high chance it will spuriously fail in my bot).

I think that the doc-coverage code is well tested. I would still like to test running the bot without the bot coverage to make sure that works, but all in all, I'm done with my work here.

[flacjacket]

This looks neat, and it looks good to me. I'll test it out when I'm back home.

[asmeurer]

OK, this has sat here forever, and I've been using it for quite some time without any other comments (positive or negative). I'm going to merge it.