Add testing files / coverage tests?

[svenevs]

It would be nice to include some unit tests. I'm not exactly sure how these tests would be performed or validate though. I will have to think a lot more about how this would go down.

[tkhyn]

I second that. Did you give it some thought?

To me it seems there are several things that would need to be tested:

1. Unit test of exhale components. Should be relatively easy.
2. Doxygen stdin input: check that exhale generates the correct input for Doxygen depending on configuration
3. Check that for a given set of doxygen XML files, the RST output is as expected
4. full black-box tests from sources of sample projects documented with Doxygen (optional but quite relevant in my opinion)

1, 2 and 3 can be implemented easily, maybe with a few tweaks in the source code to better separate components and make them easily testable.

4 would be a bit harder (and would require Doxygen to be installed on the test platform, which is not a biggie) but not infeasible.

What do you think? Is there any test framework you prefer? I tend to use py.test on my recent projects. If I can find some time I'll be happy to give it a go (as debugging and making sure nothing breaks is much easier with tests).

[svenevs]

If you want to get started I'd be very grateful! The tests are going to ultimately require tracking some sample files, so I think a tests/ folder would be appropriate. If you wanted to get the ball rolling with (1) then I think the easiest starting place would be to test some of the random string processing functions in the util module.

py.test seems like a great choice.

I'd like to hold of on (2)-(4) until I can implement multi-project support, as that will make the tests easier to control / write in the first place. The changes I've been slowly working on for that make externally controlling the execution (e.g. not needing sphinx) significantly more manageable. Right now it's really annoying to actually get things to run independent of sphinx.

The main hurdle is that no RST can be generated without a full execution. The main index.xml is needed to build the graph, and only when the graph is built can anything be generated. This is the main testing site, it being able to build tests a lot of things just by building (e.g. tree view vs bootstrap tree view vs no tree view). This "is" (4) (currently)...

Thanks for your interest in Exhale! I definitely don't expect you to submit anything on this, realistically I don't have a lot of bandwidth for exhale at this point, and that won't be changing until the summer. The windows issues should be easy to fix this weekend, but adding in the tests is lower on the priority list for now :wink:

[tkhyn]

Cool

In order to test subsequent changes regarding #30 I built a mini test framework which actually ends up doing (4). I've polished it a bit and created a branch if you want to get some inspiration: https://github.com/tkhyn/exhale/tree/testing. It can easily be modified for multi-project testing.

It uses zc.buildout to create an isolated test environment. To run the tests:

cd /path/to/exhale
pip install zc.buildout
buildout
bin/tests

And you can even generate coverage statistics by running bin/tests --cov

Note: doxygen will need to be installed for the 2 tests to pass

[tkhyn]

I forgot to add a quick note regarding utils.fancyError in try/except blocks instead of letting the actual root exception being raised: the tests output is blank when it happens, which makes it quite hard to find where and why the actual error occurs.

[svenevs]

@tkhyn this is sweet! My understanding of this is zc.buildout is able to pretend to be a sphinx application?!!!

sphinx.errors.ConfigError: Exhale: the file [./_doxml/index.xml] does not exist.  Has Doxygen been run?

So _doxml was created deep within the fiery pits of tests/. There may need to be some finessing, Exhale has very explicit path requirements. But I'm having trouble even forcing an error.

        # The Doxyfile *must* be at the same level as conf.py
        # This is done so that when separate source / build directories are being used,
        # we can guarantee where the Doxyfile is.
        # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
        raise RuntimeError("Source={}, Conf={}".format(app.srcdir, app.confdir))
        if exhaleUseDoxyfile:

As you can tell I'm new to buildout, but adding that explicit error should be what happens with the tests now. The CFG file doesn't appear to install the exhale egg, when testing how do I make local changes (like some kind of buildout clean) and then re-run tests so that the new changes are in the tests?

This is really excellent, please open up a WIP PR. This will take a while to merge in, but it'll allow me to add commits to the branch once you open the PR :)

[svenevs]

I know how to hack out fancyError so that it at least includes caller information btw. I'm going to look into finding a way to flag the tests are being run (e.g. setting an environment variable EXHALE_TESTS_ARE_RUNNING or something).

[tkhyn]

OK. This output directory error highlights another difference between Windows and Linux regarding the XML output directory.

I have just fixed it for Linux in the testing branch, but now it fails on Windows unless the following patch is applied:

diff --git a/exhale/configs.py b/exhale/configs.py
index a33a8a1..cbb31e3 100644
--- a/exhale/configs.py
+++ b/exhale/configs.py
@@ -884,6 +884,9 @@ DEFAULT_DOXYGEN_STDIN_BASE = textwrap.dedent(r'''
     GENERATE_LATEX         = NO
     # Both breathe and exhale need the xml.
     GENERATE_XML           = YES
+    # XML output path must be set to . for the output path to match
+    # breathe_projects[breathe_default_project]
+    XML_OUTPUT = .
     # Set to NO if you do not want the Doxygen program listing included.
     XML_PROGRAMLISTING     = YES
     # Allow for rst directives and advanced functions e.g. grid tables

My understanding of this is zc.buildout is able to pretend to be a sphinx application?!!!

Not really, buildout is 'just' a far superior alternative to virtual environments. Basically all the packages can be installed as 'eggs' in one directory (you may want to create a ~/.buildout/default.cfg file containing this), and every project picks the eggs it needs. No need to have one (or more!) big fat virtual env for every project and install the same dependencies X times. If you look at the created scripts (bin/tests on Unix, bin/tests-script.py on Windows) you'll understand what's going on: the scripts simply add paths of available eggs to sys.path and then runs whatever script you tell it to run. There are hundreds of buildout recipes available to generate all sorts of scripts. I just can't do without it anymore, it's so convenient!

More explanations on how the tests work in the WIP PR #31

[svenevs]

Testing files are on master now, the code coverage badge will be my indication that more testing needs to happen...but there are some deeper changes needed before trying to get closer to 100% ;)