Enable documentation testing

[astrofrog]

@embray @eteq @cdeil - how do we enable the testing of examples in documentation? Should it just work? (If so, it doesn't, so this may indicate a bug in the testing framework)

[cdeil]

You have to set doctest_plus = enabled in setup.config (see e.g. photutils/setup.cfg).

Looks like this is not documented: http://astropy.readthedocs.org/en/latest/search.html?q=doctest_plus&check_keywords=yes&area=default

[astrofrog]

I tried that but it didn't seem to work, it just tests the docstrings, but not the actual rst files.

[astrofrog]

As a side note, there are a couple of failures in docstrings for array_utils when I enable doctest_plus

[embray]

It should just work. How are you running the tests?

[astrofrog]

python setup.py test

It doesn't seem to be working though - can you give it a try to see if it works for you?

[embray]

I put doctest_plus = enabled under the [pytest] section in setup.cfg and it worked for me:

../../../internal/1/root/src/imageutils/docs/index.rst .
../../../internal/1/root/src/imageutils/docs/imageutils/index.rst .
../../../internal/1/root/src/imageutils/docs/imageutils/normalization.rst .

There's still an open issue somewhere to not require that. Originally it was for compatibility with existing projects that might want to work toward becoming an affiliated package but not necessarily use our package template or test framework immediately. But in practice that seems to be a non-issue, so my doctest plugin should probably just be enabled by default. But that's another issue.

[astrofrog]

That's weird, it's not working for me. I get:

imageutils/array_utils.py FF.
imageutils/normalization/tests/test_interval.py .........
imageutils/normalization/tests/test_norm.py .....
imageutils/normalization/tests/test_stretch.py ...........................................................................
imageutils/normalization/tests/test_ui.py ....
imageutils/tests/test_array_utils.py ............
imageutils/tests/test_sampling.py ............

I'll dig around.

[astrofrog]

Oh sorry, I should read more closely:

Running doctests in .rst files is not supported on Python 3.x

Ignore me :)

[cdeil]

Can we make full Python 3 support in Astropy (including Sphinx build and docs tests) a priority on the Trello board for Astropy 1.0? It's one of the very few reasons for me to have to go back to Python 2 and it would just be so nice to be able have full Python 3 support for Astropy and affiliated package development.

[astrofrog]

@cdeil - the sphinx builds should work now, but yes we should make the tests work too. Actually the reason they don't is because of things like print statements in the docs. If you have something like print(x, y) it will give different output in Python 2 and 3. Having said that, we should be able to (in the worst case) create a +PRINT directive that knows to ignore differences in the presence/absence of brackets. Anyway, feel free to add a card to trello!

[embray]

That's something I've been wanting too. Lately I've been neglecting to run some doctests because of this, and breaking them without knowing.

Unfortunately it may be a non-trivial problem since the doctests will have to work in all Python versions, but still be readable and not involve any multilingual cruft. At the same time automatic translation (i.e. 2to3) can't be fully relied on either.

At the same time, not all doctests (in fact maybe a minority) require any significant changes to be cross-language. In astropy/astropy@60fa98e26fd63f5d1d78e3f1a27263b0c0e6dda3 I added some initial support for environment markers on the doctest-skip:: directive. Currently it only supports skipping tests on Windows because that was the case I needed it for. But there was a note in there about supporting a full range of environment markers, and that would include Python version as well.

So with that, we could enable most doctests on Python 3, and just skip the few that don't work. Eventually should also just make Python 3 the default... :)

[astrofrog]

@embray - what do you think about my idea regarding print statements? If we write the test as:

>>> print(a, b)
1, 2

then it will read ok, and of course on Python 2 it will give (1, 2), but if we can add a directive that ignores brackets, it will pass.

[embray]

Can't we just implicitly use from __future__ import print_function in the doctests?

[astrofrog]

If we can do that, even better! I'm not familiar with the pytest plugin that is doing the sphinx docs testing, but if you think this would be easy, then can you create a branch with that fix? We could then try and spread out the work of actually fixing the doctests, and open PRs to your branch (we can probably fix the print statements automatically though).