doctest plugin runs when unwanted

[cjw296]

This file fails when run explicitly with pytest:

$ pytest --doctest-glob *.nope README.rst 
...
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
...
...______ [doctest] README.rst __
093         'banner': Required(str),
...
102 >>> print(open('/etc/my_app/config.yaml').read())
UNEXPECTED EXCEPTION: FileNotFoundError(2, 'No such file or directory')

Things work fine if I just pytest in the root directory. The problem is that the doctest plugin is kicking in when it has not been requested, and indeed even when I try and explicitly stop it from working on this file. I use sybil for testing examples in my documentation as it provides for more flexibility.

Here's my full environment:

$ pip freeze
alabaster==0.7.12
atomicwrites==1.3.0
attrs==19.1.0
Babel==2.6.0
bleach==3.1.0
certifi==2019.3.9
chardet==3.0.4
-e git+git@github.com:Simplistix/configurator.git@1090b4db182490e523ca9b7415fd87ba784e25ae#egg=configurator
coverage==4.5.3
docutils==0.14
idna==2.8
imagesize==1.1.0
Jinja2==2.10
MarkupSafe==1.1.1
mock==2.0.0
more-itertools==6.0.0
packaging==19.0
pbr==5.1.3
pkginfo==1.5.0.1
pluggy==0.9.0
py==1.8.0
pyfakefs==3.5.8
Pygments==2.3.1
pyparsing==2.3.1
pytest==4.3.1
pytest-cov==2.6.1
pytz==2018.9
PyYAML==5.1
readme-renderer==24.0
requests==2.21.0
requests-toolbelt==0.9.1
setuptools-git==1.2
six==1.12.0
snowballstemmer==1.2.1
Sphinx==1.8.5
sphinxcontrib-websupport==1.1.0
sybil==1.0.9
testfixtures==6.6.1
toml==0.10.0
tqdm==4.31.1
twine==1.13.0
urllib3==1.24.1
voluptuous==0.11.5
webencodings==0.5.1

[blueyed]

pytest --doctest-glob *.nope README.rst

Haven't investigate, but make sure that *.nope does not get expanded by your shell already, i.e. quote it.

[blueyed]

Related code: https://github.com/blueyed/pytest/blob/033849a31ec7d2b9f0493fa51c77289eab657af1/src/_pytest/doctest.py#L104-L107. Added in acd286f8 initially.

The glob pattern is ignored for txt and rst files given on the command line explicitly.

Why do you specify README.rst explicitly?

[cjw296]

I was very sure that *.nope did not match any files ;-)

"The glob pattern is ignored for txt and rst files given on the command line explicitly." feels like a bug to me. Why specify README.rst explicitly? Same reason people normally specify a subset of their tests, I just wanted to run those tests, not the whole unit test suite.

[blueyed]

Yeah.. but if you specify "README.rst" it runs the (doc)tests from it.

It is like withtestpaths but specifying a file explicitly.

[cjw296]

It evidently does, but why should it? it doesn't when I don't otherwise limit discovery. If I do, pytest some/test_file.py, it doesn't suddenly decide to ignore my plugin configuration and do something I don't want ;-)

To be clear: this is far from a show stopper, but it is a bit of a wart and it would be great to find a way to fix it!

[blueyed]

If I do, pytest some/test_file.py, it doesn't suddenly decide to ignore my plugin configuration and do something I don't want ;-)

But pytest t-foo.py ignores testpaths also.

[nicoddemus]

If you pass a file explicitly on the command-line, pytest will run that file as a test, regardless of test file pattern configuration or testpaths. For example if you have tests on foo.py, running just pytest . won't pick up foo.py, but pytest foo.py will. The same happens for doctests as @blueyed pointed out in the code.

I think the rationale is that if you are explicitly passing the file in the command-line, you are telling pytest to run tests from that file.

Not sure what were the requirements that lead to this, but I don't think we should change this without a good reason because this will certainly break some workflows out there.

[cjw296]

[fixed this comment up as I realised I was misreading some output]

Sorry, but this doesn't feel right. -vv output gives some more detail: it looks like the file is being processed twice, once with the stuff I've configured in conftest.py and then additionally, erroneously, as a doctest:

$ pytest -vvx README.rst 
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /Users/chris/virtualenvs/configurator_37/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/chris/vcs/git/configurator, inifile:
plugins: cov-2.6.1, pyfakefs-3.5.8
collected 14 items                                                                                                                                                    

README.rst::line:26,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [  7%]
README.rst::line:45,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [ 14%]
README.rst::line:66,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [ 21%]
README.rst::line:92,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [ 28%]
README.rst::line:107,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 35%]
README.rst::line:108,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 42%]
README.rst::line:112,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 50%]
README.rst::line:114,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 57%]
README.rst::line:116,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 64%]
README.rst::line:122,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 71%]
README.rst::line:124,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 78%]
README.rst::line:126,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 85%]
README.rst::line:128,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                            [ 92%]
README.rst::README.rst <- /Users/chris/vcs/git/configurator/README.rst FAILED                                                                                   [100%]

Now, if I specify an item within the file, things work as expected:

$ pytest -vvx README.rst::line:26,column:1
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.1, pytest-4.3.1, py-1.8.0, pluggy-0.9.0 -- /Users/chris/virtualenvs/configurator_37/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/chris/vcs/git/configurator, inifile:
plugins: cov-2.6.1, pyfakefs-3.5.8
collected 1 item                                                                                                                                                      

README.rst::line:26,column:1 <- /Users/chris/vcs/git/configurator/README.rst PASSED                                                                             [100%]

====================================================================== 1 passed in 0.08 seconds =======================================================================

The problem is that these tests shared state within a module, they're testing the examples within documentation using Sybil, so I really want to be able to run a single file without the annoying and erroneous doctest stuff kicking in.

How do I do that?

[blueyed]

once with the stuff I've configured in conftest.py

What is that? Do you mean setup.cfg/pytest.ini maybe instead?

I suggest digging into the code maybe yourself (I've left a pointer already).

(btw: with 4.4 it would also display "testpaths: …" in the header, I wonder if that covers this case also?)

[cjw296]

No, I mean conftest.py. I'm afraid I can't follow how displaying "testpaths: …" in the header is related to this.

[blueyed]

I'm afraid I can't follow how displaying "testpaths: …" in the header is related to this.

It is not really, but I just wondered if files for doctests would show up there (like they should).

[cjw296]

Just to be clear: I'm not using doctests and I don't want to be using doctests. I want pytest to stop erroneously treating my non-doctest .rst files as doctests, which it only does if I specify a particular file on the command line.

My currently workaround is to do:

pytest -k README.rst

That works fine, so there's no massive priority on fixing this, but it feels like you're trying to tell me it's not a bug, in which case I will respectfully disagree ;-)

[blueyed]

Try -p no:doctest then maybe (might require pytest 4.4 / current features branch though).

[cjw296]

I already have a workaround thanks...

[blueyed]

And I agree that the collection logic in the doctest plugin is a bit wonky.

As for your workaround: it will collect all files, and then filter based on a pattern, which is slower than collecting just one file.

[cjw296]

Just to note that this is still an issue in pytest 6, but that putting addopts = -p no:doctest in the config file does stop the pytest doctest plugin from getting in the way.

[nicoddemus]

Thanks @cjw296,

Not sure how we could change the defaults so they don't break existing workflows though. Any ideas? If so I suggest we open a new "proposal" with a plan to deprecate automatically running doctests in .rst and .txt files, but in a way that won't get users by surprise (for example, someone upgrades pytest to a version which no longer runs those files as doctests, and doesn't notice).