[FeatureRequest] Add --doctest-only to only collect doctests

[dhirschfeld]

When using pytest on CI you often want to test doctests / unittests / intergration tests separately. To enable this usecase it would be good to have a --doctest-only flag to enable running only the doctests

[dhirschfeld]

xref: https://github.com/astropy/pytest-doctestplus/issues/14

[RonnyPfannschmidt]

based on context i am wondering if we should perhaps move towards making doctestplus the default doctest implementation

if memory serves me right, none of the core devs of pytest are deeply involved or happily using doctest

[nicoddemus]

if we should perhaps move towards making doctestplus the default doctest implementation

You mean integrating it into the core?

[RonnyPfannschmidt]

@nicoddemus no, taking it out ^^

[dhirschfeld]

I'm not fussed where it's implemented - I just opened an issue here since pytest already has some doctest functionality and this would be a great enhancement IMHO.

I'd be happy to install doctestplus if they implemented this - it would be it's killer feature for me!

[RonnyPfannschmidt]

i started a conversation with the relevant maintainers

[bluetech]

Because doctest is implemented as a logically independent plugin, there are two implementation strategies/perspectives for achieving this:

In doctest plugin

The doctest plugin adds a --doctest-only flag, implements a hook to filter out any non-doctest items.

Advantages:

• --doctest-only makes it easy to find for people looking to achieve this particular aim (run only doctests)
• Can be implemented/isolated in the doctest plugin

Disadvantages:

• Useless work - regular items are collected only to be discarded
• Solution is very specific to doctests

In python plugin

The python plugin adds a flag to disable its own collection, e.g. --no-python-modules (name modeled after --doctest-modules, but other options can be --no-py-modules, --no-py-collect, ...)

Advantages/Disadvantages: basically the opposite of previous one.

---

While I think it made perfect sense for pytest-doctestplus to implement the first solution, I think for pytest the second solution is preferable.

[bluetech]

BTW, reading the pytest-doctestplus code, what they do is ignore all test_*.py files in --doctest-only mode. But test_*.py can also contain doctests, so IMO this is not very good. But currently they have no alternative if they want to avoid collecting the files/importing the modules (rather than collecting + discarding the items).