Docutils backend

[tony]

I think having docutils as a backend parser will make the experience more resilient + make myst easier to integrate. But I am not all the way through the APIs yet.

I have a new branch bootstrapping at https://github.com/tony/pytest-sphinx/commits/docutils-parser. Super preliminary.

I notice now you've already attempted this approach: https://github.com/thisch/pytest-sphinx/tree/docutils-experimental. What made you pause on this approach?

Both look similar we're both using sphinx's doctest.py directives. We even have the same conditional (except you used .traverse() and I use findall()), also I'm looking back at pytest's doctest.py

[tony]

Another reminder I'm on top of this and haven't forgotten this.

I'm going to read the doctest, sphinx, pytest, etc. related code head to toe.

[tony]

doctest:

• cpython: cpython/doctest.py, docs
• pytest: _pytest/doctest.py, tests
• sphinx: sphinx/doctest.py, docs

[tony]

@thisch Another reminder I haven't forgotten this. I may not work on it for the next 3 weekends.

This weekend I am going to work on cleaning up another project (just to decompress and get that work done).

[twmr]

@tony Sorry for the long silence, I just came back from vacation.

Thx a lot for suggesting a docutils backend! This is a brilliant idea, which I also had when I started the project, but I didn't manage to dig through the docutils API - I decided to implement a quick, but not very robust solution, which I wanted to replace later.

What made you pause on this approach?

As written above, I didn't manager to fully dig through the docutils API. I hope you have more succes than me ;)

[tony]

@thisch Thank you for getting back!

Thx a lot for suggesting a docutils backend! This is a brilliant idea, which I also had when I started the project, but I didn't manage to dig through the docutils API - I decided to implement a quick, but not very robust solution, which I wanted to replace later.

That is what I gather! It's really tricky to do without an intimate understanding of the docutils and doctest APIs.

As written above, I didn't manager to fully dig through the docutils API. I hope you have more succes than me ;)

Let's see how it goes. No one else in all of python has managed a bullet-proof version of this.

Question: A bit of the complexity comes from testcode and testoutput from sphinx.ext.doctest.

That's not standard doctest. That's something sphinx.ext.doctest bolts on. Somewhat rare: ".. testcode::" and ".. testoutput::"

Are you willing to forgo those?

Technically - what I am looking for is a "docutils-doctest" / "docutils-doctest-pytest", I don't care about sphinx's related stuff - it's nice to have if people need it, but it's non standard. It's something I'd add later - but is it really valuable enough to you? (maybe it is)

I'd propose doctest + docutils working - so the scope isn't overwhelming. But that may have friction with the project's goals.

[twmr]

Question: A bit of the complexity comes from testcode and testoutput from sphinx.ext.doctest.

Yes, I agree. Those directives, which are not standard, make it complicated to finish/finalize the implementation of the pytest plugin for sphinx.ext.doctest.

Are you willing to forgo those?

Well, then we would have to rename the project from pytest-sphinx to sth else and the sphinx-specific part could be moved into sphinx-core (see https://github.com/sphinx-doc/sphinx/pull/10393#issuecomment-1112050867)

what I am looking for is a "docutils-doctest" / "docutils-doctest-pytest", I don't care about sphinx's related stuff

What would this plugin do that is currently not done by pytest-core?

but is it really valuable enough to you? (maybe it is)

Not so. The rst files in our projects could all be changed s.t. the std >>> doctest is used without the .. doctest:: directive. The only benefit pytest-sphinx IMO provides for us is that ..testcode:: + .. testoutput:: makes the rst files easier to read (we don't use the grouping feature and the options/skipping features that are supported by sphinx.ext.doctest).

[tony]

Well, then we would have to rename the project from pytest-sphinx to sth else

Yes it looks like that's what it'd be. It'd really be a separate project then

and the sphinx-specific part could be moved into sphinx-core (see https://github.com/sphinx-doc/sphinx/pull/10393#issuecomment-1112050867)

We'll see if sphinx-core accepts any pieces of this project, in whole or part.

IMO I'm surprised sphinx.ext.doctest is even part of sphinx. It doesn't really need sphinx, since it can be assumed the tests are in the same file.

What would this plugin do that is currently not done by pytest-core?

At the end of the day: its really creating a doctest parser using docutils to parse files.

Parse .rst (and later, .md) directives / nodes into doctest.Example's.

After that, figure out how to glue it into pytest.doctest (docs, source). That may involve a PR to pytest.doctest or overriding their fixtures, not sure yet.

Not so. Our projects could all be changed s.t. the std >>> doctest is used. The only benefit IMO for us is that ..testcode:: + .. testoutput:: makes the rst files easier to read

Good to know

(we don't use the grouping feature and the options/skipping features that are supported by sphinx.ext.doctest).

Also good to know.

"grouping" - whatever those do - aren't part of standard library doctest (docs, source, typings)

[tony]

@thisch

https://github.com/astropy/pytest-doctestplus, pytestdoctest_plus.sphinx.doctestplus

via https://docs.pytest.org/en/7.1.x/how-to/doctest.html#alternatives

• pytest-doctestplus: provides advanced doctest support and enables the testing of reStructuredText (“.rst”) files.

[tony]

sybil:

https://sybil.readthedocs.io/en/latest/parsers.html

via https://docs.pytest.org/en/7.1.x/how-to/doctest.html#alternatives

Sybil: provides a way to test examples in your documentation by parsing them from the documentation source and evaluating the parsed examples as part of your normal test run.

[tony]

@thisch

https://github.com/rstcheck/rstcheck

v5 and below, one file: rstcheck

v6.0+:

• rstcheck-core, rstcheck_core/checker.py
• rstcheck-sphinx

[tony]

None of the existing solutions have the architecture I'm expecting:

• Entry points: Pure doctest, or optionally pytest
• API: doctest + docutils

All of them have an issue:

• Using plain old regexes rather than docutils to fetch tests
• Creating a totally separate test framework that's not doctest or pytest, or something that's genericized and master of none
• Using sphinx when not necessary to, including using Builder

The idea I have is pytest docs/ should just be able to collect doctests from rst and markdown.

[twmr]

Thx a lot for keeping an eye on related packages in the open source community! It's a pity that none of them does what we need for pytest-sphinx.

I've seen that you are still working on https://github.com/tony/pytest-sphinx/commits/docutils-parser can you give me an update on the status of this branch? Does it make sense to maybe reach out to the docutils developers in case we need help?

The idea I have is pytest docs/ should just be able to collect doctests from rst and markdown.

I totally agree.

[tony]

I've seen that you are still working on https://github.com/tony/pytest-sphinx/commits/docutils-parser can you give me an update on the status of this branch?

I'm just experimenting / familiarizing with doctest now:

pip install myst-parser docutils

# markdown (docutils w/ myst-parser)
python -m doctest_docutils examples/test.md -v

# reStructuredText (docutils)
python -m doctest_docutils examples/test.rst -v

# both files
python -m doctest_docutils examples/test.rst examples/test.md -v

# debug logging
python -m doctest_docutils examples/test.rst examples/test.md -v --log-level DEBUG

If you have entr(1), here's what I use:

ls *.py examples/*.* tests/*.py | entr -s 'python -m doctest_docutils examples/test.rst examples/test.md -v --log-level DEBUG'

Does it make sense to maybe reach out to the docutils developers in case we need help?

we can consider that if we need to but I feel I have a good control over docutils parts now. The doctest part itself, e.g. line numbers and making sure we're doctest vanilla compatible + pytest compatible (and the code is clean) is what's next.

[tony]

@thisch I have #38, this is still a work in progress

It has example commands

Questions for discussion are:

• Is my approach aligned with pytest-sphinx and what you want it to do, or is this a new thing?

  In general, since that way I approach it may be radically different from what you're expecting. Or maybe it's okay.

• Is the dropping of sphinx related things okay?

  • Is dropping .. testcode::, .. testoutput:: okay?
  • Is dropping options .. docutils::'s, e.g. :hide:, :options:, :pyversion:, etc okay?

Technically, my desire / ambition is really to support standard doctest + pytest via docutils (no sphinx).

The rationale I have of dropping sphinx-related parts is they add a lot of complexity compared to the value they bring. I could add them - but it'd involve going much deeper to override: DocTestParser._parse_example(). My solution uses doctest's own parser, I made a finder - DocutilsDocTestFinder - to find directives, then it lets vanilla DocTestParser do the work.

[twmr]

We unfortunately can't drop support for the sphinx directives that are currently not supported by your PR, as this would silently remove support from testing code in those directives. We can however bundle both your code as well as my old code in one plugin until your code is added to pytest-core (sse https://github.com/pytest-dev/pytest/discussions/10155).

[tony]

@thisch Here is what I'm doing at the moment

https://github.com/git-pull/gp-libs, atm it's in src as doctest_docutils.py and pytest_doctest_docutils.py

What I intend to do is develop / dogfood / incubate there and then collaborate on what to bring over here.

We unfortunately can't drop support for the sphinx directives that are currently not supported by your PR,

I'm not against those directives - other than being stretched then just getting regular doctest's bullet-proofed.

What I am thinking is, after doctest_docutils is functioning, have a doctest_sphinx module that handles those sophisticated cases.

testcode is nice: It's clean, it doesn't need the ... line continuations.

My efforts in doctest_docutils will help sphinx directives, but I may not be the champion to do it. My projects don't use those directives. I can't battle test them in the same way against my own open source projects

[tony]

@thisch I said I'd reach out with an update. This is from across the pond:

My projects are now all using gp-lib's pytest-doctest-docutils

Example: libtmux

libtmux (source):

Documentation w/ docutils:

README.md (raw)

Note: pytest README.md requires you have a conftest.py directly in the project root. In this case README.md is done via docs/index.md via include, so docs/conftest.py is ran

docs/topics/traversal.md (raw)

Configuration:

Doctests support pytest fixtures through doctest_namespace

See add_doctest_fixtures() in src/libtmux/conftest.py