Doctest with MyST may be broken or incorrectly documented

[sscherfke]

I try to use Sybil with a Markdown file that contains normal Python code block and doctests.

Testing doctests does not work in several ways though and I am not sure whether I am doing something wrong, or whether there is a (documentation) issue in Sybil.

I am using:

• Python 3.10.9
• Sphinx 5.3.0 / 6.1.3
• myst-parser 0.18.1
• Sybil 4.0.0

Minimal example

docs/index.md

# Test

## 1

print(2) 1

2

>>> print(2)
2

3

>>> print(3)
3

4

>>> print(4)
4

### `docs/conf.py`

```python
project = "Test"
author = "Monty"
release = "1.0.0"
version = "1.0"

extensions = [
    "myst_parser",
]
templates_path = ["_templates"]
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]

conftest.py

from doctest import ELLIPSIS

from sybil import Sybil
from sybil.parsers.myst import DocTestDirectiveParser, PythonCodeBlockParser
from sybil.parsers.rest import DocTestParser

pytest_collect_file = Sybil(
    parsers=[
        DocTestParser(optionflags=ELLIPSIS),
        DocTestDirectiveParser(optionflags=ELLIPSIS),
        PythonCodeBlockParser(),
    ],
    patterns=["*.md"],
).pytest()

Problems

1. Sphinx does not recognice the {doctest} directive and will not show it, but the docs state otherwise
2. If I only use the PythonCodeBlockParser, only examples 2 and 3 are collected, but sucessfully tested
3. If I only use the DocTestDirectiveParser, only example 4 is collected, but successfully tested. But it leads to Sphinx errors (see 1.).
4. If I only use the DocTestParser, all four examples are collected, but doctest considers the closing three backticks as part of the example and fails with sth. like 1\n''' != 1`.
5. If I use all parsers (or at least DocTestParser and PythonCodeBlockParser (which I'd like to do in my real project, Sybil fails because of overlapping regions. I can "fix" this by removing the python/{code-block} python from examples 2 and 3, but then I get the same error as in point 4. again.

[cjw296]

1. If you want to use the {doctest} directive, you need to include the doctest extension in your conf.py:

extensions = [
    "myst_parser",
    "sphinx.ext.doctest",
]

[cjw296]

2. This is the approach I'd recommend for the actual example doc you've posted, but I'd change example 1 and 4 to start with "```python".

[cjw296]

5. If you need to use different parser configurations for different files within the same set of documentation, I'd suggest this pattern.

[cjw296]

Let me know if this works for you, if it does, I'll close this issue and cut a release for the #44 fix :-)

[sscherfke]

I’ll see if I can get away with not using the Doctest parser and only use the PythonCodeBlock parser.

Anyways, I think that the docs should be updated to avoid confusion:

• You need the "sphinx.ext.doctest" extension for https://sybil.readthedocs.io/en/latest/myst.html#doctest-directive to work
• Using the DocTestParser and PythonCodeBlock parser at the same time will lead to "overlapping region" errors if you show an interactive shell session within a python code block

Both issues apply to rest documents, too.

And thansk for the quick response. Appart from this issue, Sybil is realy awesome :)

[cjw296]

Both issues apply to rest documents, too.

Oh? Can you give me an example?

[sscherfke]

Test
====

1
-

::

  >>> print(2)
  1

2
-

.. code-block:: python

    >>> print(2)
    2

from doctest import ELLIPSIS

from sybil import Sybil
from sybil.parsers.rest import DocTestParser, PythonCodeBlockParser

pytest_collect_file = Sybil(
    parsers=[
        DocTestParser(optionflags=ELLIPSIS),
        PythonCodeBlockParser(),
    ],
    patterns=["*.rst"],
).pytest()

E   ValueError: <Region start=45 end=93 <bound method PythonCodeBlockParser.evaluate of <conftest.PythonCod
eBlockParser object at 0x105abaec0>>> from line 15, column 1 to line 20, column 1 overlaps <Region start=69
 end=92 <sybil.evaluators.doctest.DocTestEvaluator object at 0x105abad70>> from line 17, column 1 to line 1
9, column 1

And to use .. doctest, you still need the corresponding extension in your conf.py :)

[cjw296]

Okay, the overlap check is doing its job there. You should not put doctest examples inside a .. code-block:: python, I'll put a note in the docs.

I've also added notes highlighting that you need to enable the doctest extension to use those directives.

I'll close this issue out now, feel free to re-open / comment if you feel there's more to, open a new issue if it's not related to the stuff above.

[sscherfke]

Point 4 is still an issue if you use the DocTestParser with markdown documents:

If I only use the DocTestParser, all four examples are collected, but doctest considers the closing three backticks as part of the example and fails with sth. like 1\n''' != 1`.

In mardown documents, the following snippet:

print(3) 3

would result in the assertion assert "3\n```" == "3" which will fail.

[cjw296]

What happens if you do:

print(3) 3

?

[sscherfke]

Then the test passes, but I get an empty, ugly extra line when rendering the docs:

🤪

[cjw296]

I guess use one of the other parser options then...

[sscherfke]

The thing is, that the doctestparser does not recognize codeblocks but only looks for >>> and the corresponding output lines.

Doctest examples are even collected when they are not fenced. But they are then not render correctly.

[cjw296]

It's the ReST doctest parser, it's there as a "may work for you" thing, the docs clearly introduce better alternatives before hand, so I'd suggest using one of them.