search

simplistix / sybil

Automated testing for the examples in your documentation.
https://sybil.readthedocs.io/en/latest/
Other
74 stars 14 forks source link

Support for `testcode` and `testoutput` in `DocTestDirectiveParser`s #110

Closed davidselassie closed 7 months ago

davidselassie commented 9 months ago

I'd love to have support for the testcode and testoutput directives from sphinx.ext.doctest since we already have a good amount of long-form documentation that uses them and like to be able to add some explanatory text before the output, which is harder to do with doctest blocks. You said in the docs adding more doctest features wouldn't be hard, so thought I'd ask! Thanks for your work!

cjw296 commented 9 months ago

Can you point me at some of this documentation (specifically it's source code!) so I have some examples to code against?

cjw296 commented 8 months ago

@davidselassie - nudge ^^

davidselassie commented 7 months ago

A little more backstory: I had previously written a custom pytest test generator https://github.com/bytewax/bytewax/blob/367ab94f82cdc9c54b3f701def8f7123c1ba6a4a/docs/conftest.py which does something almost identical to https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html#directive-testcode and {testcode} / {testoutput}, but it uses python / {testoutput} block pairs. We weren't using Sphinx, so it would have been more complicated to get the code rendering working correctly for {testcode} blocks so just piggy backed on normal python blocks.

So https://github.com/bytewax/bytewax/blob/73d8e2fdb9caccb5a6964f573b8c74676790dc1d/docs/articles/concepts/dataflow-programming.md and https://github.com/bytewax/bytewax/blob/73d8e2fdb9caccb5a6964f573b8c74676790dc1d/docs/articles/concepts/joins.md are two documentation files that use {testoutput} blocks, but they are paired with a preceding python block to work with our other documentation renderer, so not exactly the same format.

My goal was to use Sybil eventually instead of the custom test generator I wrote because it supports other test syntaxes (like >>> doctests) which my custom test generator doesn't support. I'll go and try to make versions which use the proper pairing, and update my custom generator to know that the tests are correct, but hopefully the above is enough to get started.

FWIW, it looks like https://github.com/sphinx-doc/sphinx/blob/b7f708dc634732aa90978e1c141ba6bd2af7ee84/tests/roots/test-ext-doctest/doctest.txt is the source Sphinx itself uses to test those directives, but they're RST not Markdown. Could be another good source of tests.

cjw296 commented 7 months ago

Hmm, okay, so its not actually doctest you're after but support for your own auto-generated test stuff? I'm not sure it makes sense to support testcode and testoutput in DocTestDirectiveParser itself, since, as best I can tell, sphinx.ext.doctest is ReST only; I can't find any mention of it in the MyST docs, can you point me to them?

I'm also concerned about adding official support for these directives given the complication of "groups", "skipif", "pyversion" and whatever else is lurking there.

I also appear to have been mistaken that sphinx.ext.doctest was deprecated, I think?

If all of the above is correct, why not knock up your own parsers using the appropriate directive lexers, etc? If that would work, I just need to crank out a release with the nested Markdown block parsing...

cjw296 commented 7 months ago

Okay, closing for now as there's been now followup. I'll crank out out the promised release in the next few days...

cjw296 commented 6 months ago

https://pypi.org/project/sybil/6.1.0/ out now, let me know how you get on with creating your own parsers! :-)