Namespace spanning examples

[ksuess]

Hi! Sybil is awesome! I would like to use Sybil for testing code examples in the documentation of plone.api https://github.com/plone/plone.api. Plone needs a TestCase layer with an app instance to work on. I understand that Sybil can be initialized with a setup(namespace). But it's quite tricky to pass the app untouched from example to example. So after several approaches I ended up with the following suggestion: Allow the sybil.integration.unitttest.TestCase to be initialized not only with a single example, but optional also with a list of examples.

https://github.com/simplistix/sybil/pull/40

What's your opinion?

[ksuess]

BTW my test looks than like:

test_doctests.py

from plone.api.tests.base import INTEGRATION_TESTING
from plone.testing import layered
from plone.testing.zope import Browser
from sybil import Sybil
from sybil.integration.unittest import TestCase
from sybil.parsers.doctest import DocTestParser
from sybil.parsers.myst.codeblock import PythonCodeBlockParser
from unittest import TestSuite

def sybil_setup(namespace):
    """Shared test environment set-up, ran before every test."""
    layer = INTEGRATION_TESTING
    browser = Browser(namespace.get('app', layer['app']))
    if not namespace.get('portal'):
        namespace.update(
            {
                'portal': namespace.get('portal', layer['portal']),
                'request': namespace.get('request', layer['request']),
                'browser': browser,
            },
        )

sb = Sybil(
    parsers=[
        DocTestParser(),
        PythonCodeBlockParser(),
    ],
    path='./doctests',
    pattern='*.md',
    setup=sybil_setup,
)

def _load_tests(loader=None, tests=None, pattern=None):
    doctests_suites = []
    for path in sorted(sb.path.glob('**/*')):
        if path.is_file() and sb.should_parse(path):
            document = sb.parse(path)

            SybilTestCase = type(
                document.path,
                (TestCase,),
                dict(
                    sybil=sb,
                    namespace=document.namespace,
                ),
            )

            testsuite_of_document = TestSuite()
            examples = [example for example in document]
            stc = SybilTestCase(examples)
            stc.namespace.update(
                {
                    'self': stc,
                },
            )
            testsuite_of_document.addTest(stc)
            doctests_suites.append(
                layered(
                    testsuite_of_document,
                    layer=INTEGRATION_TESTING,
                ),
            )
    return TestSuite(doctests_suites)

load_tests = _load_tests

[cjw296]

Test case layers are a thing I haven't bumped into in many many years. Can you give me a minimal example of the layers pattern you're referring to here, with your app instance and a couple of example tests?

It's a shame the Plone community hasn't yet moved to pytest and its fixtures pattern, which appears to now be the standard across just about all of the Python community. With that pattern, you could have your app instance as a session-scoped fixture and nothing more needs doing.

And no, I don't think what you're suggesting is a good idea. Each test evaluates exactly one example, and I don't think that should change.

[ksuess]

Well, yes, pytest with a session scope fixture would be great. I tried now to understand what zope.testrunner is doing, as I for example realized, that the class method sybil.integration.unittest.TestCase.setUpClass is not called when I run a test with zope.testrunner. The method is called in sybil test run. This could be where I should pass the layer from TestCase to TestCase. But it's really hard to understand where to hook in. What is obvious, is that the objects created are passed via namespace from Example to Example, but loose and are not accessible from the layer. So it's not clear to me if I need to force zope.runner to provide a "Session like" layer or if I can let my test provide Sybil with a "namespace updater".

I am working on https://github.com/plone/plone.api/tree/re-add-doctests-sybil Maybe you could spare some time to give me a hint where to force the layer to be passed and updated from Example to Example. For testing and developing I added a simple doc file https://github.com/plone/plone.api/blob/re-add-doctests-sybil/src/plone/api/tests/doctests/testdoc.md My test is https://github.com/plone/plone.api/blob/re-add-doctests-sybil/src/plone/api/tests/test_doctests_sybil_myst.py (testing just the simple doc file) The test is using already an additional PythonCodeBlockParser for MyST markdown. But that should not bother. The new PythonCodeBlockParser is available in the branch mentioned above, as I pull sybil from https://github.com/rohberg/sybil/tree/myst-2-code-block-single-example-testcase

[cjw296]

"The test is using already an additional PythonCodeBlockParser for MyST markdown" - A MySTPythonCodeBlockParser would be a welcome addition to Sybil! If you're able to contribute that as a separate PR, that would be great!

I chucked some comments in https://github.com/rohberg/sybil/commit/243f55b757915047ffc209bc86059ed4e67215f2 to help with this :-)

[cjw296]

Reading this code, what would happen if, instead, you just did:

make_sybil_suite = Sybil(
    parsers=[
        DocTestParser(),
        MySTPythonCodeBlockParser(),
    ],
    path='./doctests',
    pattern='*.md',
    setup=sybil_setup,
).unittest()

def load_tests(loader=None, tests=None, pattern=None):
    suite = make_sybil_suite(loader, tests, pattern)
    return layered(suite, layer=INTEGRATION_TESTING)

[cjw296]

Looking at the code for layered, if you need to get access to the layer inside the examples in your documentation, then there's more work to be done, since Sybil's unittest integration doesn't (currently) provide _dt_test.

[cjw296]

Closing as no followup.

[stevepiercy]

Just an FYI, @ksuess opted to implement this feature in manuel in https://github.com/benji-york/manuel/pull/27. I am not sure if it can be reused for this package, but since it is inspired by manuel...?

[cjw296]

🤷

[cjw296]

@stevepiercy - just landed more extensive support for MyST in Sybil: https://sybil.readthedocs.io/en/latest/myst.html Might be fun to see what could be done about the test case layers thing if the Plone world hasn't yet moved to pytest and fixtures...

[stevepiercy]

Thanks for letting me know. I'll keep Sybil in mind, should the need arise to test code in the Plone 6 docs.