Erotemic / xdoctest

A rewrite of Python's builtin doctest module (with pytest plugin integration) with AST instead of REGEX.
Apache License 2.0
205 stars 12 forks source link

Fix test failure if pytest's flaky plugin is installed #104

Closed mgorny closed 3 years ago

mgorny commented 3 years ago

Disable pytest's flaky plugin in test_simple_pytest_import_error_cli in order to fix a test failure due to it mangling pytest's return code on import error.

Erotemic commented 3 years ago

What environment was this error caused in? None of the dashboards seem to have caught it. I'm happy to merge this, but I'd also like to reproduce the error so I can understand it and prevent similar errors in the future.

(I'm also wondering why the CircleCI dashboards aren't running, but maybe they will if I merge this into the development branch for 0.15.9, probably a permission issue.).

mgorny commented 3 years ago

It happens if flaky is installed on the system. I suppose you could catch it via CI if you install that package in addition to the dependencies.

Erotemic commented 3 years ago

I was able to reproduce this. It looks like with flaky installed the error code returned is a 3 instead of 1:

===================================================================================== FAILURES =====================================================================================
_______________________________________________________________________ test_simple_pytest_import_error_cli ________________________________________________________________________

    def test_simple_pytest_import_error_cli():
        """
        This test case triggers an excessively long callback in xdoctest <
        dev/0.15.7

        xdoctest ~/code/xdoctest/testing/test_pytest_cli.py test_simple_pytest_import_error_cli

        import sys, ubelt
        sys.path.append(ubelt.expandpath('~/code/xdoctest/testing'))
        from test_pytest_cli import *  # NOQA
        """
        module_text = utils.codeblock(
            '''
            # There are lines before the bad line
            import os
            import sys
            import does_not_exist

            def module_func1():
                """
                This module has a doctest

                Example:
                    >>> print('hello world')
                """
            ''')
        temp_module = util_misc.TempModule(module_text, modname='imperr_test_mod')
        command = sys.executable + ' -m pytest -v -s --xdoctest-verbose=3 --xdoctest ' + temp_module.dpath
        print(command)
        info = cmd(command)
        print(info['out'])

        # info = cmd('pytest --xdoctest ' + temp_module.modpath)
        # print(info['out'])

>       assert info['ret'] == 1
E       assert 3 == 1

testing/test_pytest_cli.py:78: AssertionError
============================================================================= short test summary info ==============================================================================
FAILED testing/test_pytest_cli.py::test_simple_pytest_import_error_cli - assert 3 == 1
=========================================================================== 1 failed, 4 passed in 3.26s ============================================================================

Perhaps we could just change the assert to check that assert info['ret'] != 0 and not have to explicitly call out a specific plugin, which I would like to avoid, xdoctest should try to avoid conflicting with other established plugins.

mgorny commented 3 years ago

I'm happy with any solution that makes the tests pass.

Erotemic commented 3 years ago

If you make modify the PR to make that change I'll merge it and push out a new patch.

Erotemic commented 3 years ago

Thanks for this bug report! I pushed up the appropriate commits to this branch. Merging.

Do you need a new version of xdoctest with this fix released on pypi anytime soon? If so I can push one up and bump the patch version, if not I might wait a bit before making the new release.