search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.01k stars 2.67k forks source link

Test works normally but fails under pytest with assertion rewriting #4412

Closed oscarbenjamin closed 5 years ago

oscarbenjamin commented 5 years ago

Given a test file p.py containing:

def test_add():
    add = lambda *t: sum(t)
    l = range(8)
    e = iter(l)
    assert sum(l[:4]) == add(*[next(e) for j in range(4)])

the test doesn't work under pytest with assertion rewriting. Note that the expression on the rhs of == will return a different result when evaluated a second time. The first time it is evaluated it will give 6 and the second time 22.

The test passes (as expected) when imported normally:

>>> import p
>>> p.test_add()

It also works under pytest without assertion rewriting

$ pytest p.py --assert=plain
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.6.2, pytest-3.10.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/enojb/current/sympy/sympy, inifile:
plugins: faulthandler-1.5.0
collected 1 item                                                                                                                                             

p.py .                                                                                                                                                 [100%]

================================================================== 1 passed in 0.01 seconds ==================================================================

However it fails under pytest with assertion rewriting

$ pytest p.py
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.6.2, pytest-3.10.0, py-1.7.0, pluggy-0.8.0
rootdir: /Users/enojb/current/sympy/sympy, inifile:
plugins: faulthandler-1.5.0
collected 1 item                                                                                                                                             

p.py F                                                                                                                                                 [100%]

========================================================================== FAILURES ==========================================================================
__________________________________________________________________________ test_add __________________________________________________________________________

    def test_add():
        add = lambda *t: sum(t)
        l = range(8)
        e = iter(l)
>       assert sum(l[:4]) == add(*[next(e) for j in range(4)])
E       assert 6 == 22
E        +  where 6 = sum(range(0, 4))
E        +  and   22 = <function test_add.<locals>.<lambda> at 0x10454e048>(*[0, 1, 2, 3])

p.py:6: AssertionError
================================================================== 1 failed in 0.08 seconds ==================================================================
RonnyPfannschmidt commented 5 years ago

what's even more surprising is, that the debugged values from the arguments are actually correct at least for display

asottile commented 5 years ago

Here's the rewritten code:


import builtins as @py_builtins
import _pytest.assertion.rewrite as @pytest_ar

def test_add():
    add = (lambda *t: sum(t))
    l = range(8)
    e = iter(l)
    @py_assert1 = l[:4]
    @py_assert3 = sum(@py_assert1)
    @py_assert7 = [next(e) for j in range(4)]
    @py_assert9 = add(*[next(e) for j in range(4)])
    @py_assert5 = (@py_assert3 == @py_assert9)
    if (not @py_assert5):
        @py_format11 = (@pytest_ar._call_reprcompare(('==',), (@py_assert5,), ('%(py4)s\n{%(py4)s = %(py0)s(%(py2)s)\n} == %(py10)s\n{%(py10)s = %(py6)s(*%(py8)s)\n}',), (@py_assert3, @py_assert9)) % {
            'py0': (@pytest_ar._saferepr(sum) if (('sum' in @py_builtins.locals()) or @pytest_ar._should_repr_global_name(sum)) else 'sum'),
            'py2': @pytest_ar._saferepr(@py_assert1),
            'py4': @pytest_ar._saferepr(@py_assert3),
            'py6': (@pytest_ar._saferepr(add) if (('add' in @py_builtins.locals()) or @pytest_ar._should_repr_global_name(add)) else 'add'),
            'py8': @pytest_ar._saferepr(@py_assert7),
            'py10': @pytest_ar._saferepr(@py_assert9),
        })
        @py_format13 = (('' + 'assert %(py12)s') % {
            'py12': @py_format11,
        })
        raise AssertionError(@pytest_ar._format_explanation(@py_format13))
    @py_assert1 = @py_assert3 = @py_assert5 = @py_assert7 = @py_assert9 = None

retrieved with this patch:

         state.trace("failed to parse: %r" % (fn,))
         return None, None
     rewrite_asserts(tree, fn, config)
+    import astunparse
+    with open(fn.strpath + '.rewritten', 'w') as f:
+        f.write(astunparse.unparse(tree))
     try:
         co = compile(tree, fn.strpath, "exec", dont_inherit=True)
     except SyntaxError:

Here's an ever-so-slightly shorter reproduction:

def test():
    f = lambda x: x
    x = iter([1, 2, 3])
    assert 2 * next(x) == f(*[next(x)])
import builtins as @py_builtins
import _pytest.assertion.rewrite as @pytest_ar

def test():
    f = (lambda x: x)
    x = iter([1, 2, 3])
    @py_assert0 = 2
    @py_assert4 = next(x)
    @py_assert6 = (@py_assert0 * @py_assert4)
    @py_assert9 = [next(x)]
    @py_assert11 = f(*[next(x)])
    @py_assert7 = (@py_assert6 == @py_assert11)
    if (not @py_assert7):
        @py_format13 = (@pytest_ar._call_reprcompare(('==',), (@py_assert7,), ('(%(py1)s * %(py5)s\n{%(py5)s = %(py2)s(%(py3)s)\n}) == %(py12)s\n{%(py12)s = %(py8)s(*%(py10)s)\n}',), (@py_assert6, @py_assert11)) % {
            'py1': @pytest_ar._saferepr(@py_assert0),
            'py2': (@pytest_ar._saferepr(next) if (('next' in @py_builtins.locals()) or @pytest_ar._should_repr_global_name(next)) else 'next'),
            'py3': (@pytest_ar._saferepr(x) if (('x' in @py_builtins.locals()) or @pytest_ar._should_repr_global_name(x)) else 'x'),
            'py5': @pytest_ar._saferepr(@py_assert4),
            'py8': (@pytest_ar._saferepr(f) if (('f' in @py_builtins.locals()) or @pytest_ar._should_repr_global_name(f)) else 'f'),
            'py10': @pytest_ar._saferepr(@py_assert9),
            'py12': @pytest_ar._saferepr(@py_assert11),
        })
        @py_format15 = (('' + 'assert %(py14)s') % {
            'py14': @py_format13,
        })
        raise AssertionError(@pytest_ar._format_explanation(@py_format15))
    @py_assert0 = @py_assert4 = @py_assert6 = @py_assert7 = @py_assert9 = @py_assert11 = None
asottile commented 5 years ago

4414 has my proposed fix for this!

oscarbenjamin commented 5 years ago

Thanks for the quick work!

oscarbenjamin commented 5 years ago

So it turns out that the fix in #4414 works for the simplified test case I brought to this issue but not for my actual problem which is this function here

You can reproduce my original problem with

$ git clone https://github.com/sympy/sympy.git
$ cd sympy/
$ pytest sympy/integrals/tests/test_integrals.py -k test_series

The function in question looks like

def test_series():
    from sympy.abc import x
    i = Integral(cos(x), (x, x))
    e = i.lseries(x)
    assert i.nseries(x, n=8).removeO() == Add(*[next(e) for j in range(4)])
asottile commented 5 years ago

It seems to work for me (?)

$ pytest sympy/integrals/tests/test_integrals.py -k test_series
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-4.0.1.dev17+ga281d662, py-1.7.0, pluggy-0.8.0
architecture: 64-bit
cache:        yes
ground types: python 

rootdir: /tmp/sympy, inifile:
collected 138 items / 137 deselected                                           

sympy/integrals/tests/test_integrals.py .                                [100%]

=================== 1 passed, 137 deselected in 1.66 seconds ===================

Are you sure you're using the bleeding edge version of pytest? we haven't made a release yet with that fix

RonnyPfannschmidt commented 5 years ago

@oscarbenjamin did you clear your pyc file cache?

@asottile we might need an extra magic marker so pytest can learn to distinguish between different versions of its ast rewriting

oscarbenjamin commented 5 years ago

Thanks guys! I tested with a fresh clone of sympy and pytest and it worked fine.

Going back to my previous setup I fixed it by deleting __pycache__:

$ pytest sympy/integrals/tests/test_integrals.py -k test_series
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.7.1, pytest-4.0.1.dev15+gcdbe2299, py-1.7.0, pluggy-0.8.0
architecture: 64-bit
cache:        yes
ground types: python 

rootdir: /Users/enojb/current/sympy/sympy, inifile:
plugins: xdist-1.24.1, forked-0.2
collected 138 items / 137 deselected                                                                                                                         

sympy/integrals/tests/test_integrals.py F                                                                                                              [100%]

========================================================================== FAILURES ==========================================================================
________________________________________________________________________ test_series _________________________________________________________________________

    def test_series():
        from sympy.abc import x
        i = Integral(cos(x), (x, x))
        e = i.lseries(x)
>       assert i.nseries(x, n=8).removeO() == Add(*[next(e) for j in range(4)])
E       assert -x**7/5040 + x**5/120 - x**3/6 + x == -x**15/1307674368000 + x**13/6227020800 - x**11/39916800 + x**9/362880
E        +  where -x**7/5040 + x**5/120 - x**3/6 + x = <bound method Add.removeO of x - x**3/6 + x**5/120 - x**7/5040 + O(x**9)>()
E        +    where <bound method Add.removeO of x - x**3/6 + x**5/120 - x**7/5040 + O(x**9)> = x - x**3/6 + x**5/120 - x**7/5040 + O(x**9).removeO
E        +      where x - x**3/6 + x**5/120 - x**7/5040 + O(x**9) = <bound method Expr.nseries of Integral(cos(x), (x, x))>(x, n=8)
E        +        where <bound method Expr.nseries of Integral(cos(x), (x, x))> = Integral(cos(x), (x, x)).nseries
E        +  and   -x**15/1307674368000 + x**13/6227020800 - x**11/39916800 + x**9/362880 = Add(*[x, -x**3/6, x**5/120, -x**7/5040])

sympy/integrals/tests/test_integrals.py:916: AssertionError
                                                                       DO *NOT* COMMIT!                                                                       
========================================================== 1 failed, 137 deselected in 1.34 seconds ==========================================================
$ rm -r sympy/integrals/tests/__pycache__/
$ pytest sympy/integrals/tests/test_integrals.py -k test_series
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.7.1, pytest-4.0.1.dev15+gcdbe2299, py-1.7.0, pluggy-0.8.0
architecture: 64-bit
cache:        yes
ground types: python 

rootdir: /Users/enojb/current/sympy/sympy, inifile:
plugins: xdist-1.24.1, forked-0.2
collected 138 items / 137 deselected                                                                                                                         

sympy/integrals/tests/test_integrals.py .                                                                                                              [100%]

========================================================== 1 passed, 137 deselected in 1.83 seconds ==========================================================

So yes, this issue is fixed.