search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.62k stars 2.58k forks source link

Selftests sometimes fail on pypy: `TypeError: expected some sort of stmt, but got <_ast.Load object ...>` #11771

Open bluetech opened 6 months ago

bluetech commented 6 months ago

Example: https://github.com/pytest-dev/pytest/actions/runs/7408595776/job/20157160097

This doesn't always happen, maybe 50% of the time. Started in the last few days. I checked if the PyPy version changed during these days, but seems not (it's 3.8.16).

The error is:

 __________________ ERROR collecting testing/_py/test_local.py __________________
/opt/hostedtoolcache/PyPy/3.8.16/x64/lib/pypy3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1023: in _gcd_import
    ???
<frozen importlib._bootstrap>:1000: in _find_and_load
    ???
<frozen importlib._bootstrap>:984: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:680: in _load_unlocked
    ???
.tox/pypy3-xdist/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:166: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
.tox/pypy3-xdist/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:350: in _rewrite_test
    co = compile(tree, strfn, "exec", dont_inherit=True)
E   TypeError: expected some sort of stmt, but got <_ast.Store object at 0x0000000002142880>

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

skirpichev commented 4 months ago

I have same failure on older versions as well. E.g. 7.4.3: https://github.com/mpmath/mpmath/actions/runs/8030300102?pr=759

This happens for all tested in the mpmath pypy versions (3.8-3.10). But most often on 3.10.

skirpichev commented 3 months ago

Another example: https://github.com/pytest-dev/pytest/issues/11168#issuecomment-1714358183

@cfbolz, I don't see a bugreport in pypy issues. Is there any chance this might be related to pytest-dev/pytest#11168 (an fixed in pypy)?

cfbolz commented 3 months ago

@skirpichev it's not impossible that it's the same problem, yes. You can try a nightly PyPy build (or wait for the next release but that is still a bit out).

skirpichev commented 3 months ago

@cfbolz, thanks for a suggestion. I did tests for nightly builds (on 3.8-3.10) and, after ~20 re-runs, I got a TypeError on pypy3.8, which is similar to this issue:

``` ============================= test session starts ============================== platform linux -- Python 3.8.16[pypy-7.3.12-alpha], pytest-8.1.1, pluggy-1.4.0 mpmath backend: python mpmath mp class: mpmath version: 1.4.0a1.dev43+g534bc79 Python version: 3.8.16 (d8e6b37f5231c7b59b2b639a651dd696a3732cc3, Jan 09 2023, 00:30:15) [PyPy 7.3.12-alpha0 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)] rootdir: /home/runner/work/mpmath/mpmath configfile: pyproject.toml testpaths: mpmath, docs plugins: cov-5.0.0, xdist-3.5.0 created: 4/4 workers 4 workers [2294 items] ........................................................................ [ 3%] ........................................................................ [ 6%] ........................................................................ [ 9%] ........................................................................ [ 12%] ........................................................................ [ 15%] ........................................................................ [ 18%] ........................................................................ [ 21%] ........................................................................ [ 25%] ........................................................................ [ 28%] ........................................................................ [ 31%] ........................................................................ [ 34%] ........................................................................ [ 37%] ........................................................................ [ 40%] ........................................................................ [ 43%] ........................................................................ [ 47%] ........................................................................ [ 50%] ........................................................................ [ 53%] ........................................................................ [ 56%] ........................................................................ [ 59%] ........................................................................ [ 62%] ........................................................................ [ 65%] ........................................................................ [ 69%] ........................................................................ [ 72%] ......................................................................... [ 75%] ........................................................................ [ 78%] ........................................................................ [ 81%] ........................................................................ [ 84%] ........................................................................ [ 87%] ........................................x............................... [ 91%] .......................................x................................ [ 94%] ...................................s.................................... [ 97%] ............................................................. [100%] ==================================== ERRORS ==================================== _______________ ERROR collecting mpmath/tests/test_basic_ops.py ________________ ../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) :1023: in _gcd_import ??? :1000: in _find_and_load ??? :984: in _find_and_load_unlocked ??? :680: in _load_unlocked ??? ../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:169: in exec_module source_stat, co = _rewrite_test(fn, self.config) ../../_temp/d367cbdc-8911-4ebe-aab6-2151925ee5c2/pypy-c-jit-106755-d8e6b37f5231-linux64/lib/pypy3.8/site-packages/_pytest/assertion/rewrite.py:353: in _rewrite_test co = compile(tree, strfn, "exec", dont_inherit=True) E TypeError: expected some sort of stmt, but got <_ast.Attribute object at 0x00007f101d7a6920> ============================= slowest 20 durations ============================= 34.04s call mpmath/tests/test_functions2.py::test_bessel_zeros_extra 28.52s call mpmath/tests/test_torture.py::test_asymp[-150-False23] 16.64s call mpmath/functions/hypergeometric.py::mpmath.functions.hypergeometric.hyper 11.94s call mpmath/tests/test_extra_zeta.py::test_zetazero[241389216-97490234.22767118] 10.72s call mpmath/functions/zeta.py::mpmath.functions.zeta.secondzeta 9.91s call mpmath/tests/test_elliptic.py::test_elliptic_integrals 9.25s call mpmath/ctx_fp.py::mpmath.ctx_fp.FPContext.bessely 8.75s call mpmath/functions/hypergeometric.py::mpmath.functions.hypergeometric.hyp3f2 8.59s call mpmath/ctx_fp.py::mpmath.ctx_fp.FPContext.siegelz 8.44s call mpmath/calculus/extrapolation.py::mpmath.calculus.extrapolation.nsum 8.39s call mpmath/tests/test_torture.py::test_asymp[-150-False9] 7.87s call mpmath/functions/zetazeros.py::mpmath.functions.zetazeros.nzeros 7.58s call mpmath/tests/test_torture.py::test_asymp[f_wrapped-1500-False1] 7.53s call mpmath/tests/test_quad.py::test_quadosc 7.50s call mpmath/calculus/quadrature.py::mpmath.calculus.quadrature.QuadratureMethods.quad 7.45s call mpmath/tests/test_torture.py::test_asymp[f_wrapped-90-False] 7.27s call mpmath/tests/test_torture.py::test_asymp[f-5000-True1] 7.22s call mpmath/tests/test_torture.py::test_asymp[-150-False27] 7.00s call mpmath/tests/test_torture.py::test_asymp[-150-False37] 6.74s call mpmath/tests/test_torture.py::test_asymp[-150-False36] =========================== short test summary info ============================ ERROR mpmath/tests/test_basic_ops.py - TypeError: expected some sort of stmt, but got <_ast.Attribute object at 0x00007f101d7a6920> ======= 2291 passed, 1 skipped, 2 xfailed, 1 error in 203.43s (0:03:23) ======== ```

Summary page: https://github.com/mpmath/mpmath/actions/runs/8491432157?pr=772

Maybe this fix isn't merged yet to the 3.8 branch (as it was stated in your blog post)? If so, I'll continue testing.

cfbolz commented 3 months ago

Yes, the 3.8 variants are not maintained any more, you need to try 3.9 or 3.10. Thanks for trying this out!

skirpichev commented 3 months ago

Well, I did ~50 re-runs of mpmath's tests on pypy3.9 and pypy3.10 - without any failures. Given the previous statistics, probably there is a chance that this issue was fixed in PyPy's nightly builds. I'll switch regular tests for pypy3.9/10 to them.

Thanks for bugfix!

cfbolz commented 3 months ago

Well, I did ~50 re-runs of mpmath's tests on pypy3.9 and pypy3.10 - without any failures.

thank you for trying this, that's very valuable feedback!

webknjaz commented 1 day ago

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

During the sprints, we were seeing 3–4 variants of different AST object reprs there. Hopefully, bumping PyPy in CI will fix this. Ronny suggested this, but nobody got to actually doing it...

webknjaz commented 1 day ago

FTR @LilyFoote linked https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html in https://github.com/pytest-dev/pytest/issues/11168#issuecomment-2023988486 but that reference never got cross-posted into this issue.

webknjaz commented 1 day ago

Action item for whoever gets to prepare the PR:

x612skm commented 1 day ago

Working on this issue

webknjaz commented 19 hours ago

Sometimes it says TypeError: expected some sort of stmt, but got <_ast.Load object at 0x000000000365d040> instead.

I even saw a case of NameError, which seems to have the same root cause:

 E   NameError: name '@py_assert2' is not defined

(https://github.com/pytest-dev/pytest/actions/runs/9746885084/job/26898335196#step:6:97)