Python 3.11: comments / docstrings / future imports at the beginning of code cells cause crashes

[jerry-git]

Versions: Python 3.11 and ipytest==0.13.1.

Comments at the beginning of code cells cause crashes after ipytest.autoconfig() has been executed. Here's a simple example:

FWIW, I tried running the ipytest test suite with 3.11, no failures.

[chmp]

Hey @jerry-git Thanks a lot for the report.

I can reproduce the issue. This is somehow connected to pytest's assertion rewriting. I will have to figure out where the error comes from.

Until I do, you can disable assertion rewriting. While this will disable the nice error messages, at least there won't be any errors. You can change the config via:

import ipytest
ipytest.autoconfig(rewrite_asserts=False)

[chmp]

Explanation:

1. The assert rewriter does not include end_lineno for its generated AST nodes
2. Ipytest calls fix_missing_locations which copies in an invalid range for the location (source)
3. The Python compiler fails

Related PyTest issue:

• https://github.com/pytest-dev/pytest/issues/10008

Simple repo:

import ast
from _pytest.assertion.rewrite import rewrite_asserts

code = """# hello
a = 2
"""
mod = ast.parse(code)
rewrite_asserts(mod, b"")
mod = ast.fix_missing_locations(mod)
compile(mod, "foo.py", "exec")

# ValueError: AST node line range (2, 1) is not valid

[chmp]

@jerry-git FYI, when I execute the integration tests locally with python 3.11, I get a test failure:

``` python x.py precommit :: python.exe -m black ipytest tests Example.ipynb x.py minidoc.py All done! ✨ 🍰 ✨ 19 files left unchanged. :: python.exe -m ruff . Update documentation :: python.exe -m pytest tests 0.00s - Debugger warning: It seems that frozen modules are being used, which may 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off 0.00s - to python to disable frozen modules. 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation. ================================================================================================================================= test session starts ================================================================================================================================= platform win32 -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 plugins: anyio-3.6.2, nbval-0.10.0, asyncio-0.21.0 asyncio: mode=Mode.STRICT collected 67 items tests\test_config.py .. [ 2%] tests\test_doctest.py .. [ 5%] tests\test_force_reload.py .... [ 11%] tests\test_ipytest.py ........................................... [ 76%] tests\test_issue_71.py .......... [ 91%] tests\test_raise_on_error.py ...... [100%] ================================================================================================================================= 67 passed in 2.48s ================================================================================================================================== :: python.exe -m pytest --nbval-lax --current-env Example.ipynb 0.00s - Debugger warning: It seems that frozen modules are being used, which may 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off 0.00s - to python to disable frozen modules. 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation. ================================================================================================================================= test session starts ================================================================================================================================= platform win32 -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0 plugins: anyio-3.6.2, nbval-0.10.0, asyncio-0.21.0 asyncio: mode=Mode.STRICT collected 11 items Example.ipynb ..F.....s.. [100%] ====================================================================================================================================== FAILURES ======================================================================================================================================= ________________________________________________________________________________________________________________________________ Example.ipynb::Cell 2 ________________________________________________________________________________________________________________________________ Notebook cell execution failed Cell 2: Cell execution caused an exception Input: %%ipytest # define the tests def test_my_func(): assert my_func(0) == 0 assert my_func(1) == 0 assert my_func(2) == 2 assert my_func(3) == 2 def my_func(x): return x // 2 * 2 Traceback: --------------------------------------------------------------------------- ValueError Traceback (most recent call last) File ~\miniconda3\envs\py311\Lib\codeop.py:118, in Compile.__call__(self, source, filename, symbol) 117 def __call__(self, source, filename, symbol): --> 118 codeob = compile(source, filename, symbol, self.flags, True) 119 for feature in _features: 120 if codeob.co_flags & feature.compiler_flag: ValueError: AST node line range (3, 1) is not valid ```

[chmp]

@jerry-git I released ipytest==0.13.2b0 with a fix. Would be cool, if you could have look, whether this solves your issue. I will also test it a for a bit and then do a proper 0.13.2 release.

[The-Compiler]

@chmp Appreciate the heads-up over at the pytest repo, and the quick fix! I use ipytest for company trainings/talks about pytest, and I bet this would have bitten me with a live demo-heavy talk next week :sweat_smile:

[chmp]

@The-Compiler thanks for the feedback. Always nice to hear that someone is using your stuff :) Not sure I will manage a proper release until next week, so hopefully the pre-release works for you. I would be interested to hear whether your content works with the fix without any issues.

[jerry-git]

when I execute the integration tests locally with python 3.11, I get a test failure ...

That's great! I guess I didn't run the integration tests, just the unit ones.

Would be cool, if you could have look, whether this solves your issue.

Yup, works 👍

Thanks for the swift reaction 🙏

[chmp]

Thanks for the feedback. Glad everything works.

[chmp]

Released as v0.13.2