Make this working with pytest?

[skirpichev]

Take an example (ideas were installed from the git):

$ cat conftest.py 
from ideas.examples import fractions_ast
fractions_ast.add_hook()
$ cat test_fractions.py
import fractions

def test_not_float():
    assert isinstance(1/2, fractions.Fraction)
$ pytest -q test_fractions.py
.                                                                                                     [100%]
1 passed in 0.01s

Great!

But pytest does some magic AST transformation with the assert statement. Thus for following:

$ cat test_fractions.py 
import fractions

def test_not_float():
    assert isinstance(1/2, fractions.Fraction)

def test_arith():
    assert 1/2 + 3/5 == fractions.Fraction(11, 12)  # wrong!

new test blow up:

$ pytest -q test_fractions.py 
.F                                                                                                                                                     [100%]
========================================================================== FAILURES ==========================================================================
_________________________________________________________________________ test_arith _________________________________________________________________________

Traceback (most recent call last):
  File "/home/sk/venv/ideas/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/main.py", line 304, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/terminal.py", line 813, in pytest_sessionfinish
    self.config.hook.pytest_terminal_summary(
  File "/usr/lib/python3/dist-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 92, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3/dist-packages/pluggy/manager.py", line 83, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3/dist-packages/pluggy/callers.py", line 182, in _multicall
    next(gen)  # first yield
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/terminal.py", line 828, in pytest_terminal_summary
    self.summary_failures()
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/terminal.py", line 1011, in summary_failures
    self._outrep_summary(rep)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/terminal.py", line 1030, in _outrep_summary
    rep.toterminal(self._tw)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/reports.py", line 87, in toterminal
    longrepr_terminal.toterminal(out)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/_code/code.py", line 995, in toterminal
    element[0].toterminal(tw)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/_code/code.py", line 1025, in toterminal
    entry.toterminal(tw)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/_code/code.py", line 1119, in toterminal
    self._write_entry_lines(tw)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/_code/code.py", line 1101, in _write_entry_lines
    tw._write_source(source_lines, indents)
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/_io/terminalwriter.py", line 192, in _write_source
    new_lines = self._highlight(source).splitlines()
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/_pytest/_io/terminalwriter.py", line 201, in _highlight
    from pygments.formatters.terminal import TerminalFormatter
  File "/usr/lib/python3/dist-packages/pygments/formatters/__init__.py", line 18, in <module>
    from pygments.formatters._mapping import FORMATTERS
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "/home/sk/venv/ideas/lib/python3.9/site-packages/ideas/import_hook.py", line 144, in exec_module
    with open(self.filename, mode="r+b") as f:
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/pygments/formatters/_mapping.py'

[aroberge]

Thanks for the suggestion. Unfortunately, I have no idea how I could even begin to address this. The AST transformation example is just a proof of concept. Of course, if one uses ideas to modify the AST and also uses pytest, which also does AST transformations as you point out, there is no guarantee that the two types of transformations will, in general, be compatible. And, as you show, they (currently) are not.

Unfortunately, while it is an interesting suggestion, I will have to close this issue as being out of scope for this project, and beyond my capabilities. If you have concrete suggestions to offer as to how I might go about to address this, please feel free to reopen this issue.

[skirpichev]

On Tue, Jun 08, 2021 at 03:53:53AM -0700, André Roberge wrote:

If you have concrete suggestions to offer as to how I might go about to address this

Probably, pytest can adopt the ideas machinery to handle their ast transformation. That was my initial guess, probably there are other ways to handle this issue, maybe within the ideas project itself.

If you are ok with this solution - I'll look into to provide a patch (for pytest, but that might require some changes on the ideas side).

[aroberge]

I would be ok with this solution. However, note that ideas is really just a toy project created by a dabbling amateur whereas pytest is an industry-grade project created by professional programmers: I would not be surprised at all if any patch submitted to pytest to solve this issue were to be rejected by the maintainers of that project. At the same time, I am open to anyone helping contributing to making ideas more robust and useful - and helping me learn more in doing so.

[skirpichev]

any patch submitted to pytest to solve this issue were to be rejected by the maintainers of that project

Probably, so. But I would like to see huge number of Rational(n, m) in my tests being reduced to n/m. Apparently, the pytest machinery is not flexible enough to add additional ast/token transformations - that's why I think they could use your framework.

If you do think this issue can't be solved in the ideas itself - feel free to close it. I'll open a PR if I find a way ;-)

[skirpichev]

Just for record, I corrected the example - it was wrong. The problem happens only for invalid asserts: tracebacks aren't very helpfull.

[skirpichev]

FYI: permission error can be fixed easily. In fact, I don't know why you open files for updating here: 'rb' mode seems to be more correct.

Unfortunately, pytest's tracebacks aren't very helpful with the ideas:

$ pytest test_fractions.py
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.0
rootdir: /home/sk/ideas
plugins: cov-2.10.1
collected 3 items                                                              

test_fractions.py ..F                                                    [100%]

=================================== FAILURES ===================================
_______________________________ test_arith_wrong _______________________________

>   ???
E   AssertionError

test_fractions.py:14: AssertionError
=========================== short test summary info ============================
FAILED test_fractions.py::test_arith_wrong - AssertionError
========================= 1 failed, 2 passed in 0.71s ==========================

$ cat test_fractions.py
import fractions

def test_not_float():
    assert isinstance(1/2, fractions.Fraction)

def test_arith_correct():
    assert 1/2 + 3/5 == fractions.Fraction(11, 10)

def test_arith_wrong():
    assert 1/2 + 3/5 == fractions.Fraction(11, 12)

BTW, I think you may change the API (at least for AST transformations, but, perhaps - for everything) to be like the IPython ast_transformers option. I.e. a list of AST transformations to easily combine them. Currently your users forced to make custom transform_ast() function, which essentially just run sequentially all transformations. This is merely a boilerplate code. If this does make sense for you - I'll open a separate issue.

[aroberge]

Reopening this issue to keep track of work to do.