search

pytest-dev / py

Python development support library (note: maintenance only)
MIT License
69 stars 105 forks source link

Exception in wrapped list comprehension causes "TokenError: EOF in multi-line statement" #68

Closed pytestbot closed 8 years ago

pytestbot commented 8 years ago

The following example raises a TokenError: EOF in multi-line statement in py 1.4.29 (tested with pytest 2.7.2).

#!python

def test_pytestbug():
    for a in [a for a in
              CAUSE_ERROR]: pass

Downgrading to py 1.4.28 (and pytest 2.7.1) fixes it.

Full stacktrace:

#!python

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/main.py", line 84, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/main.py", line 122, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/main.py", line 142, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 393, in execute
INTERNALERROR>     return wrapped_call(method(*args), self.execute)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 113, in wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 138, in get_result
INTERNALERROR>     py.builtin._reraise(*ex)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 123, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/runner.py", line 65, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/runner.py", line 75, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/runner.py", line 121, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 521, in __call__
INTERNALERROR>     return self._docall(self.methods, kwargs)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 528, in _docall
INTERNALERROR>     firstresult=self.firstresult).execute()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 393, in execute
INTERNALERROR>     return wrapped_call(method(*args), self.execute)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 109, in wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/skipping.py", line 158, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 138, in get_result
INTERNALERROR>     py.builtin._reraise(*ex)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 123, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/core.py", line 394, in execute
INTERNALERROR>     res = method(*args)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/runner.py", line 224, in pytest_runtest_makereport
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/python.py", line 625, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/python.py", line 618, in _repr_failure_py
INTERNALERROR>     style=style)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/_pytest/main.py", line 410, in _repr_failure_py
INTERNALERROR>     style=style, tbfilter=tbfilter)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/code.py", line 412, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/code.py", line 590, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/code.py", line 582, in repr_traceback
INTERNALERROR>     reprentry = self.repr_traceback_entry(entry, einfo)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/code.py", line 528, in repr_traceback_entry
INTERNALERROR>     source = self._getentrysource(entry)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/code.py", line 455, in _getentrysource
INTERNALERROR>     source = entry.getsource(self.astcache)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/code.py", line 204, in getsource
INTERNALERROR>     astnode=astnode)
INTERNALERROR>   File "/root/.virtualenv/lib/python2.6/site-packages/py/_code/source.py", line 384, in getstatementrange_ast
INTERNALERROR>     for tok in tokenize.generate_tokens(lambda: next(it)):
INTERNALERROR>   File "/usr/lib/python2.6/tokenize.py", line 352, in generate_tokens
INTERNALERROR>     raise TokenError, ("EOF in multi-line statement", (lnum, 0))
INTERNALERROR> TokenError: ('EOF in multi-line statement', (4, 0))
pytestbot commented 8 years ago

Original comment by @hheimbuerger

1.4.30 seems to fix all of our remaining issues regarding these parsing errors, thanks!

pytestbot commented 8 years ago

Original comment by @hheimbuerger

@hpk42 I can confirm that this fixes the specific real-life test I derived the minimal example from. Thanks!

pytestbot commented 8 years ago

Original comment by @hpk42

i released py==1.4.30 to pypi

pytestbot commented 8 years ago

Original comment by @hpk42

fix issue68 and refine the algorithm for finding line numbers for traceback reporting

→ <<cset 366ab346610c>>

pytestbot commented 8 years ago

Original comment by @hpk42

i fixed it and have a release candidate. could you try pip install -i https://devpi.net/hpk/dev/ -U py which should get you py-1.4.30?