Exception on modifying query

[prayzzz]

A RuntimeError is raised while adding order by to a query with limit.

Environment

• Powershell via ConEmu on Windows 10
• mycli version 1.17.0

Steps to reproduce

• connect
• issue a query like select * from table limit 10
• get this query back with arrow-up
• add order by clause
• when hitting space after by a RuntimeError is raised
• AutoCompletion is broken afterwards

Exception

Traceback (most recent call last):
  File "c:\python37\lib\site-packages\mycli\packages\parseutils.py", line 95, in extract_from_part
    raise StopIteration
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):w to complete suggestion
  File "c:\python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "c:\python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "c:\python37\lib\site-packages\prompt_toolkit\interface.py", line 865, in run
    completions = list(buffer.completer.get_completions(document, complete_event))
  File "c:\python37\lib\site-packages\mycli\sqlcompleter.py", line 258, in get_completions
    suggestions = suggest_type(document.text, document.text_before_cursor)
  File "c:\python37\lib\site-packages\mycli\packages\completion_engine.py", line 93, in suggest_type
    full_text, identifier)
  File "c:\python37\lib\site-packages\mycli\packages\completion_engine.py", line 203, in suggest_based_on_last_token
    return [{'type': 'column', 'tables': extract_tables(full_text)}]
  File "c:\python37\lib\site-packages\mycli\packages\parseutils.py", line 154, in extract_tables
    return list(extract_table_identifiers(stream))
  File "c:\python37\lib\site-packages\mycli\packages\parseutils.py", line 113, in extract_table_identifiers
    for item in token_stream:
RuntimeError: generator raised StopIteration

[dvnno]

Having the same issue using Git Bash in Windows 10 while trying to type an INSERT INTO query. Upon typing the open parenthesis, (, I get the same RuntimeError. Noticed another open ticket for pgcli with the same issue here: https://github.com/dbcli/pgcli/issues/902. I think it may be related to Python version 3.7.0 (prayzzz is also using 3.7.0 as is the user I mentioned with the similar issue in pgcli).

Environment

• Git Bash 2.18.0.1 on Windows 10
• mycli version 1.17.0
• Python version 3.7.0

Steps to Reproduce

• Connect and select any database
• Try an INSERT INTO query
• Upon typing the ( after the table name, error is thrown and stack trace printed
• After error is thrown, everything seems to function fine except for autocomplete. Can issue the same INSERT INTO query that caused the error without issue.

Stack Trace

Traceback (most recent call last):
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\site-packages\prompt_toolkit\interface.py", line 865, in run
    completions = list(buffer.completer.get_completions(document, complete_event))
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\site-packages\mycli\sqlcompleter.py", line 258, in get_completions
    suggestions = suggest_type(document.text, document.text_before_cursor)
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\site-packages\mycli\packages\completion_engine.py", line 93, in suggest_type
    full_text, identifier)
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\site-packages\mycli\packages\completion_engine.py", line 201, in suggest_based_on_last_token
    return [{'type': 'column', 'tables': extract_tables(full_text)}]
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\site-packages\mycli\packages\parseutils.py", line 154, in extract_tables
    return list(extract_table_identifiers(stream))
  File "c:\users\dvnno\appdata\local\programs\python\python37\lib\site-packages\mycli\packages\parseutils.py", line 113, in extract_table_identifiers
    for item in token_stream:
RuntimeError: generator raised StopIteration

If any other information is needed, let me know.

[meeuw]

Hmm could you please post the output from pip freeze ? This might be related to an invalid version of sqlparse.

[dvnno]

Sure thing.

asn1crypto==0.24.0
cffi==1.11.5
cli-helpers==1.0.2
click==6.7
configobj==5.0.6
cryptography==2.3
idna==2.7
mycli==1.17.0
prompt-toolkit==1.0.15
pycparser==2.18
Pygments==2.2.0
PyMySQL==0.9.2
six==1.11.0
sqlparse==0.2.4
tabulate==0.8.2
terminaltables==3.1.0
wcwidth==0.1.7

[dvnno]

Updated my Arch Linux system and with it Python to 3.7.0. Thought the same thing might happen. Sure enough, reinstalled mycli using pip and got the same error while trying to do an insert as soon as I typed the (. Here's the system info, stack trace, and the pip freeze output:

Environment

• Bash terminal in Arch Linux with Gnome
• mycli 1.17.0
• Python 3.7.0

Stack Trace

Traceback (most recent call last):                         
  File "/usr/lib/python3.7/site-packages/mycli/packages/parseutils.py", line 83, in extract_from_part                                      
    raise StopIteration                                    
StopIteration                                              

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/site-packages/prompt_toolkit/interface.py", line 865, in run
    completions = list(buffer.completer.get_completions(document, complete_event))
  File "/usr/lib/python3.7/site-packages/mycli/sqlcompleter.py", line 258, in get_completions
    suggestions = suggest_type(document.text, document.text_before_cursor)
  File "/usr/lib/python3.7/site-packages/mycli/packages/completion_engine.py", line 93, in suggest_type
    full_text, identifier)
  File "/usr/lib/python3.7/site-packages/mycli/packages/completion_engine.py", line 201, in suggest_based_on_last_token
    return [{'type': 'column', 'tables': extract_tables(full_text)}]
  File "/usr/lib/python3.7/site-packages/mycli/packages/parseutils.py", line 154, in extract_tables
    return list(extract_table_identifiers(stream))
  File "/usr/lib/python3.7/site-packages/mycli/packages/parseutils.py", line 113, in extract_table_identifiers
    for item in token_stream:
RuntimeError: generator raised StopIteration

pip freeze

alabaster==0.7.11
anytree==2.4.3
appdirs==1.4.3
asn1crypto==0.24.0
astroid==2.0.1
Babel==2.6.0
backcall==0.1.0
Beaker==1.10.0
Brlapi==0.6.7
btrfsutil==1.0.0
CacheControl==0.12.5
cffi==1.11.5
chardet==3.0.4
chrome-gnome-shell==0.0.0
cli-helpers==1.0.2
click==6.7
colorama==0.3.9
configobj==5.0.6
cryptography==2.3
decorator==4.3.0
distlib==0.2.7
distro==1.3.0
docutils==0.14
evdev==1.0.0
html5lib==1.0.1
idna==2.7
imagesize==1.0.0
ipython==6.5.0
ipython-genutils==0.1.0
isc==2.0
isort==4.3.4
jedi==0.12.1
Jinja2==2.10
lazy-object-proxy==1.3.1
lensfun==0.3.2
lockfile==0.12.2
louis==3.6.0
lxml==4.2.4
Mako==1.0.7
MarkupSafe==1.0
mccabe==0.6.1
msgpack==0.5.6
mycli==1.17.0
packaging==17.1
parso==0.3.1
pexpect==4.6.0
pickleshare==0.7.4
progress==1.4
prompt-toolkit==1.0.15
ptyprocess==0.6.0
pycairo==1.17.0
pycodestyle==2.4.0
pycparser==2.18
pyflakes==2.0.0
Pygments==2.2.0
pygobject==3.28.3
pylint==2.0.1
PyMySQL==0.9.2
pyparsing==2.2.0
python-libtorrent==1.1.9
pytoml==0.1.16
pytz==2018.5
pyudev==0.21.0.dev20180701
pyxdg==0.26
requests==2.19.1
retrying==1.3.3
simplegeneric==0.8.1
simplejson==3.16.0
six==1.11.0
snowballstemmer==1.2.1
Sphinx==1.7.6
sphinx-rtd-theme==0.4.0
sphinxcontrib-websupport==1.1.0
SQLAlchemy==1.2.10
sqlparse==0.2.4
tabulate==0.8.2
team==1.0
terminaltables==3.1.0
traitlets==4.3.2
typed-ast==1.1.0
urllib3==1.23
wcwidth==0.1.7
webencodings==0.5.1
Whoosh==2.7.4
wrapt==1.10.11

[zyguan]

extract_from_part raises StopIteration which turns to RuntimeError since python 3.7 3.5 according to PEP479.

https://github.com/dbcli/mycli/blob/543e4b6a35b8a5bab66507c0a4e1cfe35cd88170/mycli/packages/parseutils.py#L75-L108

[meeuw]

Thanks a lot @zyguan

I'm unable to reproduce this issue, I suppose we should break instead of raise StopIteration?

[zyguan]

Hi, @meeuw , I think return is better because it works out of the box in inner loop. You can reproduce the issue by changing this sql to select * from table limit 10 (run it in python 3.7 3.5 or above).

There are some other raise StopIteration statements. It would be better to fix them all togather : )

[meeuw]

I'm using Python 3.6.6 on Fedora and I cannot reproduce this. I think it is somehow related to Windows.

I'm not sure if we should use return because that might raise a StopIteration as well.

Soon I'll try to reproduce this in Windows and write a fix.

[dvnno]

@meeuw I happened to me on my Arch Linux system with Python 3.7.0. Any system that I have with Python 3.7.0 throws the error.

[zyguan]

@meeuw maybe the following screencasts would help:

• https://asciinema.org/a/nykChhVx7eSxnHrYSTIu56sAh
• https://asciinema.org/a/Xh6eUqMgnAUb6n0BhFYlOo18u
• https://asciinema.org/a/3FeirugolvmOFt8PTk4NnGq9c

[zyguan]

Also, PEP479 mentions that:

Finally, the proposal also clears up the confusion about how to terminate a generator: the proper way is return, not raise StopIteration.

[meeuw]

Hmm, I suppose it's Python 3.7 then. I don't have Py3.7 yet and it isn't supported by Travis (without hacks) yet so no official support yet (from me :smile:). We accept pull requests though if you'd like to create a fix in the meantime. Please make sure it's compatible with older versions of Python (like Python 2.7).

[zyguan]

@meeuw Yes, you're right, there is no problem in python 3.5 & 3.6.

PEP 479 is enabled for all code in Python 3.7

https://docs.python.org/3/whatsnew/3.7.html#changes-in-python-behavior

I will fix it later : )

[eirnym]

When it would be released?

[tsroten]

@eirnym This is now fixed in version 1.18.0! You can get the newest version via pip or check out the release.