Semshi produces errors with __future__ annotations

[dkuhlman]

Semshi is a great piece of code. It's very useful and helpful.

But, apparently, Semshi has issues with files containing the following line:

from __future__ import annotations

When I edit a file containing the above line, I get the following error message:

Error detected while processing function <SNR>41_filetype_changed[4]..remote#define#CommandBootstrap[5]..remote#define#request:
line    2:
Error invoking '/home/dkuhlman/.vim/pack/addons/start/semshi/rplugin/python3/semshi:command:Semshi' on channel 3 (python3-rplugin-host):
error caught in request handler '/home/dkuhlman/.vim/pack/addons/start/semshi/rplugin/python3/semshi:command:Semshi [['enable']]':
Traceback (most recent call last):
  File "/home/dkuhlman/.vim/pack/addons/start/semshi/rplugin/python3/semshi/node.py", line 55, in __init__
    self.symbol = self.env[-1].lookup(self.symname)
  File "/usr/lib/python3.10/symtable.py", line 125, in lookup
    flags = self._table.symbols[name]
KeyError: 'str'

During handling of the above exception, another exception occurred:

[etc]

If I comment out the __future__ line, the error messages go away. Alternatively, if I uninstall Semshi, Neovim runs without errors.

I'm running Semshi with the following:

Python 3.10.1 Neovim compiled from Github source. Arch Linux 5.15.7-arch1-1

Dave

[bennett-nguyen]

The same bug occurs on Windows too, I tried commenting out the import __future__ line, and the highlighting goes back to normal. But oddly, this bug doesn't occur on Ubuntu (more specifically Ubuntu 20.04.3 LTS x86_64)

[ObserverOfTime]

That's because Ubuntu 20.04 doesn't have Python 3.10. Nothing odd about it…

[bennett-nguyen]

@ObserverOfTime I switched from Python 3.10.1 to Python 3.7.9 on my Windows machine and now I can see the highlighting again. So what really happened here? Did Python 3.10 make shemshi buggy or something?

[blueyed]

I have not tried, but the interesting part is hidden by [etc] (i.e. the real error).

[ObserverOfTime]

The [etc] part is not interesting since it's just the exception Semshi raises after the real error. Looking into it, this is caused by symtable not exposing annotation symbols when the import is used. The offending Python commit is most likely python/cpython@ad106c6.

from symtable import symtable

future = 'from __future__ import annotations\n'
code = 'ham: str = "ham"'

symtable(code, 'ham.py', 'exec').lookup('str') # works
symtable(future + code, 'ham.py', 'exec').lookup('str') # throws

[blueyed]

The offending Python commit is most likely python/cpython@ad106c6.

Yes, confirmed. Is there an issue (on bugs.python.org) for this already? If not, I think we should create one.

Interestingly it also fails also only in the 2nd case with Python 3.11.0a4+ (44afdbd), although the future import is mandatory there according to https://docs.python.org/3.11/library/__future__.html.

[ObserverOfTime]

Considering this is intended behaviour, I don't think submitting an issue will change anything. This would probably have to be fixed with a workaround on Semshi's side.

[blueyed]

Yeah, I've only figured out later that it might be intentional, which wasn't clear (to me) from the issue's title (https://bugs.python.org/issue42725).

It is still not clear to me though, if not symtable could/should grab/set the symbols from __annotations__ / via inspect.get_annotations() (https://docs.python.org/3.10/howto/annotations.html#accessing-the-annotations-dict-of-an-object-in-python-3-10-and-newer).

It is also suspicious that Python 3.11 without the future-annotation behaves differently (although it should be mandatory/enabled as per docs), but that might just not be the case already.

[noudin-ledger]

So, is there any idea how to fix this?

[aubustou]

I committed a fix on a forked repo: https://github.com/aubustou/semshi It's rather ugly but it works.

[wookayin]

I'm working on this to add PEP 563 support and fix the bug. If you're interested, please have a look at a PR wookayin/semshi#2 in my fork.

UPDATE: Merged.