Several issues around `PosixPath`

WhyNotHugo commented 4 years ago

When I try to jump to a definition using Jedi 0.17.0 or 0.17.2, I'm getting this error:

[coc.nvim] Jedi error: Traceback (most recent call last):
  File "completion.py", line 670, in watch
    response = self._process_request(rq)
  File "completion.py", line 612, in _process_request
    script.goto(line, column, follow_imports=True), request["id"]
  File "/usr/lib/python3.8/site-packages/jedi/api/helpers.py", line 465, in wrapper
    return func(self, line, column, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/api/__init__.py", line 371, in goto
    return self._goto(line, column, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/api/__init__.py", line 396, in _goto
    names = list(name.goto())
  File "/usr/lib/python3.8/site-packages/jedi/inference/names.py", line 203, in goto
    values = infer_call_of_leaf(context, name, cut_own_trailer=True)
  File "/usr/lib/python3.8/site-packages/jedi/inference/helpers.py", line 105, in infer_call_of_leaf
    values = context.infer_node(base)
  File "/usr/lib/python3.8/site-packages/jedi/inference/context.py", line 222, in infer_node
    return infer_node(self, node)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 137, in infer_node
    return _infer_node_if_inferred(context, element)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 150, in _infer_node_if_inferred
    return _infer_node_cached(context, element)
  File "/usr/lib/python3.8/site-packages/jedi/inference/cache.py", line 43, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 155, in _infer_node_cached
    return _infer_node(context, element)
  File "/usr/lib/python3.8/site-packages/jedi/debug.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 63, in wrapper
    return func(context, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 165, in _infer_node
    return infer_atom(context, element)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 290, in infer_atom
    return context.py__getattribute__(atom, position=position)
  File "/usr/lib/python3.8/site-packages/jedi/inference/context.py", line 75, in py__getattribute__
    values = ValueSet.from_sets(name.infer() for name in names)
  File "/usr/lib/python3.8/site-packages/jedi/common/value.py", line 69, in from_sets
    for set_ in sets:
  File "/usr/lib/python3.8/site-packages/jedi/inference/context.py", line 75, in <genexpr>
    values = ValueSet.from_sets(name.infer() for name in names)
  File "/usr/lib/python3.8/site-packages/jedi/inference/names.py", line 279, in infer
    return tree_name_to_values(
  File "/usr/lib/python3.8/site-packages/jedi/plugins/__init__.py", line 21, in wrapper
    return built_functions[public_name](*args, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/plugins/stdlib.py", line 842, in wrapper
    return func(inference_state, context, tree_name)
  File "/usr/lib/python3.8/site-packages/jedi/inference/syntax_tree.py", line 715, in tree_name_to_values
    types = imports.infer_import(context, tree_name)
  File "/usr/lib/python3.8/site-packages/jedi/inference/cache.py", line 43, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/usr/lib/python3.8/site-packages/jedi/inference/imports.py", line 55, in infer_import
    _prepare_infer_import(module_context, tree_name)
  File "/usr/lib/python3.8/site-packages/jedi/inference/imports.py", line 115, in _prepare_infer_import
    importer = Importer(module_context.inference_state, tuple(import_path),
  File "/usr/lib/python3.8/site-packages/jedi/inference/imports.py", line 169, in __init__
    debug.speed('import %s %s' % (import_path, module_context))
  File "/usr/lib/python3.8/site-packages/jedi/inference/context.py", line 216, in __repr__
    return '%s(%s)' % (self.__class__.__name__, self._value)
  File "/usr/lib/python3.8/site-packages/jedi/inference/value/module.py", line 228, in __repr__
    self.is_stub()
  File "/usr/lib/python3.8/site-packages/jedi/inference/value/module.py", line 159, in is_stub
    if self._path is not None and self._path.endswith('.pyi'):
AttributeError: 'PosixPath' object has no attribute 'endswith'

I tried using Jedi master, but I get this other error (again, when jumping to a definition):

[coc.nvim] Jedi error: Traceback (most recent call last):
  File "completion.py", line 670, in watch
    response = self._process_request(rq)
  File "completion.py", line 614, in _process_request
    return json.dumps({"id": request["id"], "results": defs})
  File "/usr/lib/python3.8/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable

I'm getting the impression that some library (or maybe python itself?) started returning PosixPath instead of str somewhere, and this isn't being handled properly.

Output of pip freeze:

aiohttp==3.6.2
aiohttp-socks==0.4.2
aiorpcX==0.18.4
apipkg==1.5
apparmor==2.13.4
appdirs==1.4.4
argcomplete==1.11.1
asn1crypto==1.4.0
aspy.refactor-imports==2.1.1
asttokens==2.0.3
async-timeout==3.0.1
atomicwrites==1.4.0
attrs==19.3.0
Automat==20.2.0
awscli==1.18.120
awslogs==0.11.0
backcall==0.2.0
bcrypt==3.1.7
Beaker==1.11.0
bitstring==3.1.7
black==19.10b0
boto3==1.14.43
botocore==1.17.43
btrfsutil==1.2.0
CacheControl==0.12.6
cached-property==1.5.1
cachy==0.3.0
certifi==2020.6.20
cffi==1.14.2
cfgv==3.2.0
chardet==3.0.4
cleo==0.7.6
click==7.1.2
click-log==0.3.2
click-threading==0.4.4
clikit==0.4.2
colorama==0.4.3
commonmark==0.9.1
configobj==5.0.6
constantly==15.1.0
contextlib2==0.6.0.post1
cryptography==3.0
daemonize==2.5.0
decorator==4.4.2
deluge==2.0.4.dev38
distlib==0.3.1
distro==1.5.0
dnspython==2.0.0
docker==4.3.0
docker-compose==1.26.2
docker-pycreds==0.4.0
dockerpty==0.4.1
docopt==0.6.2
docutils==0.16
dparse==0.5.1
ecdsa==0.15
Electrum==4.0.2
entrypoints==0.3
evdev==1.3.0
filelock==3.0.12
flake8==3.8.3
flake8-bugbear==20.1.4
flake8-comprehensions==3.2.2
flake8-import-order==0.18.1
future==0.18.2
GeoIP==1.3.2
gpg==1.14.0
greenlet==0.4.16
gunicorn==20.0.4
html5lib==1.1
humanize==0.5.1
hyperlink==20.0.1
icalendar==4.0.6
identify==1.4.28
idna==2.10
importlib-metadata==1.6.1
incremental==17.5.0
iniconfig==1.0.1
ipython==7.17.0
ipython-genutils==0.1.0
isc==2.0
jedi==0.17.2
jeepney==0.4.3
jmespath==0.10.0
jsonrpclib-pelix==0.4.1
jsonschema==3.2.0
keyring==21.3.0
khal==0.10.2
langdetect==1.0.8
LibAppArmor==2.13.4
lit==0.10.1.dev0
lockfile==0.12.2
louis==3.14.0
lxml==4.5.2
Mako==1.1.3
MarkupSafe==1.1.1
mccabe==0.6.1
meld==3.20.2
meson==0.55.1
more-itertools==8.4.0
msgpack==1.0.0
multidict==4.7.6
mypy==0.782
mypy-extensions==0.4.3
nodeenv==1.4.0
notify2==0.3.1
numpy==1.19.1
oauthlib==3.1.0
openrazer==2.8.0
openrazer-daemon==2.8.0
ordered-set==4.0.2
packaging==20.4
paramiko==2.7.1
parsedatetime==2.6
parso==0.8.0
pastel==0.2.0
pathspec==0.8.0
pbkdf2==1.3
pep517==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.2.0
pipenv==2020.8.13
pkginfo==1.5.0.1
pluggy==0.13.1
ply==3.11
poetry==1.0.10
pre-commit==2.6.0
progress==1.5
progressbar2==3.51.4
prompt-toolkit==3.0.6
protobuf==3.12.4
psutil==5.7.2
ptyprocess==0.6.0
pwquality==1.4.2
py==1.9.0
pyaes==1.6.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.19.1
pycodestyle==2.6.0
pycparser==2.20
pycryptodome==3.9.8
pycryptodomex==3.9.7
pyflakes==2.2.0
Pygments==2.6.1
PyGObject==3.36.1
PyHamcrest==1.9.0
pylev==1.3.0
pymediainfo==4.2.1
PyNaCl==1.3.0
pynvim==0.4.0
pyOpenSSL==19.1.0
pyparsing==2.4.7
PyQt5==5.15.0
PyQt5-sip==12.8.0
pyrsistent==0.16.0
PySocks==1.7.1
pytest==6.0.1
python-dateutil==2.8.1
python-dotenv==0.14.0
python-libtorrent==1.2.8
python-utils==2.4.0
pytz==2020.1
pyudev==0.22.0.dev20200622
pyupgrade==2.7.2
pyxdg==0.26
PyYAML==5.3.1
qrcode==6.1
Reflector==2020.3.21.11.40.36
regex==2020.7.14
rencode==1.0.6
reorder-python-imports==2.3.0
requests==2.24.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
resolvelib==0.4.0
retrying==1.3.3
rsa==4.6
s3transfer==0.3.3
safety==1.9.0
schedule==0.6.0
SecretStorage==3.1.2
service-identity==18.1.0
setproctitle==1.1.10
shellingham==1.3.1
sip==4.19.23
six==1.15.0
solaar==1.0.3
speedtest-cli==2.1.2
stevedore==3.2.0
subdownloader==2.1.0
tabulate==0.8.7
team==1.0
termcolor==1.1.0
texttable==1.6.2
todoman==3.8.0
tokenize-rt==4.0.0
toml==0.10.1
tomlkit==0.7.0
torbrowser-launcher==0.3.2
tox==3.19.0
traitlets==4.3.3
trash-cli==0.17.1.14
Twisted==20.3.0
txdbus==1.1.0
typed-ast==1.4.1
typing-extensions==3.7.4.3
tzlocal==2.1
urllib3==1.25.10
urwid==2.1.1
vdirsyncer==0.16.9.dev0+gb5dd092.d20200713
virtualenv==20.0.23
virtualenv-clone==0.5.4
virtualenvwrapper==4.8.4
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.57.0
yarl==1.5.1
youtube-dl==2020.7.28
zipp==3.1.0
zope.interface==5.1.0

algmyr commented 4 years ago

I'm getting the impression that some library (or maybe python itself?) started returning PosixPath instead of str somewhere, and this isn't being handled properly.

I dug a bit and jedi changed code to work with Path objects in https://github.com/davidhalter/jedi/commit/db0e90763be0f65de1a03a270f10272b46184892. Moving to master fixes the first error.

A naive fix for this particular issue is changing https://github.com/neoclide/coc-python/blob/master/pythonFiles/completion.py#L420 to do module_path.as_posix() (I suspect this is not the right thing on windows). However this is a more overarching issue that would need to be dealt with (I see a lot of other stuff that will break in the same way).

Possible solutions

Convert to string as soon as the path is retrieved from jedi and keep the rest of the code unchanged
Adopt the code to use the same Path as jedi and only stringify for the sake of serialization

I'm not intimate enough with this code to know what's the best course of action, but I think the info here will make the work for anyone who knows the codebase easier.

WhyNotHugo commented 4 years ago

Path objects implement __str__ they way you'd expect them to work, so str(module_path) is the right way to go and should work on any platform

Also str(module_path) should avoid breakage if someone is using an old version of Jedi, since it'll just return the same string.

nixypanda commented 3 years ago

Faced this issue, it looks like this fix is merged too, but I haven't seen a release for this package in a while. (Forgive my ignorance but) Can you help me understand why that is? I mean shouldn't this be released to npm so that people can use it without having to use master etc.?

neoclide / coc-python

Several issues around `PosixPath` #263