cannot instantiate windowspath on cross-platform vim/python

btel commented 3 years ago

I am running vim on windows in msys2 shell. vim python is executed via msys2-installed python, but the python executable (configured in environment_path) is native-windows python from anaconda:

#### Jedi-vim debug information

##### jedi-vim version

 - jedi-vim git version: 7ac81cb-dirty
 - jedi git submodule status: +65bc1c117b3175cb4d492484775c3fd7f207bc92 pythonx/jedi (v0.18.0-83-g65bc1c11)
 - parso git submodule status:  ef90bba3b3d4310c345885a1368dbfc8937765a3 pythonx/parso (v0.8.1-1-gef90bba)

##### Global Python

Using Python version 3 to access Jedi.
 - global sys.executable: `/usr/bin/python3.exe`
 - global sys.version: `3.8.7 (default, Jan 15 2021, 14:12:45), [GCC 10.2.0]`
 - global site module: `/usr/lib/python3.8/site.py`

##### Jedi

 - path: `/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/__init__.py`
 - version: 0.18.0

##### Jedi environment: <Environment: 3.8.10 in C:\Users\btel\Anaconda3\envs\myenv>

 - executable: C:\Users\btel\Anaconda3\envs\myenv\python.exe
 - sys_path:
    - `C:\Users\btel\repos\myenv`
    - `C:\Users\btel\Anaconda3\envs\myenv\python38.zip`
    - `C:\Users\btel\Anaconda3\envs\myenv\DLLs`
    - `C:\Users\btel\Anaconda3\envs\myenv\lib`
    - `C:\Users\btel\Anaconda3\envs\myenv`
    - `C:\Users\btel\AppData\Roaming\Python\Python38\site-packages`
    - `C:\Users\btel\Anaconda3\envs\myenv\lib\site-packages`
    - `c:\users\b2278748\repos\myenv\myenv-lib`
    - `C:\Users\btel\Anaconda3\envs\myenv\lib\site-packages\win32`
    - `C:\Users\btel\Anaconda3\envs\myenv\lib\site-packages\win32\lib`
    - `C:\Users\btel\Anaconda3\envs\myenv\lib\site-packages\Pythonwin`

When I try do go to definition from vim, I will get the following error:

  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/api/helpers.py", line 189, in filter_follow_imports
    name.goto(),
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/names.py", line 157, in goto
    module_names = goto_import(context, name)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/cache.py", line 44, in wrapper
    rv = function(obj, *args, **kwargs)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/imports.py", line 75, in goto_import
    _prepare_infer_import(module_context, tree_name)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/imports.py", line 116, in _prepare_infer_import
    return from_import_name, tuple(import_path), import_node.level, importer.follow()
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/imports.py", line 296, in follow
    return import_module_by_names(
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/imports.py", line 376, in import_module_by_names
    value_set = ValueSet.from_sets([
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/imports.py", line 377, in <listcomp>
    import_module(
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/plugins/__init__.py", line 21, in wrapper
    return built_functions[public_name](*args, **kwargs)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/plugins/flask.py", line 20, in wrapper
    return callback(inference_state, import_names, module_context, *args, **kwargs)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/gradual/typeshed.py", line 115, in wrapper
    python_value_set = ValueSet.from_sets(
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/base_value.py", line 426, in from_sets
    for set_ in sets:
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/gradual/typeshed.py", line 116, in <genexpr>
    func(inference_state, import_names, p, sys_path,)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/imports.py", line 411, in import_module
    file_io_or_ns, is_pkg = inference_state.compiled_subprocess.get_module_info(
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/compiled/subprocess/__init__.py", line 130, in wrapper
    result = self._compiled_subprocess.run(
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/compiled/subprocess/__init__.py", line 227, in run
    return self._send(id(inference_state), function, args, kwargs)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/inference/compiled/subprocess/__init__.py", line 249, in _send
    is_exception, traceback, result = pickle_load(self._get_process().stdout)
  File "/c/Users/btel/.vim/plugged/jedi-vim/pythonx/jedi/jedi/_compatibility.py", line 12, in pickle_load
    return pickle.load(file)
  File "/usr/lib/python3.8/pathlib.py", line 1043, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'WindowsPath' on your system

I think that the error might be caused by jedi pickling a WindowsPath (while running winodws executable) but then it can't be unpickled in vim python which is a posix executable. Would this explain this error? Is there an easy workaround or I should not mix the platforms when running jedi?

davidhalter commented 3 years ago

Can you run Jedi directly and use the InterpreterEnvironment so that we can have a useful stacktrace? I think that would help.

davidhalter commented 3 years ago

Ok I guess I'm not even sure anymore if that would help. My guess is that there is something fundamentally broken that you are not allowed to instantiate WindowsPath on your Windows system. I would probably report this somewhere else.

You should probably try to invoke it directly and see if the problem still happens at that point.

btel commented 3 years ago

Thanks for the quick reply.

In my mind, the problem is that I am running posix python in vim (using msys2 shell), so it expects the posix path. On the other hand, I think jedi runs anaconda python (win64 exeutable, from environment_path) in subprocess and pickles windows path (from pathlib) and it cannot be unpickled in posix python.

When you look into pahtlib code, you find that it uses os.name to determine the platform:

https://github.com/python/cpython/blob/d672dd34f0bc3addeaf1789d4183e3a37ab110d5/Lib/pathlib.py#L1080-L1081

On my system:

Anaconda3 python:

$ winpty /c/Users/btel/Anaconda3/envs/myenv/python
Python 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)] :: An
aconda, Inc. on win32
>>> import os
>>> print(os.name)
nt

msys2 (posix) python:

$ /usr/bin/python
Python 3.8.7 (default, Jan 15 2021, 14:12:45)
[GCC 10.2.0] on msys
>>> import os
>>> print(os.name)
posix
>>>

so in one case pathlib will create WindowsPath object but then it does not know how to convert it into PosixPath when loading the pickle in vim python.

Does it make sense for you?

I don't know whether this should be reported to python tracker or it might be fixed in jedi (maybe by pickling strings instead of pathlib objects?).

There is a similar issue in fastai: https://github.com/fastai/fastai/issues/1482

btel commented 3 years ago

@davidhalter I came up with a minimal example showing why this error might occur:

# content of dump_path.py
import pickle

import pathlib

p = pathlib.Path('.')

with open('path.pkl', 'wb') as fid:
    pickle.dump(p, fid)

# content  of load_path.py
import pickle

with open("path.pkl", 'rb') as fid:
    p = pickle.load(fid)

Then:

$ ~/Anaconda3/envs/myenv/python.exe  dump_path.py

$ /usr/bin/python.exe  load_path.py
Traceback (most recent call last):
  File "load_path.py", line 4, in <module>
    p = pickle.load(fid)
  File "/usr/lib/python3.8/pathlib.py", line 1043, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'WindowsPath' on your system

Do you think this could be happening in jedi (/usr/bin/python is embedded in vim, wherease Anaconda python is on my jedi environment_path)? Can it be fixed in jedi or should I report elsewhere? Thanks!

davidhalter commented 3 years ago

I kind of understand why this is happening now, thanks for reproducing that. However I feel like this is just too niche of a use case. If anything your VIM should probably just use InterpreterEnvironment instead of a subprocess. I'm not sure if this is currently supported by jedi-vim, but that should be the way to go. IMO Jedi does it fine here and your use case is just not supported. But the InterpreterEnvironment should do the job for you.

btel commented 3 years ago

Thanks for the answer @davidhalter. I appreciate your advice. I will try InterpreterEnvironment. Just a quick question: does the InterpreterEnvironment respect the paths of the python from environment? Otherwise, should I retrieve the paths via subprocess and then patch InterpreterEnvironment with the correct paths?

davidhalter commented 3 years ago

You can always provide specific paths, but the InterpreterEnvironment uses the sys path of its running Python process instead of starting a subprocess.

davidhalter / jedi

cannot instantiate windowspath on cross-platform vim/python #1804