Providing a file name on command line breaks python path

[altunyurt]

I'm using localvimrc master branch with neovim 0.6.1

I've created a virtual environment using poetry under /tlist, which containns the following lines in /tlist/.lvimrc file

let cwd = getcwd()
let g:python3_host_prog = cwd . "/.venv/bin/python"

My default python3_host_prog is set as follows in the ~/.vimrc file

let g:python3_host_prog = '/home/den/.pyenv/versions/editors/bin/python'

If i start vim with a file name as argument, such as

poetry run vim app.py

and run the command

:python3 import sys; print(sys.path)

I get the following output

['/home/den/.pyenv/versions/3.10.4/lib/python310.zip', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10/lib-dynload', 
'/home/den/.pyenv/versions/editors/lib/python3.10/site-packages',  <== default host prog 
'_vim_path_']  

If i run vim without any file name as argument, such as

poetry run vim

I get this following output upon running the same command

['/home/den/.pyenv/versions/3.10.4/lib/python310.zip', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10/lib-dynload', 
'/tlist/.venv/lib/python3.10/site-packages',           <= virtual env host prog
'_vim_path_'] 

Following is the debug output of running vim as follows poetry run vim --cmd "let g:localvimrc_debug=99" -c "LocalVimRCDebugDump localvimrc_debug.txt" app.py

== START settings ================================ 
localvimrc_enable = "1" 
localvimrc_name = "['.lvimrc']" 
localvimrc_event = "['BufWinEnter']" 
localvimrc_event_pattern = "'*'" 
localvimrc_reverse = "0" 
localvimrc_count = "-1"
localvimrc_file_directory_only = "0"
localvimrc_sandbox = "1"
localvimrc_ask = "0"
localvimrc_whitelist = "['^$']"
localvimrc_blacklist = "['^$']"
localvimrc_persistent = "0"
localvimrc_persistence_file = "'/home/den/.localvimrc_persistent'"
localvimrc_autocmd = "1"
localvimrc_python2_enable = "1"
localvimrc_python3_enable = "1"
localvimrc_debug = "99"
localvimrc_debug_lines = "100"
localvimrc_checksum_func = "function('sha256')"
localvimrc_python_command = "'not checked for python'"
== END settings ==================================
autocommand triggered on event BufWinEnter
== START LocalVimRC() ============================
[*]running for file "/tlist/app.py"                                                       
searching directory "/tlist"
found files: [{'resolved': '/tlist/.lvimrc', 'unresolved': '/tlist/.lvimrc'}]
candidate files: [{'resolved': '/tlist/.lvimrc', 'unresolved': '/tlist/.lvimrc'}]
processing "/tlist/.lvimrc"
stored information: answer = '' sandbox answer = '' checksum = ''
checksum calc -> /tlist/.lvimrc : c5a5370ac5aaeda38b0d84ac46b3ba9fc7cfafefe4cf2c51f0f62affcbd357d8
checksum mismatch, not reusing answer
[**]g:localvimrc_file = /tlist/app.py, g:localvimrc_file_dir = /tlist 
g:localvimrc_script = /tlist/.lvimrc, g:localvimrc_script_dir = /tlist
g:localvimrc_script_unresolved = /tlist/.lvimrc, g:localvimrc_script_dir_unresolved = /tlist
g:localvimrc_sourced_once = 0, g:localvimrc_sourced_once_for_file = 0
using sandbox                                                                                      
sourcing script, command is: "sandbox source /tlist/.lvimrc"                                      
checksum calc -> /tlist/.lvimrc : c5a5370ac5aaeda38b0d84ac46b3ba9fc7cfafefe4cf2c51f0f62affcbd357d8
== END LocalVimRC() ==============================               

Lines starting with [*] and [**] have the following differences when vim is run without a filename argument. Rest of the logs are the same.

running for file ""
g:localvimrc_file = , g:localvimrc_file_dir = . 

Starting without a filename and opening the file with :e app.py does not make any differences.

Starting with a filename such as poetry run vim ./app.py is as the same as running with a filename.

[embear]

Starting without a filename and opening the file with :e app.py does not make any differences. By that you mean that die output of :python3 import sys; print(sys.path) is still the same as when you start vim without a filename ?

From what I (believe to) understand the problem is twofold here:

1. localvimrc does only load .lvimrc files when an actual file is loaded (but which must not exist before running vim). The file location is used to determine all the local vimrc files that need to be loaded. The current working directory is never used for searching local vimrc files. This is by design and enables you to start vim in a directory that contains a local vimrc file but edit a file in a completely different part of the filesystem and you will get only the settings for the edited file from the local vimrc existing over there.
2. This part is guessing: When you are starting vim without an argument and are loading the file with :e app.py then the local vimrc file should be loaded and :echo g:python3_host_prog should print the correct path. But most likely this is too late and the changed variable is not used. It seems that the deoplete plugin uses this variable, right? If so, then there is an explanation and a possible solution for that in #87

[altunyurt]

Starting without a filename and opening the file with :e app.py does not make any differences. By that you mean that die output of :python3 import sys; print(sys.path) is still the same as when you start vim without a filename ?

1: poetry run vim
# inside vim session 
2: :python3 import sys; print(sys.path)
3: :e app.py 
4: :python3 import sys; print(sys.path)

The output of 2 and 4 are the same and includes the correct path, that is overriden in .lvimrc

1: poetry run vim app.py
2: :python3 import sys; print(sys.path)

3: poetry run vim ./app.py  # relative or absolute path 
4: :python3 import sys; print(sys.path)

Output of 2 and 4 are also same and contains the default path, even though it is supposed to be overriden in .lvimrc

From what I (believe to) understand the problem is twofold here:

1. localvimrc does only load `.lvimrc` files when an actual file is loaded (but which must not exist before running vim). The file location is used to determine all the local vimrc files that need to be loaded. The current working directory is **never** used for searching local vimrc files. This is by design and enables you to start vim in a directory that contains a local vimrc file but edit a file in a completely different part of the filesystem and you will get only the settings for the edited file from the local vimrc existing over there.

.lvimrc is loaded on all occasions (verified by adding 'echo' statements in .lvimrc), but the path overriding only occurs when vim is started with an empty buffer, without providing any file arguments. If a filename provided on command line, even though the .lvimrc is loaded, the python path is not overriden by the one declared in .lvimrc. Or it might be that, somehow vim might be re-overriding the one defined in .lvimrc again, setting the path to default one.

2. This part is guessing: When you are starting vim without an argument and are loading the file with `:e app.py` then the local vimrc file should be loaded and `:echo g:python3_host_prog` should print the correct path. But most likely this is too late and the changed variable is not used. It seems that the deoplete plugin uses this variable, right? If so, then there is an explanation and a possible solution for that in [Working example of using this plugin with ALE? #87](https://github.com/embear/vim-localvimrc/issues/87)

I'll check the issue.

[embear]

This is really weird. Interesting would be the value of :echo g:python3_host_prog in cases 2 and 4 of both scenarios. It should be the set and have the value you are setting in your .lvimrc. If not then there is a problem with the directly with the localvimrc plugin. If the variable values are the same and correct value in all four cases that would suggest a race condition. What is plugin that evaluates the variable g:python3_host_prog? Is it deoplete? If so, could you please add a echom to your .lvimrc and to autoload/deoplete/init.vim right where the variable g:python3_host_prog is assigned to python3 (should be around line 41. After that please check if the order of the two messages changes in both scenarios (could be checked with :messages)?

[altunyurt]

The variable values are not the same in all 4 cases. I tried to group the ones that have the same output, but i should have also provided the relevant outputs. Here are the outputs and scenarios all together.

I'm running vim in /tlist directory, where there sits a .lvimrc with the following contents:

# .lvimrc 
let cwd = getcwd()
let g:python3_host_prog = cwd . "/.venv/bin/python"

This is the expected result, but i have to start vim without a file name argument, then open the file to edit.

1: poetry run vim
# inside vim session 
2: :python3 import sys; print(sys.path)
3: :e app.py 
4: :python3 import sys; print(sys.path)

['/home/den/.pyenv/versions/3.10.4/lib/python310.zip', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10/lib-dynload', 
'/tlist/.venv/lib/python3.10/site-packages',           <= this is what i need 
'_vim_path_'] 

This is the unexpected result. When i run vim with a file name argument, the python path is set to default environment path.

1: poetry run vim app.py
2: :python3 import sys; print(sys.path)

3: poetry run vim ./app.py  # relative or absolute path 
4: :python3 import sys; print(sys.path)

['/home/den/.pyenv/versions/3.10.4/lib/python310.zip', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10', 
'/home/den/.pyenv/versions/3.10.4/lib/python3.10/lib-dynload', 
'/home/den/.pyenv/versions/editors/lib/python3.10/site-packages',  <== default host prog 
'_vim_path_']  

I don't have deoplete plugin installed. Running ag python3_host_prog in .vim only gives the following output. Apparently none of my installed plugins are setting the python3_host_prog

plugged/ncm2-jedi/README.md
16:specified by `:help g:python3_host_prog`.

plugged/nvim-yarp/autoload/yarp/pyx.vim
46:    let g:_yarp_py3 = expand(get(g:, 'python3_host_prog', ''), 1)
64:        call self.error("###### Please configure g:python3_host_prog properly ######")

plugged/nvim-yarp/README.md
12:  - `g:python3_host_prog` pointed to your python3 executable, or `echo

plugged/ncm2/doc/ncm2.txt
44:  `let g:python3_host_prog=/path/to/python/executable/` in vimrc.

plugged/ncm2/README.md
29:  `let g:python3_host_prog=/path/to/python/executable/` in vimrc.

I only once set the g:python3_host_prog in .vimrc, but i expect it to be overwritten by .lvimrc which happens only when i start vim without a file name.

.vimrc
184:let g:python3_host_prog = '/home/den/.pyenv/versions/editors/bin/python'

EDIT: I also tried disabling all plugins but vim-plug and localvimrc, and the same issue still remains. Adding echom or echomsg to .lvimrc provides no extra messages

[embear]

Possibly I was not clear in my previous message: I was not asking for the output of your :python3 import sys; print(sys.path). I was asking for the output of :echo g:python3_host_prog! You made already clear in your first and second message, that Python outputs wrong paths. Essentially localvimrc just sets the variable g:python3_host_prog. What happens after that depends on plugins and their order or on neovim. I am trying to narrow down the cause of that difference in the Python output.

EDIT: It seems that g:python3_host_prog is handled by neovim and not by a plugin. So, when the :echo g:python3_host_prog outputs the same value in all scenarios (which it does in my quick test with neovim) then the localvimrc plugin itself can't do much about the behavior of your :python3 import sys; print(sys.path). Nevertheless I'd like to understand cause the problem and possibly find a workaround.

[altunyurt]

Possibly I was not clear in my previous message: I was not asking for the output of your :python3 import sys; print(sys.path). I was asking for the output of :echo g:python3_host_prog!

I'd misread your message, sorry. Here're the outputs:

Without file name argument

1: poetry run vim 
2: :echo g:python3_host_prog
/tlist/.venv/bin/python

3: :e app.py
4: :echo g:python3_host_prog
/tlist/.venv/bin/python

With file name argument

1: poetry run vim app.py 
2: :echo g:python3_host_prog
/tlist/.venv/bin/python

3:  poetry run vim ./app.py
4: :echo g:python3_host_prog
/tlist/.venv/bin/python

Indeed in all scenarios the python3_host_prog values are the same. I've focused on the module paths and missed this part :facepalm:

[embear]

When reading the documentation of g:python3_host_prog in neovim I think, that possibly a plugin gets activated before localvimrc and uses has("python3") or uses the python interpreter. According to the documentation changes to g_python3_host_prog will have no effect after that. This his highly dependent on your setup. Possibly you can use BufReadPre and/or BufNew to move local vimrc reading before other plugins? It is not trivial to solve such kind of dependency problems without having the ability to assign priorities to plugins.

[embear]

Any insights on this issue that could be shared here and might help others? Otherwise I'd close this issue soon.

[altunyurt]

Ithink It's as you said, somehow i'll need to force vim to load localvimrc before any plugin is activated. I don't have anymore information to add. For now i'm running vim then loading the file with :e . Thanks for the help and the valuable information on vim internals.