Closed fast-90 closed 2 months ago
I haven't used lspce with virtual environment, actually I have never used python's virtual environment. I'll first learn how to use it and then try to answer your question.
Thanks for giving lspce a try.
Alright, will have a close look on this one! I don't think my Elisp coding skills and LSP understanding is good enough to help with the code, but if I can help in any other way please let me know.
Thanks. If you have any other sugguestions/problems, feel free to create issues. Thanks in advance.
Hi @fast-90 Sorry for late feedback, I finally found time to try to solve this problem, and I guess I have some progress now.
I pushed a new branch virtualenv
, where lspce supports workspace/didChangeConfiguration
notification. Using this new feature, lspce can send some workspace configuration (read from .dir-locals.el) to lsp servers (very similar to what eglot does, actually I learnt this from eglot).
Then I did some test. The following is what I did:
~/tmp/venv
mkdir -p ~/tmp/venv && cd ~/tmp/venv/ && git init .
python -m venv .venv
source .venv/bin/activate
pip install baostock -i https://pypi.tuna.tsinghua.edu.cn/simple/ --trusted-host pypi.tuna.tsinghua.edu.cn
((python-mode
. ((lspce-workspace-configuration
. (:pylsp (:plugins (:jedi_completion (:include_params t
:fuzzy t)
:jedi (:environment "/home/lucency/tmp/venv/.venv/bin/python")
:pylint (:enabled :json-false)))))
(indent-tabs-mode . nil)))
(python-ts-mode
. ((lspce-workspace-configuration
. (:pylsp (:plugins (:jedi_completion (:include_params t
:fuzzy t)
:jedi (:environment "/home/lucency/tmp/venv/.venv/bin/python")
:pylint (:enabled :json-false)))))
(indent-tabs-mode . nil))))
import baostock as bs
import pandas as pd
lg = bs.login()
print('login respond error_code:'+lg.error_code) print('login respond error_msg:'+lg.error_msg)
7. Open bs.py in Emacs and enable `lspce-mode`
When I inputted "bs.", completion poped up, and since I only installed baostock in this virtualenv but not the system, I guessed lspce recognised virtualenv now.
Is this what you wanted? If not, what did you want actually?
Hey @zbelial I experience the same problems on my mac with Python projects. I also use local .venv (i.e. .venv folder suited inside the project's folder) and poetry to activate it (see https://github.com/cybniv/poetry.el). So far it seems that the config you suggested solves the issue. Here is my setup:
((python-ts-mode
(lspce-workspace-configuration .
(:pylsp (:plugins (:jedi_completion (:include_params t :fuzzy t)
:jedi (:environment "/<path-to-project>/.venv/bin/python")
:flake8 (:enabled :json-false)
:rope_completion (:enabled :json-true)))))))
Hi @alkurbatov , thanks for letting me know that the branch virtualenv
works. I've not merged this branch into master because of no response(also because I don't write python), but now I think I can merge :) Thanks again.
BTW I stole the idea from eglot. Thank you, eglot.
Sorry, I haven't tested the particular branch. It works with the master and the config described above.
I can retest with the branch if needed.
It's wired. The master branch doesn't have lspce-workspace-configuration
, so maybe something happens somewhere else and makes it work. Can you confirm that without the above config, it doesn't work? Thanks.
It's wired. The master branch doesn't have
lspce-workspace-configuration
, so maybe something happens somewhere else and makes it work. Can you confirm that without the above config, it doesn't work? Thanks.
I confirm that it doesn't work without lspce-workspace-configuration and :jedi (:environment
in the master branch. I can't inspect imports and autocompletion doesn't see data from venv.
Although after your comment it looks weird that the setting helps.
I'll doublecheck with the virtualenv branch just in case.
'll doublecheck with the virtualenv branch just in case.
Thanks @alkurbatov .
Oh, well, things got more weird.
If I use the virtualenv branch, the overal experience looks similar to the master branch: autocompletion works, I can inspect some of the modules, no errors reported by pylsp (e.g. unknown imports).
However:
M-.
(might be related to xref) in both branches.Running backends: python-flymake, flymake-collection-flake8, flymake-collection-mypy, lspce-flymake-backend
xref--not-found-error: No definitions found for: SIGTERM
QuitError during redisplay: (jit-lock-function 662) signaled (args-out-of-range "" 0)
Error during redisplay: (jit-lock-function 660) signaled (args-out-of-range "" 0)
Error during redisplay: (jit-lock-function 1338) signaled (wrong-type-argument listp t)
See the screenshot:
Some times it works, but often don't. This error not happens in the master branch as I can't simply jump into the problematic string.
And errors like Error during redisplay: (jit-lock-function 660) signaled (args-out-of-range "" 0)
happen quite often with different numbers in the background.
Thanks a lot!
Some times it works, but often don't. This error not happens in the master branch as I can't simply jump into the problematic string.
Maybe it's because virtualenv branch is out of sync with master ( just a guess though), I'll merge master with it.
I'm not sure why redisplay
errors happen, I think I'll test it first, when it works, I'll let you know, then you can give it another try.
Hi @alkurbatov , before digging into the problem, I need to know some things about how did you test the virtualenv branch, thanks.
virtualenv
branch, did you rebuild the lspce module ?Hi @zbelial
I tested again in Konsole on Linux in almost the same steps quoted below, but had no luck to reproduce what you had. I also installed python-poetry, poetry.el and pyvenv.el but haven't had time to test with them (unfamiliar with them since I'm not a python programmer). Could you give a recipe to reproduce the issue please?
An image showing what I got when testing
Hi @fast-90 Sorry for late feedback, I finally found time to try to solve this problem, and I guess I have some progress now.
I pushed a new branch
virtualenv
, where lspce supportsworkspace/didChangeConfiguration
notification. Using this new feature, lspce can send some workspace configuration (read from .dir-locals.el) to lsp servers (very similar to what eglot does, actually I learnt this from eglot).Then I did some test. The following is what I did:
1. Create a new project `~/tmp/venv`
mkdir -p ~/tmp/venv && cd ~/tmp/venv/ && git init .
2. Create a new virtual environment in this new project's directory
python -m venv .venv
3. Activate the virtualenv
source .venv/bin/activate
4. Install some python packages using pip
pip install baostock -i https://pypi.tuna.tsinghua.edu.cn/simple/ --trusted-host pypi.tuna.tsinghua.edu.cn
5. Create .dir-locals.el and fill it with
((python-mode . ((lspce-workspace-configuration . (:pylsp (:plugins (:jedi_completion (:include_params t :fuzzy t) :jedi (:environment "/home/lucency/tmp/venv/.venv/bin/python") :pylint (:enabled :json-false))))) (indent-tabs-mode . nil))) (python-ts-mode . ((lspce-workspace-configuration . (:pylsp (:plugins (:jedi_completion (:include_params t :fuzzy t) :jedi (:environment "/home/lucency/tmp/venv/.venv/bin/python") :pylint (:enabled :json-false))))) (indent-tabs-mode . nil))))
6. Create a new python file bs.py
import baostock as bs import pandas as pd #### 登陆系统 #### lg = bs.login() # 显示登陆返回信息 print('login respond error_code:'+lg.error_code) print('login respond error_msg:'+lg.error_msg)
7. Open bs.py in Emacs and enable `lspce-mode` When I inputted "bs.", completion poped up, and since I only installed baostock in this virtualenv but not the system, I guessed lspce recognised virtualenv now.
Is this what you wanted? If not, what did you want actually?
The issues listed above happen not during autocompletion, but when I click M-.
on particular import.
E.g. the issue on my screenshot was found when I clicked M-.
on SIGTERM
in:
from signal import SIGTERM
No need in poetry or venv as signal
is builtin module.
The issues listed above happen not during autocompletion, but when I click M-.
on particular import.
E.g. the issue on my screenshot was found when I clicked M-.
on SIGTERM
in:
from signal import SIGTERM
No poetry or venv need as signal
is builtin module.
Unfrotunately still no luck to reproduce your problem. I created a simple project with only one python file, and there was only a line of code from signal import SIGTERM
.
But I had another problem, M-.
couldn't find SIGTERM's definition. I'm not sure what should be blamed, lspce or pylsp (pyright worked well though). And I didn't have any errors about redisplay.
Sorry for not making any progress, but I don't know what I can do now.
But I had another problem, M-. couldn't find SIGTERM's definition.
Yeah we have slightly different behaviors:
M-.
cannot be used on SIGTERM and on some other modules too.M-.
on SIGTERM but usually it fails.I believe that since we have such difference something in the virtualenv branch works better.
I think we should reduce the variance to make things simpler. @zbelial could you please merge the virtualenv branch?
I just created a new branch master-virtualenv
, which was based on master, and I have merged the virtualenv
branch into it.
Ok, so what I have now:
With the master-virtualenv
branch and your code example:
a. Attempt to press M-.
on SIGTERM
in from signal import SIGTERM
produce user-error: No definitions found for: SIGTERM
in the Warnings buffer. No other side effects.
b. Attempt to get autocompletion in imports (e.g. type from pandas import D
and press TAB) doesn't work.
c. Attempt to call autocompletion on methods (e.g. type lg = bs.login()
and on next line lg.
and press TAB) doesn't work.
Tested on Emacs 30 and Python 3.11 and 3.12 with poetry and your pylsp config.
However, on my other more complex Python project autocomplete features work with some modules. Probably this is pylsp issue on mac? No idea.
For comparison I tried pyright, but seems that this is another difficult quest :) https://github.com/zbelial/lspce/issues/40
Ok, so what I have now:
With the
master-virtualenv
branch and your code example:a. Attempt to press
M-.
onSIGTERM
infrom signal import SIGTERM
produceuser-error: No definitions found for: SIGTERM
in the Warnings buffer. No other side effects.
Maybe pylsp has some problems in this case. I'm not sure.
b. Attempt to get autocompletion in imports (e.g. type
from pandas import D
and press TAB) doesn't work.
What do you bind TAB to? Here after typing from pandas import D
and then calling completion-at-point
(bound to C-M-i) worked.
c. Attempt to call autocompletion on methods (e.g. type
lg = bs.login()
and on next linelg.
and press TAB) doesn't work.
Worked too for me (after lg.
, completion menu appeared automatically, no need to press TAB)
Tested on Emacs 30 and Python 3.11 and 3.12 with poetry and your pylsp config.
However, on my other more complex Python project autocomplete features work with some modules. Probably this is pylsp issue on mac? No idea.
Ok, so now I can confirm that lspce works well with pyright and everything can be resolved and stepped in. So my guess is that pylsp doesn't play nice on macOS.
Great! But I think virtualenv
needs to be improved (I learnt more knowledge about lsp configuration :) ) and then It can be merged with master.
What do you bind TAB to?
(define-key company-mode-map [remap indent-for-tab-command]
#'company-indent-or-complete-common)
;; Enable indentation+completion using the TAB key.
;; completion-at-point is often bound to M-TAB.
(setq tab-always-indent 'complete)
I tried TAB, M-TAB and direct call to completion-at-point
- no effect. Popup don't appear too.
What do you bind TAB to?
(define-key company-mode-map [remap indent-for-tab-command] #'company-indent-or-complete-common) ;; Enable indentation+completion using the TAB key. ;; completion-at-point is often bound to M-TAB. (setq tab-always-indent 'complete)
I tried TAB, M-TAB and direct call to
completion-at-point
- no effect. Popup don't appear too.
I don't think I can solve this problem ATM. Let's just keep this issue open, maybe other guys will tell us what the problem is.
I think I found out why it doesn't work with the master-virtualenv branch. The issue was on my side: the venv wasn't activated properly. So in short: it would be better to merge master-virtualenv (or some parts of it) into master to make scenarios like #a work (described below).
Currently I have several behaviors: a. When I activate venv and call Emacs from the same terminal, everything works well with pylsp even without the config (might be because pylsp runs in the same venv).
b. When I call Emacs from UI, autocompletion and "jump into module" doesn't work with or without lspce-workspace-configuration setting.
c. When I call Emacs from terminal without venv activation, it behaves the same as in #b.
d. If I use the current master branch, #a doesn't work.
Speaking about the desired behavior: usually people expect that LSP will handle venv for them without manual activation of anything. To achieve it one should:
I don't understand why a
does not work when using master branch. It's so wired since if there isn't config, these two branch should behave in the same way. I'm not sure I should merge it to the master or not.
The other reason why I hesitate to merge is I hadn't learn enough about how workspace/configuration
works when I implemented this feature. I need more time to think about how to improve it.
I totally agree with you that lsp client should handle things like venv automatically, I think this time is the chance to solve it in a better way than the current.
BTW, since I'm not familiar with python and tools around it, if you'd like to contribute in this aspect, it'd be greet. :)
BTW, since I'm not familiar with python and tools around it, if you'd like to contribute in this aspect, it'd be greet. :)
That could be hard as I am not familiar with Rust and barely know Elisp. I'll try to inspect the code first.
Hi, that's ok, don't worry. I'll try to solve the issue, but I guess it may need some time though.
Hi @alkurbatov , I just got virtualenv work in lspce (only worked with pylsp and jedi-language-server, but not pyright-langserver ATM). This new method doese not depend on pyvenv or anything similar, but just a .dir-locals.el
.
The following is what I did (using master branch):
.dir-locals.el
at the root of the project
((python-mode
. ((eval . (progn
(setq-local lspce-jedi-environment "/home/XXXX/tmp/venv/.venv/bin/python")))))
(python-ts-mode
. ((eval . (progn
(setq-local lspce-jedi-environment "/home/XXXX/tmp/venv/.venv/bin/python"))))))
(defun lspce-pylsp-initializationOptions ()
(let ((options (make-hash-table :test #'equal)))
(lspce--add-option "pylsp.plugins.jedi_completion.include_params" t options)
(lspce--add-option "pylsp.plugins.jedi_completion.fuzzy" t options)
(when (boundp 'lspce-jedi-environment)
(lspce--add-option "pylsp.plugins.jedi.environment" lspce-jedi-environment options))
(lspce--add-option "pylsp.plugins.pylint.enabled" :json-false options)
options))
lspce-server-programs
(setq lspce-server-programs `(("python" "pylsp" "" lspce-pylsp-initializationOptions)))
lspce-mode
in the buffers of this project, pylsp should workThis method does not work for pyright though. I have no idea why.
If this works for you too, I'd like to close this issue. There are so many different ways to support virtualenv in Python world, and I've spent too much time on this.
Some notes about pyright.
A pyright developer said pyright will read pyrightconfig.json at the root of a project automatically, but it doesn't work for me. Its documentation says you can set python.pythonPath in settings and it will firgure out the environment, but unfortunately it doesn't work for me too. Maybe it's all my fault, I don't know. And maybe some day I'll continue to add support for pyright working with virtualenv. Now let's just call it an end.
Its documentation says you can set python.pythonPath in settings and it will firgure out the environment, but unfortunately it doesn't work for me too.
I also tried this and it didn't work. Pretty sure this is about VSCode config, not pyright.
only worked with pylsp and jedi-language-server, but not pyright-langserver ATM
I bet this is because of the way pyright identifies venvs. According to the docs, pyright uses the current available python interpreter to calculate path to venv. This is why they suggest to set pythonPath via VSCode.
If this works for you too, I'd like to close this issue. There are so many different ways to support virtualenv in Python world, and I've spent too much time on this.
Tested:
I think we have some differences in the setups that cause confusion:
pipx install python-lsp-server
). With it I had all the results described above.brew install python-lsp-server
), but it worked exactly the same.My guess:
Anyways I am more interested in pyright as pylsp has several weird bugs which are blockers to me :(
If this works for you too, I'd like to close this issue. There are so many different ways to support virtualenv in Python world, and I've spent too much time on this.
Tested:
* Worked if I do preliminary venv activation in the terminal. * It didn't work without manual venv activation or with GUI version of Emacs.
I should have mentioned that I didn't activate venv at all. Just used the .dir-locals.el file, It worked well in my test project. Let me summary.
Created a new project ~/tmp/venv
Created a new virtual environment in this new project's directory
Activated the virtualenv to install some python packages only in this virtual env.
All the above happened in terminal. Now this project had been prepared. Then I started Emacs in GUI mode, created .dir-locals.el
and opened a new file bs.py
, and input all the code mentioned above. Code completion, jumping to definitions all worked well.
I think we have some differences in the setups that cause confusion:
* My usual setup with pylsp is a separate venv created by pipx (i.e. `pipx install python-lsp-server`). With it I had all the results described above. * Just in case I tested your suggestion with the pylsp from homebrew (i.e. `brew install python-lsp-server`), but it worked exactly the same.
My guess:
* On Linux pylsp is set up in a different way, if you install it using package manager. Probably you should try to install pylsp with pipx. * There is some difference in a way venv activated. I have a feeling that this jedi setting doesn't help at all. Probably you use jedi language server?
Anyways I am more interested in pyright as pylsp has several weird bugs which are blockers to me :(
Yes, in the above comment, I used pylsp installed from Manjaro's repos, but I just uninstalled it and reinstalled using pipx, it workd too.
All in all, as I mentioned, there are different ways to activate venv in Emacs, I don't want to add support for all, so I chose the way to use .dir-locals.el
only. No need to activate venv at all.
Since pyright keeps only vscode in mind, I'd like to delete code about it from lspce actually. Sorry.
BTW, I tried pyvenv.el, and I think it enables a venv globally in Emacs, right? It's not flexible at all IMHO. But I might be wrong since I'm not familar with it.
I should have mentioned that I didn't activate venv at all.
In this case the suggested approach didn't work for me.
Since pyright keeps only vscode in mind,
I don't think they do, but VSCode is very popular. There should be the way to specify Python path somehow.
I'd like to delete code about it from lspce actually. Sorry.
I would suggest to burrow ideas from eglot as it works well with pyright.
BTW, I tried pyvenv.el, and I think it enables a venv globally in Emacs, right? It's not flexible at all IMHO. But I might be wrong since I'm not familar with it.
Me too, but I think you're right. They patch internals a lot.
I should have mentioned that I didn't activate venv at all.
In this case the suggested approach didn't work for me.
sadly.
Since pyright keeps only vscode in mind,
I don't think they do, but VSCode is very popular. There should be the way to specify Python path somehow.
At least a developer said in an issue before. I'm pretty sure about this. Maybe they have changed their mind now.
I'd like to delete code about it from lspce actually. Sorry.
I would suggest to burrow ideas from eglot as it works well with pyright.
Maybe some day. Actually I have tested pyright with eglot, it worked if I activated the venv before enabling eglot, otherwise it didn't work too.
Actually I have tested pyright with eglot, it worked if I activated the venv before enabling eglot, otherwise it didn't work too.
With eglot I know two ways to make automatic activation of venv with pyright:
I think that the issue "pyright doesn't read config by unknown reason" is what blocks it's support.
Just captured what I did to make the test.
https://github.com/user-attachments/assets/4a07fcfd-947a-4ee1-8d89-31f836db28e4
I also made pyright work with venv, but I'm not sure it's good enough or not. What you need is .dir-locals.el
too. Like the following:
((python-mode
. ((eval . (progn
(setq-local exec-path (append (list "/home/XXXX/tmp/venv/.venv/bin/") exec-path))
(setq-local lspce-inherit-exec-path t)))))
(python-ts-mode
. ((eval . (progn
(setq-local exec-path (append (list "/home/XXXX/tmp/venv/.venv/bin/") exec-path))
(setq-local lspce-inherit-exec-path t))))))
In this way, pyright can find the python interpreter in the venv.
Hi @alkurbatov , does the .dir-locals.el
work for you now? If not, could you please provide a step-by-step reproducible recipe? Or I'll close this issue ( and #41 ). But since I use Linux, even you provide a recipe, I couldn't promise to reproduce your problem.
I am unsure whether I should post my issue here or create a new one instead. The problem I am facing is about the compatibility with direnv.
Basically it provides a separate development environment within a specific directory, such as binaries and environment variables. By utilizing integration packages, Emacs or shells can automatically load the corresponding development environment if it is under the directory. I am now using it for rust development, so that different projects can have different rust versions.
With the help of dir-locals
and lspce-inherit-exec-path
, lspce
can be started successfully, but rust-analyzer
can not be compiled successfully, as indicated in the following log:
The system library `openssl` required by crate `openssl-sys` was not found.
[stderr] The file `openssl.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
[stderr] The PKG_CONFIG_PATH environment variable is not set.
As per the log, the problem seems to be that lspce
is unable to recognize the environment variables provided by direnv
. Do you have any suggestions on how to resolve this issue? I can also provide steps for reproducing the problem if you want.
I am unsure whether I should post my issue here or create a new one instead. The problem I am facing is about the compatibility with direnv.
Basically it provides a separate development environment within a specific directory, such as binaries and environment variables. By utilizing integration packages, Emacs or shells can automatically load the corresponding development environment if it is under the directory. I am now using it for rust development, so that different projects can have different rust versions.
With the help of
dir-locals
andlspce-inherit-exec-path
,lspce
can be started successfully, butrust-analyzer
can not be compiled successfully, as indicated in the following log:The system library `openssl` required by crate `openssl-sys` was not found. [stderr] The file `openssl.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory. [stderr] The PKG_CONFIG_PATH environment variable is not set.
As per the log, the problem seems to be that
lspce
is unable to recognize the environment variables provided bydirenv
. Do you have any suggestions on how to resolve this issue? I can also provide steps for reproducing the problem if you want.
Hi @WeissP, I guess it's because lspce only handles PATH now(actually I've thought about implementing this feature in a more general way so lspce can handle any environment variables but I was too lazy to implement it, haha), so other environment variables do not work as expected. Also It'll be great if you could provide steps about how to reproduce the problem so when I make a fix, I can make use of your reproducing recipe, but there's no hush.
And I'm a little busy these days, so I'm not sure when I can fix this problem. This weekend or maybe till next weekend, I could not promise.
Hi, thanks for the quick reply. I guess using current_dir could be helpful. Regarding steps for reproducing, it turned out that my approach involves a bunch of packages you will have to install (nix + nix-direnv + devshell + emacs-direnv), and I am also not sure about potential conflicts with your global Rust ecosystem (which make it harder to debug and test).
So I guess if it is possible to make pyvenv
work without passing only environment variable PATH
but by forwarding all environments under a user specified directory, my issue could also be solved. I am willing to test this in my own environment.
If you are still interested in reproducing and testing the issue yourself, I am more than willing to provide you with detailed instructions :).
Hi, @WeissP , thanks.
Hi, thanks for the quick reply. I guess using current_dir could be helpful.
I might have used current_dir
but had no luck. But I'm a bit hazy on that.
Regarding steps for reproducing, it turned out that my approach involves a bunch of packages you will have to install (nix + nix-direnv + devshell + emacs-direnv), and I am also not sure about potential conflicts with your global Rust ecosystem (which make it harder to debug and test).
So I guess if it is possible to make
pyvenv
work without passing only environment variablePATH
but by forwarding all environments under a user specified directory, my issue could also be solved. I am willing to test this in my own environment.If you are still interested in reproducing and testing the issue yourself, I am more than willing to provide you with detailed instructions :).
I'll try to fix this issue and If I have something to test, I'll ping you.
I have a question though: in your use case, how to decide which environemnt variables should be passed to lspce to start the lsp server? All that direnv is managing?
I am trying to use this with pylsp, and LSP is working since flymake is providing error diagnostics. However, it seems that it doesn't recognise my virtual environment (Eglot was able to recognise the virtual environment with my setup, see below).
Is there a way to configure it so that it recognises the virtual environment? How have you set this up for yourself?
For reference, my setup is:
.venv
folder, which is located in the project folder.pyvenv
to activate the the correct virtual environment.Please let me know if I need to provide any further information. This looks cool and I'm excited to try it out!