Incorrect python-shell-interpreter since 3afc9afa4

[cremacs1]

Description :octocat:

Pressing M-m s i in a python buffer opens a Python interpreter rather than an IPython interpreter. This behavior appears to have been introduced with commit

3afc9afa4 layers/+lang/python: enahnce the virtual environment detection

Reproduction guide :beetle:

• Start Emacs
• Make sure python is enabled in dotfile
• Go to a python buffer and press M-m s i

Observed behaviour: :eyes: :broken_heart: A python interpreter opens

Expected behaviour: :heart: :smile: An ipython interpreter opens

System Info :computer:

• OS: darwin
• Emacs: 28.2
• Spacemacs: 0.999.0
• Spacemacs branch: develop (rev. c22d3d174)
• Graphic display: t
• Running in daemon: nil
• Distribution: spacemacs
• Editing style: hybrid
• Completion: helm
• Layers:

  ((auto-completion :variables auto-completion-enable-sort-by-usage t)
  bibtex
  (latex :variables latex-view-with-pdf-tools nil latex-enable-magic nil latex-enable-auto-fill nil)
  csv
  git
  julia lsp
  (python :variables python-backend 'lsp python-lsp-server 'pyright lsp-pyright-auto-import-completions nil)
  shell-scripts syntax-checking
  (version-control :variables version-control-global-margin t version-control-diff-tool 'git-gutter vc-follow-symlinks t)
  eww emacs-lisp html
  (javascript :variables javascript-fmt-on-save t javascript-fmt-tool 'web-beautify javascript-backend 'lsp)
  helm markdown)

• System configuration features: ACL GMP GNUTLS JSON LIBXML2 MODULES NATIVE_COMP NOTIFY KQUEUE PDUMPER THREADS TOOLKIT_SCROLL_BARS XIM ZLIB

Backtrace :paw_prints:

<<BACKTRACE IF RELEVANT>>

[lebensterben]

@sunlin7 Please help.

[sunlin7]

Do you use the virtual envrionment? The https://github.com/syl20bnr/spacemacs/commit/3afc9afa4cd113f1bafee74c4ecf04af58b20372 will respect the local virtual environment first. Before that, it will always try the ipython, but if we use the global ipython with a dir-local virtual-env, will lead errors for there maybe special package installed in local while the global ipython won't be able to load a pacakge in local virtual environment. Or, please give more details for your global/dir-local python informations.

[cremacs1]

I use conda, but I believe the issue is somewhere in the commit; reverting to b28d65b7a, python-shell-interpreter becomes "ipython", but using b28d65b7a (or HEAD), it has the value "python3". I also verified that the same behavior occurs when not using conda (removing the entire anaconda subtree from PATH) and on a fresh install of spacemacs.

[sunlin7]

Could you navigate to your project root directory in Spacemacs, and evaluate these three vaules? It will help to get close to the root cause. Thanks python-shell-interpreter spacemacs--python-shell-interpreter-origin (spacemacs/pyenv-executable-find "ipython")

[cremacs1]

python-shell-interpreter: "python3" spacemacs--python-shell-interpreter-origin: "python3" (spacemacs/pyenv-executable-find "ipython"): "/opt/homebrew/anaconda3/bin/ipython"

[sunlin7]

Hi @cremacs1 , According the code https://github.com/syl20bnr/spacemacs/blob/fb3b566164bbfd8c7405087d796bc3ba88250d96/layers/%2Blang/python/funcs.el#L153-L165 and with your comment, it should be able to locate the ipython and chose the ipython. What's the result of follow command line? /opt/homebrew/anaconda3/bin/ipython --version

[cremacs1]

After some debugging, I think I have found the issue. I normally use my base conda environment and don't explicitly activate a virtual environment when I open a project. This causes the check for root-dir to fail. Activating an environment in spacemacs (using pyenv) leads to the expected correct behavior.

I think that spacemacs should detect ipython correctly even without activating a virtualenv. Would this be possible?

[lebensterben]

as mentioned by @sunlin7 in his earlier comment:

The 3afc9af will respect the local virtual environment first. Before that, it will always try the ipython

I think this is the reasonable behavior. When a virtual environment is active, we should always use that virtual environment.

I normally use my base conda environment and don't explicitly activate a virtual environment when I open a project.

So the question is, why we find a virtual environment when no such virtual environment is activated?

[sunlin7]

Thanks for comments. Virtual environment should be in the first priority.

@cremacs1 thank you debugging the issue, and if you really want use the ipython in anaconda, just use follow line in your dot emacs file: (customize-set-variable '(python-shell-interpreter "/opt/homebrew/anaconda3/bin/ipython"))

[cremacs1]

Sorry for being unclear -- my base conda environment is always active. I meant that I don't normally activate a different environment.

I think I have located the fundamental issue. When no environment is activated from emacs, the check in the first if-let in spacemacs//python-setup-shell fails, so none of the interpreter selection logic executes. When I activate a conda environment using pyvenv-activate, I get the expected behavior.

Unfortunately I'm not sure how to fix this in general while respecting the original intent, but that seems to be the issue. A workaround is @sunlin7's customization, but I think it would be better to have a general solution. Thank you.

[lebensterben]

the easiest way to see whether Conda's python is active is to run the shell command "which python" and to see whether "conda" appears in the path.

If so, then the current python interpreter is Conda' and not from any virtual environment. (usually virtual environment's python is located in a shims such as ~/.pyenv/shims/python)

[cremacs1]

Yes, it is detecting conda's python (this is expected since I haven't activated a virtualenv in emacs). But I want conda's ipython (this is achievable either by manually setting python-shell-interpreter as in https://github.com/syl20bnr/spacemacs/issues/15873#issuecomment-1382919522 or by settting it simply to "ipython".

Prior to commit 3afc9afa4, spacemacs correctly detected conda's python without having to set python-shell-interpreter by hand. I still think the issue is what I detailed https://github.com/syl20bnr/spacemacs/issues/15873#issuecomment-1384226973 ; but I'm not sure how to fix it...

[lebensterben]

I just explained how to fix it.

if conda's python appear before virtual environment's, then virtual environment is not active. Then we should fall back to whatever the original logic we used before the said commit.

[danieldjewell]

I just ran into this issue as well ... Running inside of a git repository with an active pipenv project... Whether ipython is added to the project or not, it still runs python.

Additionally, I just tested on a project/repository without any virtualenv configuration whatsoever ... same result: ipython is not launched.

In my case, ipython is globally installed (i.e. it's path is /usr/bin/ipython)

@lebensterben

the easiest way to see whether Conda's python is active is to run the shell command "which python" and to see whether "conda" appears in the path.

This assumes that conda is installed to a default path and contains conda... There is no requirement that users use the default path ... A great example of this is using mamba ... my path is ~/.mambaforge/envs/envname/bin/python

I believe utilizing environment variables would be a better way to detect a conda venv -- notably CONDA_PREFIX ... I believe those variables are present across platforms (Windows/Linux/Mac?)

However, that is conda specific ... that variable isn't present if I run, for example, pipenv shell ... PIPENV_ACTIVE=1 is there though as is VIRTUAL_ENV. However, I don't believe there's a PEP or any other standard specifying that a specific environment variable MUST be present when inside of a venv (of any type).

It might be necessary to add detection logic for each variant of venv ... (pyenv, pipenv, conda, poetry, etc.)

[lebensterben]

It might be necessary to add detection logic for each variant of venv ... (pyenv, pipenv, conda, poetry, etc.)

This is painful.

It much easier if we remove all support for virtual environment and let user handle it themselves.

[danieldjewell]

It might be necessary to add detection logic for each variant of venv ... (pyenv, pipenv, conda, poetry, etc.)

This is painful. I don't disagree. I had some thoughts on this though...

It much easier if we remove all support for virtual environment and let user handle it themselves. Easier? Yes. Not sure that it's really something that the user can/will handle themselves...

There appears to be a reasonably easy way to detect non-conda virtualenvs from within Python itself (I tried this with venv and pipenv):

import sys

if sys.base_prefix != sys.prefix: 
  virtual_env = True

This does NOT work in a conda envrionment. However, Conda environments set the environment variable "CONDA_PREFIX" so that should be reasonably easy check.

Not sure about the thoughts on embedding Python code either directly in the lisp source and/or including a "helper" file written in Python to detect. Of course, the above example code would only work if the virtual environment is actually activated... (and thus the path to the python binary has been changed)

Edit: (it might still be necessary to detect say ... Pipenv... The "pipenv" oh-my-zsh plugin just looks for a "Pipenv" file to determine whether to auto-activate https://github.com/ohmyzsh/ohmyzsh/blob/master/plugins/pipenv/pipenv.plugin.zsh ...)

[sunlin7]

@cremacs1 please help verify the solution #16011 , thanks