[Feature Request] Handling of virtual environments

[mrossinek]

Is your feature request related to a problem? Please describe. I would mostly like to understand, how you handle virtual environments, since I have not been able to fully grasp this. Consider the following:

• I have a hard-coded g:python3_host_prog=~/Installations/pynvim/venv/bin/python as suggested in :help python-virtualenv
• I use virtual environments in all of my local projects
• when starting a kernel via :MagmaInit, I would like that it starts the kernel using the active virtual environment

Describe the solution you'd like As said above, I would like the kernel started by Magma to use the currently active virtual environment, from the shell in which Neovim was started.

Describe alternatives you've considered I would like to avoid having to install all the Neovim-related python packages in every environment and remove the python3_host_prog setting, but I assume that this would be an alternative.

Additional context Years ago, I implemented a similar project, deuterium, but it has less features than yours and is currently so out-of-date, that I was looking at alternatives before trying to revive it. I found your plugin to basically do everything I want, except the handling of virtual environments.

[dccsillag]

Hey there!

The current solution is indeed the alternative you've mentioned: installing the dependencies in the virtual environment. But I wholeheartedly agree with you -- users should be able to only install magma's dependencies in a specific virtual environment and have the Jupyter kernel run in the current (the shell's) activated virtual environment.

Now, I'm pretty sure that Jupyter would have to be installed either way in every virtual environment (because otherwise we wouldn't be able to run a Jupyter kernel on that environment), but all the other dependencies should not be necessary. The issue is, I just haven't been able to figure out how to do it.

By now, I have half a mind to rewrite magma in Rust -- besides making magma much more robust and easy to maintain, this would also almost certainly solve this issue, since all you'd need would be the statically linked plugin (and users acquiring the compiled plugin shouldn't be too difficult, I believe). There are just two major issues for this:

• Writing neovim plugins in Rust might be a bit tricky (there's been some nice progress on this front recently, but the frameworks for this are still a bit recent)
• There doesn't seem to be a maintained Rust crate to work with the Jupyter protocol from the client endpoint.

The latter point, I believe, is the trickier one. Here are the two viable solutions I see:

• Implementing (and maintaining) the Jupyter protocol myself

  I'm not a fan of this -- although Rust has good ZMQ support (I think), the Jupyter protocol, built on top of ZMQ, isn't particularly simple.

• Writing a small executable in Python that provides the ZMQ messages in the form of simple HTTP requests (or similar).

  It being an executable means that it would already run in its own virtual environment, and HTTP requests would be very easy to deal with in the Rust (or eventually Lua, or whatever) front.

In conclusion: this is something I'd really like to solve, but I don't know how. If you have any ideas, please share!

[mrossinek]

Hi again! That's great to hear that you are interested in figuring this one out! I have to admit though, that it has been a few years since I looked at the code of my solution..

Now, I'm pretty sure that Jupyter would have to be installed either way in every virtual environment

I think ipykernel and jupyter_client might be sufficient, although I am not sure, what would be the "benefit" over simply installing jupyter everywhere :smile:

I have to say that I have not worked with Rust for Neovim plugins before, so I cannot speak to that.

If you have any ideas, please share!

The only thing I can share at this point, is a link to how I handle things in Deuterium: https://gitlab.com/mrossinek/deuterium/-/blob/master/autoload/deuterium.vim I think the "trick" I used there, was to start the jupyter kernel via a system call from within vim and not from within Python, which allows the shell to figure out the "correct" Python path. Not sure if any of the code there is of any help to you, though..

[dccsillag]

Ah, yeah, that makes sense. I think the Python Jupyter kernels are somewhat "blessed" by Jupyter in that they are provided by the Jupyter library itself, and not registered as kernels as would, say, IRuby or IHaskell. So it makes sense that starting the Jupyter kernel via some sort of system call instead of via Neovim's Python would solve this.

As it happens, it sounds like my idea of rewrite in Rust + write a small executable in Python that provides the Jupyter protocol in the form o simple HTTP requests would work quite well :smile: (but I'm still not sold that a full-on rewrite is the way to go).

I'm a bit short on time right now, but I'll look into this as soon as I'm able to.

[crzdg]

Would love to see this as well. Did somebody manage to come up with a workaround to tackle this shortcoming?

Unfortunately, without this, using magma is very limited for my use cases.

[phcerdan]

I would like that it starts the kernel using the active virtual environment

That's the only missing part that jupyter notebook does automatically but Magma doesn't. Magma only sees the "registered" kernels. By default there is only one python3 from /usr/bin/python3.

You would have to create a local jupyter kernel for your virtual environment, and select that with Magma: Workaround workflow: Preliminaries: set up python3_host_prog to an environment with all the magma dependencies:

In a pynvim or magma venv, install all the magma dependencies:

workon magma # Activate venv
pip install pynvim pydebug kaleido plotly jupyter-client Pillow cairosvg pnglatex

Set python3_host_prog to that venv. Don't forget the expand if using ~.

vim.g.python3_host_prog=vim.fn.expand(~/.virtualenv/magma/bin/python3)

Finally, in your work virtual environment, with no magma or neovim packages:

workon workenv # Activate
pip install ipykernel
python -m ipykernel install --user --name workenv # Install ipykernel associated to the venv. Requires ipykernel

Then, usually you want to start nvim with your workenv activated, and then do :MagmaInit workenv.

I am sure this workflow can be improved, but it works for now.

[benlubas]

This is great! That setup works for me as well. I don't find it to be too cumbersome with an additional lua script for initialization:

vim.keymap.set("n", "<localleader>ip", function()
  -- find any active python virtual environments
  local venv = os.getenv("VIRTUAL_ENV")
  if venv ~= nil then
    -- in the form of /home/your_user/.virtualenvs/VENV_NAME
    venv = string.match(venv, "/.+/(.+)")
    vim.cmd(("MagmaInit %s"):format(venv))
  else
    vim.cmd("MagmaInit python3")
  end
end, { desc = "Initialize Magma for python3", silent = true, noremap = true })

Note that the exact path will vary per person, but the match that I wrote should be general enough to always work.

I'm editing this to add that opening neovim with an active python venv absolutely killed pyright. It was taking like 4+ seconds to answer a rename variable request. The solution was adding this to my pyright configuration.

settings = {
  python = {
    analysis = {
      diagnosticMode = "openFilesOnly",
    },
  },
},