Canonical way to create a kernel when creating a new conda environment

[tdpetrou]

I'm looking to see if there is a more standard, "canonical", way to create a kernel when creating a conda environment. It seems to be a few answers that abound, but no standard. This would really help for those that teach classes or create tutorials.

As a concrete example, I will go through what happens on my mac. Let's say I create a conda environment with Python and Jupyter installed

$ conda create -n test_env python jupyter

This installs ipykernel as a dependency. I then activate the environment and launch a jupyter notebook

$ conda activate test_env
$ jupyter notebook

When the server is started, creating a notebook by clicking the 'New' button reveals a dropdown menu with 'Python 3' as the only choice (let's assume there are no other environments). Launching a new notebook from here will not execute Python from the test_env but instead from the base environment.

This seems counterintuitive, but I can get a list of the kernels to get more info with the following:

$ jupyter kernelspec list
python3       /Users/Ted/Library/Jupyter/kernels/python3

Reading the docs on the kernel specs , I see that this is part of the User path which takes precedence over the others. It appears that Env is next, followed by System.

This means that no matter what environment I am in, the base environment will take precedence for the 'python3' kernel. Now, if I look at the location where the Env kernel is, I see that a 'python3' kernel is also listed there.

$ ls /Users/Ted/anaconda3/envs/test_env/share/jupyter/kernels
python3

The most popular solution at this point appears to be to manually create a kernel with:

$ python -m ipykernel install --user --name test_env --display-name "Python (test_env)"

So, this works, and now I can access my test_env Python installation by starting a notebook with this kernel. But, it appears this is rather messy and an alternative (better?) solution would be to have no User kernels at all.

From the first response of this GitHub issue, I can just remove the User 'python3' kernel. As far as I know, this is the only documented place where this advice is given. So I do this with:

$ jupyter kernelspec remove python3

Now, this kernel is removed so when running the command jupyter notebook, it will find the 'python3' kernel for the current environment as it is next in line. Creating a new notebook now executes Python from the installation in the active environment. Moreso, any new environment will automatically be set up so that the 'python3' kernel is pointing to its environments location.

The User 'python3' kernel does not always exist

A student of mine told me that he had no User 'python3' kernel so launching his jupyter notebooks after environment creation automatically started in the 'correct' place. Apparently this is the case for some users.

If anyone could explain why some people have User 'python3' kernels and others don't that would be great.

More questions

I know this is a long post, but I feel there is lots of confusion surrounding conda environment creation and kernels.

• It seems strange to even have the opportunity to run a different kernel other than the one created specifically for the environment where jupyter is launched from. Is there a purpose for this?
• To me it would appear much more intuitive that when you begin a jupyter notebook within an environment, that environment's kernel is prioritized. Shouldn't the environment have higher priority?
• Which solution could be more canonical? Running python -m ipykernel install --user --name test_env or jupyter kernelspec remove python3?
• Is it possible that I created the User 'python3' kernel unknowingly? It does seem like many people have issues with this.

I made a similar post on Stack Overflow detailing my solution.

[jasongrout]

I'll add some more context: The original Jupyter precedence for configuration is user config overrides sysprefix config overrides system config. This is appropriate when the sysprefix level is a python distribution installed in, say, /usr/local/, as is often the case with a system python installation (i.e., a system has many top-level environments, and each environment can have many users). However, user-level environments turn this assumption partly upside down, since each environment has a separate sysprefix and you can now have multiple sysprefix environments for each user. Arguably, the appropriate precedence with virtual environments is sysprefix overrides user overrides system (since a system has many users, and a user has many environments).

[jasongrout]

As a note, I regularly delete the user-level config in ~/.jupyter, and concentrate on having only environment-level config. This can mess up some things with JupyterLab, though, since its settings are stored at the user-level by default too.

It would be nice if there was a way to switch Jupyter config to prefer user over environment, or vice versa. Even better would be a way to make an intelligent default, with the assumption that virtual environments should take precedence over user config.

[tdpetrou]

Thank you @jasongrout for the explanation. It does make sense that when a conda environment is active to have its kernel take precedence.

It seems that some people have a User-level 'python3' kernel while others do not. Do you have any clues why this is the case?

[jasongrout]

I don't know why there might be a difference, if they all followed the same instructions.

[jetilton]

I stumble my way through this every time I want to start a new project and then a lot of times I just use some bloated environment I already know works when I start a new project. I like to create my environments in the directory of my project so

$ conda create --prefix ./venv python=3.7
$ conda activate ./venv
$ conda install ipykernel

Is there a preferred way to go from here? I see lots of different routes to take. What I try to do is

$ python -m ipykernel install --prefix ./venv --display-name "project-name"

This seems to work because I get a: Installed kernelspec python3 in... but when I start up Jupyter lab from my other environment I do not see "project-name" as a kernel option.

Is there a better way to go about this?

I seem to have luck when I create an environment the standard anaconda location with:

$ conda create --name myenv python=3.7
$ conda install ipykernel
$ python -m ipykernel install --user --name myenv --display-name "myenv"

It is my preference to keep everything bundled together in the same directory so I would prefer the former to work as well.