kernels visible to conda but not to jupyter

gkovaig commented 4 years ago

OS: debian 9 Clean install on Google Cloud Platform running python notebook

Ran these commands: conda create -n fastai conda activate fastai

Created ~jupyter/.jupyter/jupyter_config.json with this content:

$ cat ~jupyter/.jupyter/jupyter_config.json { "kernel_spec_class": { "kernelspec_path": "" } }

$ python -m nb_conda_kernels list [ListKernelSpecs] [nb_conda_kernels] enabled, 3 kernels found Available kernels: conda-env-fastai-py /opt/conda/envs/fastai/share/jupyter/kernels/python3 conda-env-py27-py /opt/conda/envs/py27/share/jupyter/kernels/python2 python2 /opt/conda/share/jupyter/kernels/python2 conda-root-py /opt/conda/share/jupyter/kernels/python2 python3 /opt/conda/share/jupyter/kernels/python3

$ jupyter kernelspec list Available kernels: python2 /opt/conda/share/jupyter/kernels/python2 python3 /opt/conda/share/jupyter/kernels/python3

Attempted to shutdown and start instance. Same results. This is preventing me from seeing available kernels and choosing appropriate kernel after launching a notebook.

Appreciate any pointers on troubleshooting. Thank you!

mcg1969 commented 4 years ago

Can you share the output of python -m nb_conda_kernels.install --status --verbose?

And why did you create that jupyter_config.json file manually?

ahnchive commented 4 years ago

Same issue here! And here is the verbose from your code.

Entry points:

conda = nb_conda_kernels.discovery:CondaKernelProvider NOTE: entry points not used in Jupyter 6.1.6 jupyter_config.json entries:
/etc/jupyter:
/usr/local/etc/jupyter:
/home/usr/anaconda3/etc/jupyter: { "JupyterApp": { "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager" } }
/home/usr/.jupyter:

fcollonval commented 4 years ago

@gkovaig

If you want to install the kernel to make them available for other tools than notebook and jupyterlab. You indeed need to set a configuration file, jupyter_config.json that should contain:

{
  "CondaKernelSpecManager": {
    "kernelspec_path": "--user"
  }
}

Two main differences with what you reported:

The main entry should be the class name: CondaKernelSpecManager
The value for the path: --user => avoid "" as usually users do not have write permission on system /etc folder.

If you are really in a multi-user case, make sure the shared jupyter kernel directory is writable by your users.

@ahnchive have a look at the documentation: https://github.com/Anaconda-Platform/nb_conda_kernels#use-with-nbconvert-voila-papermill

joseph-long commented 4 years ago

I have the same issue as the OP.

Just installed nb_conda_kernels-2.2.4 | py37hc8dfbb8_0 23 KB conda-forge into my py37 environment where I start a jupyter lab with this config:

c.NotebookApp.allow_remote_access = False
c.NotebookApp.open_browser = False
c.NotebookApp.password_required = True
c.NotebookApp.port = 9999
c.NotebookApp.port_retries = 0
c.NotebookApp.quit_button = False

I also have:

$ cat /opt/miniconda3/envs/py37/etc/jupyter/jupyter_notebook_config.json
{
  "NotebookApp": {
    "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
  }
}

Symptoms identical to the OP:

$ python -m nb_conda_kernels list
[ListKernelSpecs] [nb_conda_kernels] enabled, 2 kernels found
Available kernels:
  conda-env-dev-py     /opt/miniconda3/envs/dev/share/jupyter/kernels/python3
  python3              /opt/miniconda3/envs/py37/share/jupyter/kernels/python3
  conda-env-py37-py    /opt/miniconda3/envs/py37/share/jupyter/kernels/python3

$ which python
/opt/miniconda3/envs/py37/bin/python

$ jupyter kernelspec list
Available kernels:
  python3    /opt/miniconda3/envs/py37/share/jupyter/kernels/python3

$ which jupyter
/opt/miniconda3/envs/py37/bin/jupyter

Output from verbose install:

$ python -m nb_conda_kernels.install -v --enable
Enabling nb_conda_kernels...
Entry points:
  - conda = nb_conda_kernels.discovery:CondaKernelProvider
  NOTE: entry points not used in Jupyter 5.3.1
jupyter_notebook_config.json entries:
  - /etc/jupyter: <no file>
  - /usr/local/etc/jupyter: <no file>
  * /opt/miniconda3/envs/py37/etc/jupyter: {
      "NotebookApp": {
        "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
      }
    }
  - /home/xsup/.jupyter: <no file>
Status: enabled

fcollonval commented 4 years ago

Add stated in the readme, this extension builds kernel specs dynamically. So by default jupyter kernelspec list will not see your conda environments. But the environments will be available in notebook and JupyterLab. As the first post is not clear on its usage, please describe precisely what you would like to achieve. And what is wrong with screenshots (pictures are worth thousands of words).

joseph-long commented 4 years ago

But the environments will be available in notebook and JupyterLab.

My issue is that this is not happening. Based on the README, "This extension enables a Jupyter Notebook or JupyterLab application in one conda environment to access kernels for Python, R, and other languages found in other environments." This is what I am trying to accomplish.

The problem with a screenshot in this case is that it's difficult to take a screenshot of something that's not there.

fcollonval commented 4 years ago

The problem with a screenshot in this case is that it's difficult to take a screenshot of something that's not there.

This is actually good because now I have no doubt about your use case ;)

So it definitely looks like the kernel spec manager as not been loaded in JupyterLab. Could you run jupyter lab --debug? No need to open JupyterLab itself, it should report something like that:

jupyter lab --debug
[D 15:36:26.787 LabApp] Searching ['/home/fcollonval/.jupyter', '/home/fcollonval/miniconda3/envs/jconda/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 15:36:26.788 LabApp] Looking for jupyter_config in /etc/jupyter
[D 15:36:26.788 LabApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 15:36:26.788 LabApp] Looking for jupyter_config in /home/fcollonval/miniconda3/envs/jconda/etc/jupyter
[D 15:36:26.789 LabApp] Looking for jupyter_notebook_config in /etc/jupyter
[D 15:36:26.789 LabApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 15:36:26.789 LabApp] Looking for jupyter_notebook_config in /home/fcollonval/miniconda3/envs/jconda/etc/jupyter
[D 15:36:26.789 LabApp] Loaded config file: /home/fcollonval/miniconda3/envs/jconda/etc/jupyter/jupyter_notebook_config.json
[D 15:36:26.789 LabApp] Looking for jupyter_notebook_config in /home/fcollonval/.jupyter
[D 15:36:26.790 LabApp] Loaded config file: /home/fcollonval/.jupyter/jupyter_notebook_config.json
[D 15:36:26.794 LabApp] [nb_conda_kernels] refreshing conda info
[I 15:36:27.095 LabApp] [nb_conda_kernels] enabled, 5 kernels found

There are two things that you should see:

The configuration files being loaded; check that it is loading the one in /opt/miniconda3/envs/py37/etc/jupyter in your case.
Log messages from nb_conda_kernels

If you do not see the first one, run jupyter --paths and create a jupyter_notebook_config.json in one of the config folders that contains:

{
  "NotebookApp": {
    "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
  }
}

Could you also report the content of your py37 environment: conda list -n py37?

thvasilo commented 4 years ago

I have a similar issue when trying to execute a particular kernel using papermill.

Some outputs from my side:

$ python -m nb_conda_kernels.install -v --enable
Enabling nb_conda_kernels...
Entry points:
  - conda = nb_conda_kernels.discovery:CondaKernelProvider
  NOTE: entry points not used in Jupyter 6.1.6
jupyter_notebook_config.json entries:
  - /etc/jupyter: <no file>
  - /usr/local/etc/jupyter: <no file>
  * /Users/thvasilo/opt/anaconda3/etc/jupyter: {
      "NotebookApp": {
        "kernel_spec_manager_class": "nb_conda_kernels.CondaKernelSpecManager"
      }
    }
  - /Users/thvasilo/.jupyter: <no file>
Status: enabled

$ which jupyter
/Users/thvasilo/opt/anaconda3/bin/jupyter

$ jupyter --config-dir
/Users/thvasilo/.jupyter

$ cat ~/.jupyter/jupyter_config.json
{
  "CondaKernelSpecManager": {
    "kernelspec_path": "--user"
  }
}

$ jupyter --version
jupyter core     : 4.6.3
jupyter-notebook : 6.1.1
qtconsole        : 4.7.5
ipython          : 7.16.1
ipykernel        : 5.3.4
jupyter client   : 6.1.6
jupyter lab      : 2.1.5
nbconvert        : 5.6.1
ipywidgets       : 7.5.1
nbformat         : 5.0.7
traitlets        : 4.3.3

$ jupyter kernelspec list
Available kernels:
  pycharm-46453bda    /Users/thvasilo/Library/Jupyter/kernels/pycharm-46453bda
  python3             /Users/thvasilo/opt/anaconda3/share/jupyter/kernels/python3

$ python -m nb_conda_kernels list
[ListKernelSpecs] WARNING | Config option `kernelspec_path` not recognized by `CondaKernelSpecManager`.  Did you mean `kernel_spec_class`?
[ListKernelSpecs] [nb_conda_kernels] enabled, 6 kernels found # Some kernels removed for confidentiallity
Available kernels:
  conda-env-base-np-py                                   /users/thvasilo/opt/anaconda3/envs/base-np/share/jupyter/kernels/python3
  conda-env-tables-py                                    /users/thvasilo/opt/anaconda3/envs/tables/share/jupyter/kernels/python3
  conda-root-py                                          /users/thvasilo/opt/anaconda3/share/jupyter/kernels/python3
  pycharm-46453bda                                       /Users/thvasilo/Library/Jupyter/kernels/pycharm-46453bda
  python3                                                /Users/thvasilo/opt/anaconda3/share/jupyter/kernels/python3

Although the kernels don't appear when using jupyter kernelspec list they do appear when running Jupyter notebook:

But when I try to use the kernel using papermill it doesn't work:

 papermill tabulate_test.ipynb output.ipynb -k tables
Input Notebook:  tabulate_test.ipynb
Output Notebook: output.ipynb
Executing:   0%|                                                                                                                                      | 0/4 [00:00<?, ?cell/s]
Traceback (most recent call last):
  File "/Users/thvasilo/opt/anaconda3/bin/papermill", line 11, in <module>
    sys.exit(papermill())
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/papermill/cli.py", line 238, in papermill
    execution_timeout=execution_timeout,
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/papermill/execute.py", line 106, in execute_notebook
    **engine_kwargs
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/papermill/engines.py", line 343, in execute_notebook
    cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/papermill/engines.py", line 402, in execute_managed_notebook
    return PapermillNotebookClient(nb_man, **final_kwargs).execute()
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/papermill/clientwrap.py", line 43, in execute
    with self.setup_kernel(**kwargs):
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/nbclient/client.py", line 438, in setup_kernel
    self.start_new_kernel_client(**kwargs)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/nbclient/client.py", line 393, in async_start_new_kernel_client
    **kwargs))
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/nbclient/util.py", line 85, in ensure_async
    result = await obj
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 569, in start_kernel
    kernel_cmd, kw = self.pre_start_kernel(**kw)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 257, in pre_start_kernel
    kernel_cmd = self.format_kernel_cmd(extra_arguments=extra_arguments)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 181, in format_kernel_cmd
    cmd = self.kernel_spec.argv + extra_arguments
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 87, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/Users/thvasilo/opt/anaconda3/lib/python3.7/site-packages/jupyter_client/kernelspec.py", line 235, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named tables

Using tables or conda-env-tables-py doesn't make a difference. Also note the warning [ListKernelSpecs] WARNING | Config option kernelspec_path not recognized by CondaKernelSpecManager. Did you mean kernel_spec_class?

fcollonval commented 4 years ago

[ListKernelSpecs] WARNING | Config option kernelspec_path not recognized by CondaKernelSpecManager. Did you mean kernel_spec_class?

@thvasilo The option kernelspec_path has been introduced in nb_conda_kernels 2.3.0. From the log above, it looks like you are using a older version. Could you update to the latest version and try again?

mcg1969 commented 8 months ago

closing as stale; but feel free to open if necessary

anaconda / nb_conda_kernels

kernels visible to conda but not to jupyter #185