coder / code-server

VS Code in the browser
https://coder.com
MIT License
68.82k stars 5.65k forks source link

MS Python: Failed to connect to Jupyter notebook. #2875

Closed maresb closed 3 years ago

maresb commented 3 years ago

Discussion migrated from https://github.com/cdr/code-server/issues/2341#issuecomment-796868354.

Under certain situations when running the MS Python extension, the following error can arise:

Failed to connect to Jupyter notebook. http://127.0.0.1:8889/ Error: Kernel Python 3 is not usable. Check the Jupyter output tab for more information.

This problem seems to have began for some users with code-server v3.9.

To view the Jupyter output tab, open the "Command Palette..." and type >Output: Focus on Output View. In the drop-down menu near the top right of the output window, select "Jupyter".


I run into this bug with my particularly complicated setup, so I strongly suspect that others who see this problem will have a different manifestation.

I run code-server in a Docker container which is based on Ubuntu 20.04. Moreover, I don't launch code-server directly, but I run it through JupyterLab via this plugin. It sets up a Jupyter proxy for the VS Code connection.

My suspicion is that something changed how the MS Python extension creates and connects with Jupyter kernels.

Looking at the Jupyter output, I see:

[I 19:41:19.033 NotebookApp] Serving notebooks from local directory: /home/jovyan/work
[I 19:41:19.034 NotebookApp] Jupyter Notebook 6.2.0 is running at:
[I 19:41:19.034 NotebookApp] http://127.0.0.1:8889/?token=...
[I 19:41:19.034 NotebookApp]  or http://127.0.0.1:8889/?token=...
[I 19:41:19.034 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 19:41:19.048 NotebookApp] Forbidden
[W 19:41:19.048 NotebookApp] 403 GET /api/sessions?1615574479041 (127.0.0.1) 2.170000ms referer=None
[W 19:41:19.049 NotebookApp] Forbidden
[W 19:41:19.049 NotebookApp] 403 GET /api/kernelspecs?1615574479041 (127.0.0.1) 2.120000ms referer=None
[W 19:41:19.049 NotebookApp] 403 POST /api/contents/?1615574479042 (127.0.0.1): '_xsrf' argument missing from POST
[W 19:41:19.049 NotebookApp] '_xsrf' argument missing from POST
[W 19:41:19.050 NotebookApp] 403 POST /api/contents/?1615574479042 (127.0.0.1) 2.210000ms referer=None
[W 19:41:19.054 NotebookApp] Forbidden
[W 19:41:19.054 NotebookApp] 403 GET /api/sessions?1615574479049 (127.0.0.1) 1.500000ms referer=None
[W 19:41:19.055 NotebookApp] Forbidden
[W 19:41:19.055 NotebookApp] 403 GET /api/kernelspecs?1615574479049 (127.0.0.1) 1.810000ms referer=None
[W 19:41:19.056 NotebookApp] 403 POST /api/contents/?1615574479052 (127.0.0.1): '_xsrf' argument missing from POST
[W 19:41:19.056 NotebookApp] '_xsrf' argument missing from POST
[W 19:41:19.056 NotebookApp] 403 POST /api/contents/?1615574479052 (127.0.0.1) 0.810000ms referer=None

It seems that MS Python is trying to connect to my existing JupyterLab server, but it doesn't know the correct token and is thus getting rejected.

I managed to fix it by adding the setting "python.dataScience.jupyterServerURI": "http://localhost:8888/?token=THEVALUEOFMYTOKEN", to my settings.json, or equivalently by changing the setting "Data Science: Jupyter Server URI" from local to http://localhost:8888/?token=THEVALUEOFMYTOKEN.

maresb commented 3 years ago

@phisanti, my solution is very particular to the complicated configuration I'm running. The first step is to look at the output via the method I explained above.

In case you want to run your own Jupyter server, you can run jupyter lab from the command line to start JupyterLab. After JupyterLab starts, at the bottom of the terminal where you started JupyterLab you should see:

    To access the server, open this file in a browser:
        file:///home/jovyan/.local/share/jupyter/runtime/jpserver-32688-open.html
    Or copy and paste one of these URLs:
        http://13ca0fa98ac2:8889/lab?token=7ed6ea0f36651733bc695a24eae12700938f44556978b9aa
     or http://127.0.0.1:8889/lab?token=7ed6ea0f36651733bc695a24eae12700938f44556978b9aa

You can take this URL http://127.0.0.1:8889/lab?token=7ed6ea0f36651733bc695a24eae12700938f44556978b9aa and put it into your configuration. Then the MS Python extension will (hopefully) be able to connect to your Jupyter server.

jsjoeio commented 3 years ago

Thanks for opening up a separate issue @maresb! In order for us to look into this further, do you mind updating the issue with the following:

Thank you!

maresb commented 3 years ago

@jsjoeio thanks for being so eager to debug this issue! I have not been so forthcoming with details due to the complicated nature of my setup. (See the second paragraph which I added below the horizontal rule.) I hope this explains why I don't explain how to reproduce, since that would involve setting up the Jupyter server proxy.

I recommend waiting for @phisanti to arrive and debugging his particular problem, since I suspect that his setup is more straightforward than mine. (Moreover, I have a solution for my particular case, so I opened this issue mainly for the benefit of him and those who also have this issue.)

My suspicion is that prior to code 3.9, MS Python was somehow proxying its own Jupyter server, but that from 3.9 the connection is no longer being proxied. But I don't really know so many details.

phisanti commented 3 years ago

I prepared the image like you did use docker. Then, build the machine in a remote machine on paperspace.com notebook, which uses port 8888 to push the vscode server to the browser. So you can install the image from https://hub.docker.com/repository/docker/phisanti/gradient-coder. The entry point is /run.sh. Next, I just download and install 'ms-python.python-2020.10.332292344':

curl -JLO https://open-vsx.org/api/ms-python/python/2020.10.332292344/file/ms-python.python-2020.10.332292344.vsix
code-server --install-extension ms-python.python-2020.10.332292344'.vsix

Next, I will install the required python dependencies as I am asked. My final lib looks like:

# packages in environment at /home/coder/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
argon2-cffi               20.1.0           py38h27cfd23_1  
astroid                   2.5              py38h06a4308_1  
async_generator           1.10               pyhd3eb1b0_0  
attrs                     20.3.0             pyhd3eb1b0_0  
backcall                  0.2.0              pyhd3eb1b0_0  
beautifulsoup4            4.9.3              pyha847dfd_0  
bleach                    3.3.0              pyhd3eb1b0_0  
brotlipy                  0.7.0           py38h27cfd23_1003  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2021.1.19            h06a4308_1  
certifi                   2020.12.5        py38h06a4308_0  
cffi                      1.14.3           py38h261ae71_2  
chardet                   3.0.4           py38h06a4308_1003  
conda                     4.9.2            py38h06a4308_0  
conda-build               3.21.4           py38h06a4308_0  
conda-package-handling    1.7.2            py38h03888b9_0  
cryptography              3.2.1            py38h3c74f83_1  
dbus                      1.13.18              hb2f20db_0  
decorator                 4.4.2              pyhd3eb1b0_0  
defusedxml                0.7.1              pyhd3eb1b0_0  
entrypoints               0.3                      py38_0  
expat                     2.2.10               he6710b0_2  
filelock                  3.0.12             pyhd3eb1b0_1  
fontconfig                2.13.1               h6c09931_0  
freetype                  2.10.4               h5ab3b9f_0  
glib                      2.67.4               h36276a3_1  
glob2                     0.7                pyhd3eb1b0_0  
gst-plugins-base          1.14.0               h8213a91_2  
gstreamer                 1.14.0               h28cd5cc_2  
icu                       58.2                 he6710b0_3  
idna                      2.10                       py_0  
importlib-metadata        2.0.0                      py_1  
importlib_metadata        2.0.0                         1  
ipykernel                 5.3.4            py38h5ca1d4c_0  
ipython                   7.21.0           py38hb070fc8_0  
ipython_genutils          0.2.0              pyhd3eb1b0_1  
ipywidgets                7.6.3              pyhd3eb1b0_1  
isort                     5.7.0              pyhd3eb1b0_0  
jedi                      0.17.0                   py38_0  
jinja2                    2.11.3             pyhd3eb1b0_0  
jpeg                      9b                   h024ee3a_2  
json5                     0.9.5                      py_0  
jsonschema                3.2.0                      py_2  
jupyter                   1.0.0                    py38_7  
jupyter_client            6.1.7                      py_0  
jupyter_console           6.2.0                      py_0  
jupyter_core              4.7.1            py38h06a4308_0  
jupyterlab                2.2.6                      py_0  
jupyterlab_pygments       0.1.2                      py_0  
jupyterlab_server         1.2.0                      py_0  
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1  
lazy-object-proxy         1.5.2            py38h27cfd23_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libarchive                3.4.2                h62408e4_0  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
liblief                   0.10.1               he6710b0_0  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.18               h7b6447c_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.14                 h7b6447c_0  
libxml2                   2.9.10               hb55368b_3  
lz4-c                     1.9.3                h2531618_0  
markupsafe                1.1.1            py38h7b6447c_0  
mccabe                    0.6.1                    py38_1  
mistune                   0.8.4           py38h7b6447c_1000  
nbclient                  0.5.3              pyhd3eb1b0_0  
nbconvert                 6.0.7                    py38_0  
nbformat                  5.1.2              pyhd3eb1b0_1  
ncurses                   6.2                  he6710b0_1  
nest-asyncio              1.5.1              pyhd3eb1b0_0  
notebook                  6.2.0            py38h06a4308_0  
openssl                   1.1.1j               h27cfd23_0  
packaging                 20.9               pyhd3eb1b0_0  
pandoc                    2.11                 hb0f4dca_0  
pandocfilters             1.4.3            py38h06a4308_1  
parso                     0.8.1              pyhd3eb1b0_0  
patchelf                  0.12                 h2531618_1  
pcre                      8.44                 he6710b0_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pip                       20.2.4           py38h06a4308_0  
pkginfo                   1.7.0            py38h06a4308_0  
prometheus_client         0.9.0              pyhd3eb1b0_0  
prompt-toolkit            3.0.8                      py_0  
prompt_toolkit            3.0.8                         0  
psutil                    5.8.0            py38h27cfd23_1  
ptyprocess                0.7.0              pyhd3eb1b0_2  
py-lief                   0.10.1           py38h403a769_0  
pycosat                   0.6.3            py38h7b6447c_1  
pycparser                 2.20                       py_2  
pygments                  2.8.1              pyhd3eb1b0_0  
pylint                    2.7.2            py38h06a4308_1  
pyopenssl                 19.1.0             pyhd3eb1b0_1  
pyparsing                 2.4.7              pyhd3eb1b0_0  
pyqt                      5.9.2            py38h05f1152_4  
pyrsistent                0.17.3           py38h7b6447c_0  
pysocks                   1.7.1            py38h06a4308_0  
python                    3.8.5                h7579374_1  
python-dateutil           2.8.1              pyhd3eb1b0_0  
python-libarchive-c       2.9                pyhd3eb1b0_0  
pytz                      2021.1             pyhd3eb1b0_0  
pyyaml                    5.4.1            py38h27cfd23_1  
pyzmq                     20.0.0           py38h2531618_1  
qt                        5.9.7                h5867ecd_1  
qtconsole                 5.0.2              pyhd3eb1b0_0  
qtpy                      1.9.0                      py_0  
readline                  8.0                  h7b6447c_0  
requests                  2.24.0                     py_0  
ripgrep                   12.1.1                        0  
ruamel_yaml               0.15.87          py38h7b6447c_1  
send2trash                1.5.0              pyhd3eb1b0_1  
setuptools                50.3.1           py38h06a4308_1  
sip                       4.19.13          py38he6710b0_0  
six                       1.15.0           py38h06a4308_0  
soupsieve                 2.2                pyhd3eb1b0_0  
sqlite                    3.33.0               h62c20be_0  
terminado                 0.9.2            py38h06a4308_0  
testpath                  0.4.4              pyhd3eb1b0_0  
tk                        8.6.10               hbc83047_0  
toml                      0.10.1                     py_0  
tornado                   6.1              py38h27cfd23_0  
tqdm                      4.51.0             pyhd3eb1b0_0  
traitlets                 5.0.5              pyhd3eb1b0_0  
urllib3                   1.25.11                    py_0  
wcwidth                   0.2.5                      py_0  
webencodings              0.5.1                    py38_1  
wheel                     0.35.1             pyhd3eb1b0_0  
widgetsnbextension        3.5.1                    py38_0  
wrapt                     1.12.1           py38h7b6447c_1  
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
zeromq                    4.3.3                he6710b0_3  
zipp                      3.4.0              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.5                h9ceee32_0  

Finally, I try to create a jupyter notebook, but I got the following error:

Failed to connect to Jupyter notebook. http://127.0.0.1:8889/ Error: Kernel Python 3 is not usable. Check the Jupyter output tab for more information.

If I follow the advice of @maresb and I run jupyter lab from the terminal, then the following the response is:


[I 22:10:09.745 LabApp] The port 8888 is already in use, trying another port.
[I 22:10:09.755 LabApp] JupyterLab extension loaded from /home/coder/miniconda3/lib/python3.8/site-packages/jupyterlab
[I 22:10:09.756 LabApp] JupyterLab application directory is /home/coder/miniconda3/share/jupyter/lab
[C 22:10:09.760 LabApp] Running as root is not recommended. Use --allow-root to bypass.

If I click on the palette command the option to create a new jupyter notebook with the token of my notebook, then the error looks like:

Failed to connect to remote Jupyter notebook. Check that the Jupyter Server URI setting has a valid running server specified. http://127.0.0.1:8889/token=11cf752079724762b4622643143ddb4a FetchError: request to http://127.0.0.1:8889/token=11cf752079724762b4622643143ddb4a/hub/api failed, reason: connect ECONNREFUSED 127.0.0.1:8889

If I use the token you posted, this is the error:

Failed to connect to remote Jupyter notebook. Check that the Jupyter Server URI setting has a valid running server specified. http://127.0.0.1:8889/lab Error: Kernel Python 3 is not usable. Check the Jupyter output tab for more information.


running on:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

On a final note, when I install ms-python.python-2020.3.69010, the problem seem to be gone. However, I cannot use widgets (I need nglview to display protein structures). So it seems to be an issue of that particular version with code-server or vscode.

jsjoeio commented 3 years ago

Thanks for the details @phisanti! Sounds like yours might be complex as well. Any chance you can edit your comment and outline it in a way that anyone could jump in and follow? i.e.

  1. do this
  2. do that

The more minimal the steps to reproduce, the faster we'll be able to look into this! If you need inspiration, this article has good tips.

maresb commented 3 years ago

@phisanti Does it work if you publish port 8889 from Docker?

phisanti commented 3 years ago

@maresb I have edited the entrypoint. Now it looks like:


#!/bin/bash

# Source: https://gist.github.com/earthgecko/3089509
# Generates a random alphanumeric string of length 48 (like a jupyter notebook token i.e. c8de56fa4deed24899803e93c227592aef6538f93025fe01)
if [ -z "$JUPYTER_TOKEN" ]; then
    JUPYTER_TOKEN=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 48 | head -n 1)
fi

# Note: print mocked jupyter token so that we can run this container as if it is a notebook within Gradient V1
echo "http://localhost:8889/?token=${JUPYTER_TOKEN}"
echo "http://localhost:8889/\?token\=${JUPYTER_TOKEN}"

PASSWORD=${JUPYTER_TOKEN} /usr/bin/code-server --bind-addr "0.0.0.0:8889" .

However, I cannot access the session with this configuration.

maresb commented 3 years ago

@phisanti What does your docker command look like? Did you try adding -p 8889:8889?

I don't quite understand what is the goal of editing the entrypoint. Are you trying to change the code-server port to 8889?

What I meant with my last suggestion is that you should keep code-server on 8888, but also publish port 8889 so that the browser running code-server can access the Jupyter server as well.

So I recommend undoing the change to your entrypoint. In your docker command you probably have something like -p 8888:8888 and my suggestion would be to change this to -p 8888:8888 -p 8889:8889.

phisanti commented 3 years ago

I might look like a fool because I am new to dockerand all programming related. My major is in biology and I need to run some molecular dynamics simulations.

This is how my docker file looks now:

# Import image with code-server and CUDA
FROM phisanti/cuda-code-server:latest

# Install conda
RUN curl -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
    chmod +x ~/miniconda.sh && \
    ~/miniconda.sh -b && \
    rm ~/miniconda.sh && \
    /home/coder/miniconda3/bin/conda install conda-build

ENV PATH=$PATH:/home/coder/miniconda3/bin/

# Entrypoint
COPY run.sh /run.sh
EXPOSE 8888
EXPOSE 8889
ENTRYPOINT ["/run.sh"]

You can see with all detail here. I tried to expose the port with the docker command EXPOSEas I did with 8888 (taken from the template). However, from your comment, I see the -p flag, so I deduce it should run as RUN code-server -p 8888:8888 -p 8889:8889, but I am not fully sure. Should I say that using the current image, I still get the same error Failed to connect to Jupyter notebook. http://127.0.0.1:8889/ Error: Kernel Python 3 is not usable.. I notice that it also says click on OUTPUT for more detail but nothing comes there. Is there an option to see such an output?

Sorry if it is a bad time, I have been struggling with this for at least a month.

maresb commented 3 years ago

Hi @phisanti, my apologies if I came across as condescending. You should feel welcome regardless of your background or technical expertise, so I'm sorry if I failed there. It's great that you're learning Docker, and of course I was completely lost myself as I was a beginner.

I think I know more or less what the problem is, but I was missing some info I which I need to be able to help.

What I am missing is your docker run command (or your docker-compose.yml in case you are starting the container with docker-compose). The -p is not for code-server but rather docker run. So the command should look like:

docker run -p 8888:8888 -p 8889:8889 …

Could you send me the full docker run command you're using?

By the way, the EXPOSE command in Dockerfile is effectively documentation; it doesn't actually do anything. According to the documentation

The EXPOSE instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published. To actually publish the port when running the container, use the -p flag on docker run to publish and map one or more ports, or the -P flag to publish all exposed ports and map them to high-order ports.

Personally I find this weird, confusing and unexpected on Docker's part. And as the documentation states, we need to figure out what's going on with the -p flags in your docker run command.

To explain where I'm going with this, it looks to me like a Jupyter server is being started inside of your Docker container. Then code-server tries to connect to the Jupyter server from your browser. However, the Jupyter server is not running on your native OS, but inside your Docker container. Your browser is looking on your native OS and not finding anything, and thus giving an error.

My suspicion is that if you publish port 8889 (the port where your Jupyter server is starting) from Docker to your native OS, then your browser will be able to find the Jupyter server, and the error will go away.

Does this make sense? Let me know if anything I wrote was confusing.

phisanti commented 3 years ago

So, you say that I need to run a command in the terminal like this before publish/build the image:

docker run -p 8888:8888 -p 8889:8889 …

However, this is probably the result of me building the docker image based on youtube tutorials and cut-and-paste templates. I remember I tried to download the docker software in my computer and then build an image, but I end up nowhere. Then, I saw that it was possible to create an image only through writing something called a DOCKERFILE. Thus, I copied the code-server docker image that paperspace uploaded to their github and made some changes (add CUDA and conda), which resulted in the code I sent previously. So, I discovered that I only need to link my docker account and, voilá, the image appeared in dockerhub. Since I am not using the terminal, then I guess you suggest to add a file called docker-compose.yml to the github repository? If that is correct, then, I can do it right now. I will google how to organise the file and then add that line on the right spot.

Once again, thank you @maresb for going the extra mile, I really feel you are trying to help me.

maresb commented 3 years ago

before publish/build the image

No, after.

The Docker build process generates from the Dockerfile an image, which is effectively like a copy of a hard drive which will be the basis of each container when it is run. After building, you can publish the image if you want to share it with others, or you can keep it in your local Docker cache.

In either case, once you have an image, you can start one or more containers using it. Starting a container is like starting a virtual machine. The container must run on some host on which it can use the CPU, memory, and network resources.

What is the host of your container? And how is your container starting if you are not using a docker run command or similar?

phisanti commented 3 years ago

I am running the image from paperspace gradient notebooks. I want to work with them because they allow me to use GPU, which I don't have in my local device, and its free. They have some pre-made images to start what they call notebooks. This is how it looks: image (all that grey text in the bexes is sample text)

So, I made/publish my own image. In order to run my own image, with my custom notebook I fill the following details: image

Then, after a couple of minutes, there is a web address that I can access with the code-server running. For example: https://n0ipdpto.clg07azjl.paperspacegradient.com/?token=ad434ce2f9a6b1a30ae7e2e9b9bb3d7d And it looks like: image

maresb commented 3 years ago

Ahhhhh, ok, now everything makes a lot more sense. I am not familiar with paperspace and I did not understand that you were using them as a host.

As for configuring ports through paperspace, it looks like this menu is where you would set up the port as I have been describing. Then you would need to set "Data Science: Jupyter Server URI" to https://n0ipdpto.clg07azjl.paperspacegradient.com:8889/?token=JUPYTER_SERVER_TOKEN. However, I don't know how to get the token from the Jupyter Server instance started by VS Code.

As an alternative to using VS Code's Jupyter server, I would recommend starting your own:

(For your use case, you might be better off just running the Jupyter server on paperspace and connecting to it as a remote Jupyter server using a plain VS Code installation. But let's take things one step at a time, and try the previous suggestion first.)

phisanti commented 3 years ago

This surely is the most advance thing I have ever done with a computer. I tried to run the port as you said and then running the jupyter lab with the specified port. I got the URL, tho the token is just ... but it seemed to work as I could open a new notebook? from that link. Then I put that in the settings file aaand.... nothing really changed. The kernels remain unusable.

Once again, thank you very much for all your help, but I think it is time to surrender and admit this problem is bigger than I. I guess I will have to wait until the guys at open-svx release the new version and hope that would solve the problem. Really appreciate your help man, your dedication makes this a better world.

maresb commented 3 years ago

Well it's a very good sign that you're able to connect to JupyterLab from your browser.

I think I made a mistake in my instructions. Before you give up, you might want to make one last attempt and replace the host by localhost in the "Data Science: Jupyter Server URI" setting. In particular, the setting should look like https://localhost:8880/?token=319f987ca2bf35046368734e61ba362baa081b0dcdddc17e (but with a different token).

maresb commented 3 years ago

I suppose we can reopen this if someone else has this problem, but from my perspective this is solved.

phisanti commented 3 years ago

You are right. All that could be done was done, so I will wait for the open-vsx update. Thank you very much for your help, great to talk to you.

jsjoeio commented 3 years ago

@maresb thanks for going the extra mile helping out here! ♥️ We super appreciate it.

And sorry for all the trouble/workarounds you have to do @phisanti - hoping to get the Open VSX update out in the next month or two!

gogobd commented 3 years ago

I'm having issues with jupyter too:

OS/Web Information

Steps to Reproduce

  1. Click a .ipynb File

Expected

Jupyter Notebook should be functional

Actual

Jupyter Notebook displays but no cell can run

Screenshot 2021-04-23 at 20 50 21

Notes

Previously I was able to fix this by following https://github.com/cdr/code-server/issues/2341 but that doesn't help any more.

maresb commented 3 years ago

I'd love to help, but that seems to me like some sort of javascript problem. With that I'm clueless.