search

jupyter-server / jupyter_server

The backend—i.e. core services, APIs, and REST endpoints—to Jupyter web applications.
https://jupyter-server.readthedocs.io
BSD 3-Clause "New" or "Revised" License
465 stars 279 forks source link

Pass session ID during Websocket Upgrade connection when using Gateway. #1439

Open gogasca opened 3 days ago

gogasca commented 3 days ago

Description

Connecting JupyterLab 4.2.2 or Vscode (Jupyter extension) to Google Colab Enterprise fails.

Scenario:

In Colab Enterprise we have a internal proxy which expects session_id Currently JupyterLab passes:

GET /api/kernels/342eba8f-aa8c-47a9-8fee-8541cee8ddfe/channels

vs

GET /api/kernels/342eba8f-aa8c-47a9-8fee-8541cee8ddfe/channels?session_id=4feb3f46-4e83-4fba-bebd-3aba95e96fc0

When I create a new Kernel from JupyterLab I can see the session_id HTTP query param, but when JupyterLab Gateway is building this same request it does not pass it to remote side here hence in Colab Enterprise I see this error.

2024/06/17 04:45:11 Websocket failure: failed to read a websocket message from the server: websocket: close 1006 (abnormal closure): unexpected EOF 
2024-06-16 21:45:11.616 PDT
E0617 04:45:11.404645     154 ipython_mitm.go:1398] handleKernels5: websocket request missing required parameter session_id "/api/kernels/73f2bc65-bb6a-444c-a728-b2252ec9ed37/channels" 
2024-06-16 21:45:11.632 PDT

JupyterLab to Jupyter Server session_id is correctly generated but not passed to c.GatewayClient.url here

GET /api/kernels/342eba8f-aa8c-47a9-8fee-8541cee8ddfe/channels?session_id=4feb3f46-4e83-4fba-bebd-3aba95e96fc0 HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Sec-WebSocket-Version: 13
Origin: http://localhost:8080
Sec-WebSocket-Protocol: v1.kernel.websocket.jupyter.org
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: 5E7hLeW9SV/Dck4S4jgEAQ==
DNT: 1
Connection: keep-alive, Upgrade
Cookie: username-localhost-8080=2|1:0|10:1719307104|23:username-localhost-8080|188:eyJ1c2VybmFtZSI6ICJmNjdlMDQ1YmZkMmE0MDllOTU4MzMzZThlY2NlNTcwNyIsICJuYW1lIjogIkFub255bW91cyBLb3JlIiwgImRpc3BsYXlfbmFtZSI6ICJBbm9ueW1vdXMgS29yZSIsICJpbml0aWFscyI6ICJBSyIsICJjb2xvciI6IG51bGx9|806e6a65fe8c8092fc40bef8b65ed0edeaec2f440850c61df7d63742f2de96f7; _xsrf=2|8cd09498|e00723a11572d27ce0a1f55b25dc0735|1719307104
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

Reproduce

import datetime
import subprocess
import hashlib
import json

PROXY_URL = "https://2itk4z2isb2va-dot-datalab-vm-staging.googleusercontent.com"

c.NotebookApp.open_browser = False
c.ServerApp.token = ""
c.ServerApp.password = ""
c.ServerApp.port = 8080
c.ServerApp.allow_remote_access = True
c.ServerApp.disable_check_xsrf = True

def get_gcloud_token():
  """Helper method to get an OAuth token from gcloud."""
  p = subprocess.run(
      ["gcloud", "auth", "print-access-token"],
      capture_output=True,
      check=True,
      encoding="UTF-8",
  )
  bearer_token = p.stdout.strip()
  return bearer_token

c.GatewayClient.auth_scheme = "Bearer"
c.GatewayClient.auth_token = get_gcloud_token()
c.GatewayClient.headers = json.dumps({"Origin": PROXY_URL})
c.GatewayClient.validate_cert = False
c.GatewayClient.url = PROXY_URL
c.Application.log_level = 0

Started JupyterLab locally:

jupyter lab --port-retries=0 --ip 0.0.0.0 --allow-root --config=jupyter_notebook_config.py --debug

Open JupyterLab locally and try to connect to remote server (Colab Enterprise), error is seen on JupyterLab and Colab Enterprise logs.

[I 2024-06-16 21:56:34.143 ServerApp] Request start kernel: kernel_id=None, path=''
[D 2024-06-16 21:56:34.145 ServerApp] Request new kernel at: https://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels
[I 2024-06-16 21:56:34.243 ServerApp] GatewayKernelManager started kernel: 48cf0082-b7c8-4f43-b13c-bffa39db467b, args: {'kernel_id': None, 'kernel_name': 'python3', 'env': {'TERM_PROGRAM': 'Apple_Terminal', 'SHELL': '/bin/zsh', 'TERM': 'xterm-256color', 'TMPDIR': '/var/folders/n_/vtwpxlys2cngxlzl4v35747000_wjb/T/', 'TERM_PROGRAM_VERSION': '453', 'TERM_SESSION_ID': 'CB2A71D6-9B3E-4201-8BB8-3A53D8757F89', 'USER': 'gogasca', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.3jQ3nSOvmk/Listeners', 'PATH': '/Users/gogasca/opt/anaconda3/bin:/Users/gogasca/opt/anaconda3/condabin:/Users/gogasca/Downloads/google-cloud-sdk/bin:/usr/local/git/git-google/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Applications/Wireshark.app/Contents/MacOS:/usr/local/go/bin', 'LaunchInstanceID': 'CB01220E-F880-4642-BEA1-3C64FC7484E3', '__CFBundleIdentifier': 'com.apple.Terminal', 'PWD': '/Users/gogasca/Downloads/vscode', 'XPC_FLAGS': '0x0', 'XPC_SERVICE_NAME': '0', 'SHLVL': '1', 'HOME': '/Users/gogasca', 'LOGNAME': 'gogasca', 'SECURITYSESSIONID': '186a3', 'OLDPWD': '/Users/gogasca/Downloads/vscode', 'SK_SIGNING_PLUGIN': 'gnubbyagent', 'PROMPT': '${COLOR_DIR}${PWD} ${COLOR_GIT}$(parse_git_branch)${COLOR_DEF}% > ', 'CLICOLOR': '1', 'LSCOLORS': 'ExGxBxDxCxEgEdxbxgxcxd', 'CONDA_EXE': '/Users/gogasca/opt/anaconda3/bin/conda', '_CE_M': '', '_CE_CONDA': '', 'CONDA_PYTHON_EXE': '/Users/gogasca/opt/anaconda3/bin/python', 'CONDA_SHLVL': '1', 'CONDA_PREFIX': '/Users/gogasca/opt/anaconda3', 'CONDA_DEFAULT_ENV': 'base', 'CONDA_PROMPT_MODIFIER': '(base) ', 'LANG': 'en_US.UTF-8', '_': '/Users/gogasca/opt/anaconda3/bin/jupyter', '__CF_USER_TEXT_ENCODING': '0x5722B:0x0:0x0', 'KERNEL_LAUNCH_TIMEOUT': '40', 'JPY_SESSION_NAME': '/Users/gogasca/Downloads/vscode/BigQuery basics-88b61f81-d47b-437f-b598-8becabba33f2.ipynb'}, 'cwd': '/Users/gogasca/Downloads/vscode'}
[D 2024-06-16 21:56:34.253 ServerApp] 201 POST /api/sessions?1718600194138 (be9e028811694a53aecd99ba777a6721@127.0.0.1) 111.75ms
[D 2024-06-16 21:56:34.255 ServerApp] Generating new user for token-authenticated request: 3e9e4575787a42c1b817daf8bb14ac42
[D 2024-06-16 21:56:34.256 ServerApp] 200 GET /api/sessions?1718600194253 (3e9e4575787a42c1b817daf8bb14ac42@127.0.0.1) 1.23ms
[D 2024-06-16 21:56:34.263 ServerApp] Generating new user for token-authenticated request: a12888ed50b84bf480ada03042447a13
[D 2024-06-16 21:56:34.263 ServerApp] Request list kernels: https://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels
[D 2024-06-16 21:56:34.272 ServerApp] Generating new user for token-authenticated request: 79c2aed07aef4bc4b98aca2405415a8f
[D 2024-06-16 21:56:34.275 ServerApp] 101 GET /api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels?session_id=598b4dab-b7c1-403b-aea9-c96f382b268b (79c2aed07aef4bc4b98aca2405415a8f@127.0.0.1) 3.70ms
[D 2024-06-16 21:56:34.275 ServerApp] Opening websocket /api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
[I 2024-06-16 21:56:34.275 ServerApp] Connecting to kernel 48cf0082-b7c8-4f43-b13c-bffa39db467b.
[I 2024-06-16 21:56:34.275 ServerApp] Connecting to wss://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
[D 2024-06-16 21:56:34.377 ServerApp] Connection is ready: ws: <tornado.websocket.WebSocketClientConnection object at 0x7fe27f85b8b0>
[D 2024-06-16 21:56:34.384 ServerApp] 200 GET /api/kernels?1718600194261 (a12888ed50b84bf480ada03042447a13@127.0.0.1) 121.80ms
[W 2024-06-16 21:56:34.430 ServerApp] Lost connection to Gateway: 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:34.430 ServerApp] Attempting to re-establish the connection to Gateway in 1.71 secs (1/5): 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:36.140 ServerApp] Connecting to wss://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
[D 2024-06-16 21:56:36.262 ServerApp] Connection is ready: ws: <tornado.websocket.WebSocketClientConnection object at 0x7fe27f84ba30>
[W 2024-06-16 21:56:36.323 ServerApp] Lost connection to Gateway: 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:36.323 ServerApp] Attempting to re-establish the connection to Gateway in 1.9 secs (1/5): 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:38.224 ServerApp] Connecting to wss://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels

Expected behavior

Pass session_id during channels request. I'm able to solve this issue by passing session_id and modifying CORS config in Colab Enterprise side. This issue is opened to track Jupyter Server side.

Context

This is used to allow VScode connectivity via JupyterLab Gateway feature. https://medium.com/@gogasca_/how-to-connect-from-vscode-to-vertex-notebooks-directly-8c078d538b0b

Troubleshoot Output 
Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.
Command Line Output 
Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.
Browser Output 
Paste the output from your browser Javascript console here, if applicable.