search

jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
615 stars 220 forks source link

kernel Status: Unknown in Jupyterhub 3.2.1 while connecting remote kernel vai Jupyter Enterprise Gateway 3.2.1 on K8s 1.25 AWS EKS #1360

Open roshan3133 opened 5 months ago

roshan3133 commented 5 months ago

Description

kernel Status: Unknown in Jupyterhub 3.2.1 while connecting remote kernel vai Jupyter Enterprise Gateway 3.2.1

Reproduce

  1. helm upgrade --install enterprise-gateway https://github.com/jupyter-server/enterprise_gateway/releases/download/v3.2.1/jupyter_enterprise_gateway_helm-3.2.1.tar.gz --kube-context --namespace enterprise-gateway --values ./enterprise-gateway/values.yml
  2. helm upgrade --install jupyterhub jupyterhub/jupyterhub --values ./jupyterhub/values.yml -n jupyterhub

Expected behavior

It should execute jupyter notebook to respected kernel and execute commands in notebook cell

Context

extraEnv: @ ./jupyterhub/values.yml

EDITOR: "vim"
JUPYTER_GATEWAY_URL: "http://enterprise-gateway.enterprise-gateway.svc.cluster.local:8888"
LOG_LEVEL: "DEBUG"   
JUPYTER_GATEWAY_CONNECT_TIMEOUT: "1200"
JUPYTER_GATEWAY_REQUEST_TIMEOUT: "1200"
KERNEL_USERNAME: "jovyan"
KERNEL_SERVICE_ACCOUNT_NAME: "enterprise-gateway-sa"
KERNEL_NAMESPACE: "enterprise-gateway"

extraEnv: { @ ./enterprise-gateway/values.yml

EG_INHERITED_ENVS: "PATH",
EG_KUBERNETES_LOG_LEVEL: "WARNING",
EG_SHARED_NAMESPACE : "enterprise-gateway",
EG_AUTHORIZED_USERS: "jovyan",

}

forum link

for the same - https://discourse.jupyter.org/t/failed-to-connect-jupyter-enterprise-gateway-from-jupyterhub/23319

List of all available remote kernel vai JEG

ss2

Python kernel status unknown

ss3

@ JEG remote python kernel pods logs from gateway-enterprises namespace.

/usr/local/bin/bootstrap-kernel.sh env: SHELL=/bin/bash KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 ENTERPRISE_GATEWAY_PORT_8877_TCP=tcp://172.20.93.178:8877 KERNEL_NAME=python_kubernetes ENTERPRISE_GATEWAY_SERVICE_PORT_HTTP=8888 EG_IMPERSONATION_ENABLED=False HOSTNAME=jovyan-95aae2ac-27c3-4912-a326-cedfbc52d302 LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none KERNEL_ID=95aae2ac-27c3-4912-a326-cedfbc52d302 NB_UID=1000 ENTERPRISE_GATEWAY_SERVICE_HOST=172.20.93.178 KERNEL_UID=1000 KERNEL_IMAGE=elyra/kernel-py:3.2.1 KERNEL_LAUNCH_TIMEOUT=1198 PWD=/home/jovyan ENTERPRISE_GATEWAY_PORT_8877_TCP_PROTO=tcp RESPONSE_ADDRESS=10.1.21.239:8877 ENTERPRISE_GATEWAY_SERVICE_PORT_HTTP_RESPONSE=8877 KERNEL_SERVICE_ACCOUNT_NAME=enterprise-gateway-sa ENTERPRISE_GATEWAY_PORT_8888_TCP_PORT=8888 ENTERPRISE_GATEWAY_PORT_8888_TCP_ADDR=172.20.93.178 HOME=/home/jovyan LANG=en_US.UTF-8 KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443 ENTERPRISE_GATEWAY_PORT_8877_TCP_PORT=8877 NB_GID=100 ENTERPRISE_GATEWAY_PORT_8888_TCP=tcp://172.20.93.178:8888 EG_MIN_PORT_RANGE_SIZE=1000 XDG_CACHE_HOME=/home/jovyan/.cache/ JPY_PARENT_PID=7 ENTERPRISE_GATEWAY_PORT_8877_TCP_ADDR=172.20.93.178 KERNEL_POD_NAME=jovyan-95aae2ac-27c3-4912-a326-cedfbc52d302 SHLVL=0 CONDA_DIR=/opt/conda KERNEL_GATEWAY=1 ENTERPRISE_GATEWAY_SERVICE_PORT=8888 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1 PORT_RANGE=0..0 KERNEL_USERNAME=jovyan LC_CTYPE=C.UTF-8 ENTERPRISE_GATEWAY_PORT=tcp://172.20.93.178:8888 KERNEL_LANGUAGE=python NB_USER=jovyan KUBERNETES_SERVICE_HOST=172.20.0.1 ENTERPRISE_GATEWAY_PORT_8888_TCP_PROTO=tcp LC_ALL=en_US.UTF-8 KUBERNETES_PORT=tcp://172.20.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin KERNEL_EXECUTOR_IMAGE=elyra/kernel-py:3.2.1 KERNEL_GID=100 PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCpiPIxwaSykdaPMrJ7VB2QWFGOqNJfXWrsEFkJbSTDHibHK67xo9fGeH50dlyHChgbQpR9Ry+Qjj70itw67G8pHVnYXKK9OEvMcxyChu4bATYz2Hh4 ky1y0x+nx9+ec9DqV5M45WD5cJJExACfieI0SMvNCJiJ8fTDmfzsQKEKWwIDAQAB DEBIAN_FRONTEND=noninteractive EG_MAX_PORT_RANGE_RETRIES=5 KERNEL_NAMESPACE=enterprise-gateway _=/usr/bin/env 2024-01-16T07:27:59.172889718Z + python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 95aae2ac-27c3-4912-a326-cedfbc52d302 --port-range 0..0 --response-address 10.1.21.239:8877 --public-key MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCpiPIxwaSykdaPMrJ7VB2QWFGOqNJfXWrsEFkJbSTDHibHK67xo9fGeH50dlyHChgbQpR9Ry+Qjj70itw67G8pHVnYXKK9OEvMcxyChu4bATYz2Hh4ky1y0x+nx9+ec9DqV5M45WD5cJJExACfieI0SMvNCJiJ8fTDmfzsQKEKWwIDAQAB --spark-context-initialization-mode none 2024-01-16T07:27:59.457362107Z [D 2024-01-16 07:27:59,457.457 launch_ipykernel] Using connection file '/tmp/kernel-95aae2ac-27c3-4912-a326-cedfbc52d302_p8654iyt.json'. [I 2024-01-16 07:27:59,457.457 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 49879 2024-01-16T07:27:59.458635522Z [D 2024-01-16 07:27:59,458.458 launch_ipykernel] JSON Payload 'b'{"shell_port": 48549, "iopub_port": 46659, "stdin_port": 41937, "control_port": 48795, "hb_port": 34137, "ip": "0.0.0.0", "key": "dd63bcf1-d717-45c8-a6d8-a863cc49b70d", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": 9, "pgid": 7, "comm_port": 49879, "kernel_id": "95aae2ac-27c3-4912-a326-cedfbc52d302"}' 2024-01-16T07:27:59.471050920Z [D 2024-01-16 07:27:59,470.470 launch_ipykernel] Encrypted Payload

@ Jupyter Jovyan pod logs from jupyterhub namespaces

[I 2024-01-16 09:21:37.501 ServerApp] 200 GET /user/jovyan/api/terminals?1705396900391 (jovyan@127.0.0.1) 1.18ms 2024-01-16T09:21:37.503435541Z [I 2024-01-16 09:21:37.503 ServerApp] 200 GET /user/jovyan/api/sessions?1705396900393 (jovyan@127.0.0.1) 0.99ms 2024-01-16T09:21:37.517833196Z [I 2024-01-16 09:21:37.517 ServerApp] 200 GET /user/jovyan/api/kernels?1705396900394 (jovyan@127.0.0.1) 15.01ms 2024-01-16T09:21:39.236754736Z [I 2024-01-16 09:21:39.236 ServerApp] 200 GET /user/jovyan/api/contents?content=1&1705396902121 (jovyan@127.0.0.1) 4.88ms 2024-01-16T09:21:47.908831994Z [I 2024-01-16 09:21:47.908 ServerApp] 200 GET /user/jovyan/api/terminals?1705396910794 (jovyan@127.0.0.1) 0.91ms 2024-01-16T09:21:47.910021815Z [I 2024-01-16 09:21:47.909 ServerApp] 200 GET /user/jovyan/api/sessions?1705396910795 (jovyan@127.0.0.1) 0.86ms 2024-01-16T09:21:47.931271425Z [I 2024-01-16 09:21:47.931 ServerApp] 200 GET /user/jovyan/api/kernels?1705396910796 (jovyan@127.0.0.1) 21.00ms 2024-01-16T09:21:49.908197695Z [I 2024-01-16 09:21:49.908 ServerApp] 200 GET /user/jovyan/api/contents?content=1&1705396912802 (jovyan@127.0.0.1) 4.60ms 2024-01-16T09:22:08.893515385Z [I 2024-01-16 09:22:08.893 ServerApp] 200 GET /user/jovyan/api/me?1705396931794 (jovyan@127.0.0.1) 0.92ms 2024-01-16T09:22:08.914157365Z [I 2024-01-16 09:22:08.914 ServerApp] 200 GET /user/jovyan/api/kernelspecs?1705396931796 (jovyan@127.0.0.1) 20.28ms 2024-01-16T09:23:12.475285982Z [I 2024-01-16 09:23:12.475 ServerApp] 101 GET /user/jovyan/api/events/subscribe?token=[secret] (jovyan@127.0.0.1) 0.92ms

@ JEG logs from enterprise-gateway namespaces

[D 2024-01-16 09:52:30.293 EnterpriseGatewayApp] Opening websocket /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels 2024-01-16T09:52:30.293618430Z [D 2024-01-16 09:52:30.293 EnterpriseGatewayApp] Getting buffer for 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.293653830Z [D 2024-01-16 09:52:30.293 EnterpriseGatewayApp] Clearing buffer for 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.294071185Z [D 2024-01-16 09:52:30.293 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:37401 2024-01-16T09:52:30.294372849Z [D 2024-01-16 09:52:30.294 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45771 2024-01-16T09:52:30.294604143Z [D 2024-01-16 09:52:30.294 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45683 2024-01-16T09:52:30.294807040Z [D 2024-01-16 09:52:30.294 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:55985 2024-01-16T09:52:30.295022219Z [D 2024-01-16 09:52:30.294 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45771 2024-01-16T09:52:30.295227226Z [D 2024-01-16 09:52:30.295 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45683 2024-01-16T09:52:30.306247559Z [D 2024-01-16 09:52:30.306 EnterpriseGatewayApp] Nudge: attempt 1 on kernel 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.317213473Z [D 2024-01-16 09:52:30.317 EnterpriseGatewayApp] Nudge: shell info reply received: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.317245619Z [D 2024-01-16 09:52:30.317 EnterpriseGatewayApp] Nudge: resolving shell future: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.317409505Z [D 2024-01-16 09:52:30.317 EnterpriseGatewayApp] Nudge: control info reply received: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.317546189Z [D 2024-01-16 09:52:30.317 EnterpriseGatewayApp] Nudge: IOPub received: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.317632103Z [D 2024-01-16 09:52:30.317 EnterpriseGatewayApp] Nudge: resolving iopub future: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.318313 [D 2024-01-16 09:52:30.318 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (busy) 2024-01-16T09:52:30.318947681Z [D 2024-01-16 09:52:30.318 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (busy) 2024-01-16T09:52:30.324847036Z [D 2024-01-16 09:52:30.324 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (idle) 2024-01-16T09:52:30.330626121Z [D 2024-01-16 09:52:30.330 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (idle) ** 2024-01-16T09:52:30.571523499Z [E 240116 09:52:30 web:1789] Uncaught exception GET /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels (10.1.51.152) 2024-01-16T09:52:30.571554861Z HTTPServerRequest(protocol='http', host='enterprise-gateway.enterprise-gateway.svc.cluster.local:8888', method='GET', uri='/api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels', version='HTTP/1.1', remote_ip='10.1.51.152') 2024-01-16T09:52:30.571560130Z Traceback (most recent call last): 2024-01-16T09:52:30.571563161Z File "/opt/conda/lib/python3.9/site-packages/tornado/websocket.py", line 647, in _run_callback result = callback(*args, kwargs) 2024-01-16T09:52:30.571568814Z File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/handlers.py", line 460, in on_message 2024-01-16T09:52:30.571572117Z msg = json.loads(msg) 2024-01-16T09:52:30.571574855Z File "/opt/conda/lib/python3.9/json/init.py", line 346, in loads 2024-01-16T09:52:30.571577748Z return _default_decoder.decode(s) 2024-01-16T09:52:30.571580142Z File "/opt/conda/lib/python3.9/json/decoder.py", line 337, in decode 2024-01-16T09:52:30.571582805Z obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 2024-01-16T09:52:30.571585281Z File "/opt/conda/lib/python3.9/json/decoder.py", line 355, in raw_decode 2024-01-16T09:52:30.571587687Z raise JSONDecodeError("Expecting value", s, err.value) from None 2024-01-16T09:52:30.571596687Z json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) ` 2024-01-16T09:52:30.571676704Z [D 2024-01-16 09:52:30.571 EnterpriseGatewayApp] Websocket closed 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795:c00d7be5-5fc242679e133c40cab24c39 2024-01-16T09:52:30.571791424Z [I 2024-01-16 09:52:30.571 EnterpriseGatewayApp] Starting buffering for 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795:c00d7be5-5fc242679e133c40cab24c39 2024-01-16T09:52:30.571868813Z [D 2024-01-16 09:52:30.571 EnterpriseGatewayApp] Clearing buffer for 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.940577349Z [D 2024-01-16 09:52:30.940 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels 2024-01-16T09:52:30.946350673Z [W 2024-01-16 09:52:30.946 EnterpriseGatewayApp] No session ID specified 2024-01-16T09:52:30.952057725Z [I 240116 09:52:30 web:2239] 101 GET /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels (10.1.51.152) 11.71ms 2024-01-16T09:52:30.952185076Z [D 2024-01-16 09:52:30.952 EnterpriseGatewayApp] Opening websocket /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels 2024 -01-16T09:52:30.952311797Z [D 2024-01-16 09:52:30.952 EnterpriseGatewayApp] Getting buffer for 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.952392150Z [D 2024-01-16 09:52:30.952 EnterpriseGatewayApp] Clearing buffer for 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.952791054Z [D 2024-01-16 09:52:30.952 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:37401 2024-01-16T09:52:30.953085292Z [D 2024-01-16 09:52:30.952 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45771 2024-01-16T09:52:30.953349048Z [D 2024-01-16 09:52:30.953 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45683 [D 2024-01-16 09:52:30.953 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:55985 2024-01-16T09:52:30.953859071Z [D 2024-01-16 09:52:30.953 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45771 2024-01-16T09:52:30.954128010Z [D 2024-01-16 09:52:30.954 EnterpriseGatewayApp] Connecting to: tcp://10.1.54.81:45683 2024-01-16T09:52:30.959844693Z [D 2024-01-16 09:52:30.959 EnterpriseGatewayApp] Nudge: attempt 1 on kernel 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.970785911Z [D 2024-01-16 09:52:30.970 EnterpriseGatewayApp] Nudge: shell info reply received: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.970849940Z [D 2024-01-16 09:52:30.970 EnterpriseGatewayApp] Nudge: resolving shell future: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.970998832Z [D 2024-01-16 09:52:30.970 EnterpriseGatewayApp] Nudge: control info reply received: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.976411879Z [D 2024-01-16 09:52:30.976 EnterpriseGatewayApp] Nudge: IOPub received: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.976490467Z [D 2024-01-16 09:52:30.976 EnterpriseGatewayApp] Nudge: resolving iopub future: 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795 2024-01-16T09:52:30.977053425Z [D 2024-01-16 09:52:30.976 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (busy) 2024-01-16T09:52:30.977632857Z [D 2024-01-16 09:52:30.977 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (busy) 2024-01-16T09:52:30.978464061Z [D 2024-01-16 09:52:30.978 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (idle) 2024-01-16T09:52:30.984265095Z [D 2024-01-16 09:52:30.984 EnterpriseGatewayApp] activity on 6c332ecc-0b9f-43ad-88b4-79f2cfa8c795: status (idle) [D 2024-01-16 09:52:31.007 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels 2024-01-16T09:52:31.013301319Z [W 2024-01-16 09:52:31.012 EnterpriseGatewayApp] No session ID specified 2024-01-16T09:52:31.024109715Z [I 240116 09:52:31 web:2239] 101 GET /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels (10.1.51.152) 16.90ms 2024-01-16T09:52:31.024225776Z [D 2024-01-16 09:52:31.024 EnterpriseGatewayApp] Opening websocket /api/kernels/6c332ecc-0b9f-43ad-88b4-79f2cfa8c795/channels`

welcome[bot] commented 5 months ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

roshan3133 commented 5 months ago

i am getting KERNEL_NAME is empty in newly created remote kernel pod while launching ipython kernel in jupyterhub vai JEG. is this error because of this empty KERNEL_NAME ? in JEG pod i can see {"name":"KERNEL_NAME","value":"python_kubernetes"}]

launch_ipykernel] JSON Payload 'b'{"shell_port": 55055, "iopub_port": 44507, "stdin_port": 56833, "control_port": 60735, "hb_port": 57375, "ip": "0.0.0.0", "key": "17fd8e85-7e4d-4e29-946e-9a6975c2b0cc", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": 9, "pgid": 7, "comm_port": 33997, "kernel_id": "d80c56dc-8da1-4da4-a023-a65cc99ee0ca"}'

roshan3133 commented 5 months ago

looks like empty string passing to json.decoder() function as per error, am i missing passing any extraEnv in jupyterhub yaml ?

[D 2024-01-18 13:23:40.540 EnterpriseGatewayApp] Connecting to: tcp://10.1.35.142:60601 [D 2024-01-18 13:23:40.546 EnterpriseGatewayApp] Nudge: attempt 1 on kernel d80c56dc-8da1-4da4-a023-a65cc99ee0ca [D 2024-01-18 13:23:40.553 EnterpriseGatewayApp] activity on d80c56dc-8da1-4da4-a023-a65cc99ee0ca: status (busy) [D 2024-01-18 13:23:40.553 EnterpriseGatewayApp] Nudge: IOPub received: d80c56dc-8da1-4da4-a023-a65cc99ee0ca [D 2024-01-18 13:23:40.553 EnterpriseGatewayApp] Nudge: resolving iopub future: d80c56dc-8da1-4da4-a023-a65cc99ee0ca 2024-01-18T13:23:40.553663000Z [D 2024-01-18 13:23:40.553 EnterpriseGatewayApp] Nudge: shell info reply received: d80c56dc-8da1-4da4-a023-a65cc99ee0ca [D 2024-01-18 13:23:40.553 EnterpriseGatewayApp] Nudge: resolving shell future: d80c56dc-8da1-4da4-a023-a65cc99ee0ca [D 2024-01-18 13:23:40.554 EnterpriseGatewayApp] activity on d80c56dc-8da1-4da4-a023-a65cc99ee0ca: status (busy) [D 2024-01-18 13:23:40.555 EnterpriseGatewayApp] activity on d80c56dc-8da1-4da4-a023-a65cc99ee0ca: status (idle) [D 2024-01-18 13:23:40.561 EnterpriseGatewayApp] activity on d80c56dc-8da1-4da4-a023-a65cc99ee0ca: status (idle) [E 240118 13:23:40 web:1798] Uncaught exception GET /api/kernels/d80c56dc-8da1-4da4-a023-a65cc99ee0ca/channels (10.1.56.241) HTTPServerRequest(protocol='http', host='enterprise-gateway.enterprise-gateway.svc.cluster.local:8888', method='GET', uri='/api/kernels/d80c56dc-8da1-4da4-a023-a65cc99ee0ca/channels', version='HTTP/1.1', remote_ip='10.1.56.241') 2024-01-18T13:23:40.802899366Z Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/tornado/websocket.py", line 635, in _run_callback 2024-01-18T13:23:40.802907546Z result = callback(*args, **kwargs) 2024-01-18T13:23:40.802914544Z File "/opt/conda/lib/python3.10/site-packages/jupyter_server/services/kernels/handlers.py", line 521, in on_message msg = json.loads(ws_msg) File "/opt/conda/lib/python3.10/json/__init__.py", line 346, in loads 2024-01-18T13:23:40.802927209Z return _default_decoder.decode(s) File "/opt/conda/lib/python3.10/json/decoder.py", line 337, in decode 2024-01-18T13:23:40.802935284Z obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/opt/conda/lib/python3.10/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None 2024-01-18T13:23:40.802948320Z json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) [D 2024-01-18 13:23:40.802 EnterpriseGatewayApp] Websocket closed d80c56dc-8da1-4da4-a023-a65cc99ee0ca:d0497c42-79f383cb00eeb05ff1480647