search

skai-x / elastic-jupyter-operator

Cloud-native way to provide elastic Jupyter Notebooks on Kubernetes. Run remote kernels, natively.
Apache License 2.0
197 stars 29 forks source link

Running the custom kernel example - error when starting kernel from JupyterLab #57

Open rathboma opened 2 years ago

rathboma commented 2 years ago

When going through the custom kernel example everything starts correctly in k8s, but when starting the kernel from the notebook it produced a 500 HTTP error:

image

Seems like the error is some version incompatibility with enterprise gateway. Here are the logs from the gateway:

Error: TypeError: __init__() got an unexpected keyword argument 'kernel_id'

 HTTPServerRequest(protocol='http', host='jupytergateway-elastic-with-custom-kernels.default:8888', method='POST', uri='/api/kernels', version='HTTP/1.1', remote_ip='172.17.0.1')
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/handlers.py", line 90, in post
await super(MainKernelHandler, self).post()
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/handlers.py", line 47, in post
kernel_id = await km.start_kernel(kernel_name=model["name"], path=model.get("path"))
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 156, in start_kernel
kernel_id = await super(RemoteMappingKernelManager, self).start_kernel(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 211, in start_kernel
kernel_id = await ensure_async(self.pinned_superclass.start_kernel(self, **kwargs))
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/utils.py", line 189, in ensure_async
result = await obj
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 194, in _async_start_kernel
await fut
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 165, in _add_kernel_when_ready
await kernel_awaitable
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async
return await obj
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 365, in start_kernel
await super(RemoteKernelManager, self).start_kernel(**kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 362, in _async_start_kernel
raise e
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 351, in _async_start_kernel
await ensure_async(self._launch_kernel(kernel_cmd, **kw))
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async
return await obj
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 425, in _launch_kernel
proxy = await self.process_proxy.launch_process(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/kubeflow.py", line 49, in launch_process
await super(KubeflowProcessProxy, self).launch_process(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/container.py", line 81, in launch_process
self.local_proc = self.launch_kernel(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/processproxy.py", line 456, in launch_kernel
return launch_kernel(cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 169, in launch_kernel
raise ex
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 157, in launch_kernel
proc = Popen(cmd, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'kernel_id'
[E 220201 17:06:22 web:2239] 500 POST /api/kernels (172.17.0.1) 23.14ms

This seems related to this issue: https://github.com/jupyter/enterprise_gateway/issues/1024

Seems like it's using jupyter-client 7.1.0, which might be the problem? I'll try to confirm

gaocegege commented 2 years ago

Can you please show the image tag of the gateway and kernel?

ycsong1212 commented 2 years ago

Hello, could I know whether you solve this problem? I meet the same issue, I check the version of jupyter_client is 7.1.0. Could changing the version of jupyter_client work? I try to deploy with enterprise-gateway alone instead of elastic-jupyter-operator, that works. I'll appreciate it if you could help.

ycsong1212 commented 2 years ago

Can you please show the image tag of the gateway and kernel?

Hello, Mr. Gao. I meet the same question. I try the demo as quick-start but it doesn't work. The tags of my gateway and kernel are both 2.6.0. I want to know whether some version conflict exists. I have tried deploying only with enterprise-gateway and it works. I am interested in what kind of differences between your images and theirs. I am sorry I cannot access your image information and I do not obtain the answer in your code. I'll appreciate it if you could help.

gaocegege commented 2 years ago

OK I will verify it today.

gaocegege commented 2 years ago

@ycsong1212 Did you follow the guide here? https://github.com/skai-x/elastic-jupyter-operator/tree/master/docs#remote-kernels

Remote kernel or Remote kernels with custom configuration

gaocegege commented 2 years ago

Or this https://github.com/skai-x/elastic-jupyter-operator/blob/master/docs/quick-start.md

ycsong1212 commented 2 years ago

@ycsong1212 Did you follow the guide here? https://github.com/skai-x/elastic-jupyter-operator/tree/master/docs#remote-kernels

Remote kernel or Remote kernels with custom configuration

I think I do. First I apply crds by prepare.yaml and then make deploy. I tried elastic-with-custom-kernels demo. I apply four crds under examples/elastic-with-custom-kernels directory. Then I port-forward my notebook pod. When I launch kernel, kernel error occurs. Such error is reported in gateway container.

DianaDai commented 2 years ago

I have same error， how can i fix it?

yElaine commented 2 years ago

I have same error... image

image

gateway log error message: kubectl logs jupytergateway-elastic-with-custom-kernels-75bf76cc5b-85tbw -f

[I 220707 10:14:18 web:2239] 200 GET /api/kernels (10.244.2.XX) 6.00ms [D 2022-07-07 10:14:19.932 EnterpriseGatewayApp] Found kernel python3 in /opt/conda/share/jupyter/kernels [D 2022-07-07 10:14:19.932 EnterpriseGatewayApp] Found kernel python-kubernetes in /usr/local/share/jupyter/kernels [I 220707 10:14:19 web:2239] 200 GET /api/kernelspecs (10.244.2.XX) 6.92ms [I 220707 10:14:29 web:2239] 200 GET /api/kernels (10.244.2.XX) 5.81ms [D 2022-07-07 10:14:46.055 EnterpriseGatewayApp] RemoteMappingKernelManager.start_kernel: python-kubernetes, kernel_username: jovyan [D 2022-07-07 10:14:46.061 EnterpriseGatewayApp] Instantiating kernel 'Elastic Python Kernel on Kubernetes' with process proxy: enterprise_gateway.services.processproxies.kubeflow.KubeflowProcessProxy [D 2022-07-07 10:14:46.062 EnterpriseGatewayApp] Instantiating kernel 'Elastic Python Kernel on Kubernetes' with kernel provisioner: local-provisioner [D 2022-07-07 10:14:46.063 EnterpriseGatewayApp] Starting kernel: ['kubeflow-launcher', '--verbose', '--RemoteProcessProxy.kernel-id', '1bd1e838-e1bb-4692-94da-d02d0006fa17', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', '10.244.0.YY:8877', '--RemoteProcessProxy.public-key', 'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2Sw9kvggrPA/2UnCVv9boPK3I9NWLiP9ILMjn+4V+T0FXo6vfNYwl04muPBt7a5ScpiVaPXVg3J9xedBeqJuWxEhY2s9+ZqFJ6UQQ5TU8n1T8elRlCJYZaOTywN2qNPLx3SYg1U5slLOiO79wGyeq/ASBTPcze4v3rBBa0UjS9wIDAQAB', '--kernel-template-name', 'jupyterkerneltemplate-elastic-with-custom-kernels', '--kernel-template-namespace', 'default'] [D 2022-07-07 10:14:46.063 EnterpriseGatewayApp] Launching kernel: 'Elastic Python Kernel on Kubernetes' with command: ['kubeflow-launcher', '--verbose', '--RemoteProcessProxy.kernel-id', '1bd1e838-e1bb-4692-94da-d02d0006fa17', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', '10.244.0.YY:8877', '--RemoteProcessProxy.public-key', 'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2Sw9kvggrPA/2UnCVv9boPK3I9NWLiP9ILMjn+4V+T0FXo6vfNYwl04muPBt7a5ScpiVaPXVg3J9xedBeqJuWxEhY2s9+ZqFJ6UQQ5TU8n1T8elRlCJYZaOTywN2qNPLx3SYg1U5slLOiO79wGyeq/ASBTPcze4v3rBBa0UjS9wIDAQAB', '--kernel-template-name', 'jupyterkerneltemplate-elastic-with-custom-kernels', '--kernel-template-namespace', 'default'] [W 2022-07-07 10:14:46.063 EnterpriseGatewayApp] Shared namespace has been configured. All kernels will reside in EG namespace: default [D 2022-07-07 10:14:46.063 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'SHELL': '/bin/bash', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'EG_MIRROR_WORKING_DIRS': 'false', 'KUBERNETES_SERVICE_PORT': '443', 'EG_NAMESPACE': 'default', 'HOSTNAME': 'jupytergateway-elastic-with-custom-kernels-75bf76cc5b-85tbw', 'LANGUAGE': 'en_US.UTF-8', 'EG_SHARED_NAMESPACE': 'true', 'EG_PORT': '8888', 'EG_LOG_LEVEL': 'DEBUG', 'EG_NAME': 'jupytergateway-elastic-with-custom-kernels', 'EG_KERNEL_WHITELIST': "'python-kubernetes'", 'NB_UID': '1000', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_SERVICE_HOST': '10.105.28.ZZ', 'PWD': '/usr/local/bin', 'EG_CULL_IDLE_TIMEOUT': '10', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_PORT_8888_TCP_PORT': '8888', 'EG_DEFAULT_KERNEL_NAME': 'python_kubernetes', 'EG_ENABLE_TUNNELING': 'False', 'EG_KERNEL_LAUNCH_TIMEOUT': '60', 'HOME': '/home/jovyan', 'LANG': 'en_US.UTF-8', 'KUBERNETES_PORT_443_TCP': 'tcp://10.MM.0.1:443', 'EG_LIST_KERNELS': 'True', 'EG_SSH_PORT': '2122', 'NB_GID': '100', 'KG_PORT': '8888', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_SERVICE_PORT': '8888', 'EG_CULL_CONNECTED': 'False', 'EG_PORT_RETRIES': '0', 'KG_IP': '0.0.0.0', 'EG_PORT_RANGE': '0..0', 'EG_CULL_INTERVAL': '10', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_SERVICE_PORT_GATEWAY': '8888', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_PORT_8888_TCP': 'tcp://10.105.28.ZZ:8888', 'EG_IP': '0.0.0.0', 'SHLVL': '0', 'CONDA_DIR': '/opt/conda', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_PORT_8888_TCP_ADDR': '10.105.28.ZZ', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KG_PORT_RETRIES': '0', 'KUBERNETES_PORT_443_TCP_ADDR': '10.MM.0.1', 'EG_KERNEL_IMAGE': 'ghcr.io/skai-x/jupyter-kernel-py:2.6.0', 'NB_USER': 'jovyan', 'KUBERNETES_SERVICE_HOST': '10.MM.0.1', 'LC_ALL': 'en_US.UTF-8', 'KUBERNETES_PORT': 'tcp://10.MM.0.1:443', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_PORT': 'tcp://10.105.28.ZZ:8888', 'EG_KERNEL_CLUSTER_ROLE': 'enterprise-gateway-controller', 'DEBIAN_FRONTEND': 'noninteractive', 'JUPYTERGATEWAY_ELASTIC_WITH_CUSTOM_KERNELS_PORT_8888_TCP_PROTO': 'tcp', 'KERNEL_LAUNCH_TIMEOUT': '40', 'KERNEL_USERNAME': 'jovyan', 'KERNEL_GATEWAY': '1', 'KERNEL_POD_NAME': 'jovyan-1bd1e838-e1bb-4692-94da-d02d0006fa17', 'KERNEL_SERVICE_ACCOUNT_NAME': 'default', 'KERNEL_NAMESPACE': 'default', 'KERNEL_IMAGE': 'ghcr.io/skai-x/jupyter-kernel-py:2.6.0', 'KERNEL_EXECUTOR_IMAGE': 'ghcr.io/skai-x/jupyter-kernel-py:2.6.0', 'KERNEL_UID': '1000', 'KERNEL_GID': '100', 'EG_MIN_PORT_RANGE_SIZE': '1000', 'EG_MAX_PORT_RANGE_RETRIES': '5', 'KERNEL_ID': '1bd1e838-e1bb-4692-94da-d02d0006fa17', 'KERNEL_LANGUAGE': 'python', 'EG_IMPERSONATION_ENABLED': 'False'} [E 2022-07-07 10:14:46.063 EnterpriseGatewayApp] Failed to run command: ['kubeflow-launcher', '--verbose', '--RemoteProcessProxy.kernel-id', '1bd1e838-e1bb-4692-94da-d02d0006fa17', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', '10.244.0.YY:8877', '--RemoteProcessProxy.public-key', 'MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC2Sw9kvggrPA/2UnCVv9boPK3I9NWLiP9ILMjn+4V+T0FXo6vfNYwl04muPBt7a5ScpiVaPXVg3J9xedBeqJuWxEhY2s9+ZqFJ6UQQ5TU8n1T8elRlCJYZaOTywN2qNPLx3SYg1U5slLOiO79wGyeq/ASBTPcze4v3rBBa0UjS9wIDAQAB', '--kernel-template-name', 'jupyterkerneltemplate-elastic-with-custom-kernels', '--kernel-template-namespace', 'default'] PATH='/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' with kwargs: {'kernel_id': '1bd1e838-e1bb-4692-94da-d02d0006fa17', 'stdin': -1, 'stdout': None, 'stderr': None, 'cwd': None, 'start_new_session': True}

[E 2022-07-07 10:14:46.063 EnterpriseGatewayApp] init() got an unexpected keyword argument 'kernel_id' Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 351, in _async_start_kernel await ensure_async(self._launch_kernel(kernel_cmd, kw)) File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async return await obj File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 425, in _launch_kernel proxy = await self.process_proxy.launch_process(kernel_cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/kubeflow.py", line 49, in launch_process await super(KubeflowProcessProxy, self).launch_process(kernel_cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/container.py", line 81, in launch_process self.local_proc = self.launch_kernel(kernel_cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/processproxy.py", line 456, in launch_kernel return launch_kernel(cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 169, in launch_kernel raise ex File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 157, in launch_kernel proc = Popen(cmd, kwargs) TypeError: init() got an unexpected keyword argument 'kernel_id' [E 220707 10:14:46 web:1789] Uncaught exception POST /api/kernels (10.244.2.XX) HTTPServerRequest(protocol='http', host='jupytergateway-elastic-with-custom-kernels.default:8888', method='POST', uri='/api/kernels', version='HTTP/1.1', remote_ip='10.244.2.XX') Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute result = await result File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/handlers.py", line 90, in post await super(MainKernelHandler, self).post() File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/handlers.py", line 47, in post kernel_id = await km.start_kernel(kernel_name=model["name"], path=model.get("path")) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 156, in start_kernel kernel_id = await super(RemoteMappingKernelManager, self).start_kernel(*args, kwargs) File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 211, in start_kernel kernel_id = await ensure_async(self.pinned_superclass.start_kernel(self, kwargs)) File "/opt/conda/lib/python3.9/site-packages/jupyter_server/utils.py", line 189, in ensure_async result = await obj File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 194, in _async_start_kernel await fut File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 165, in _add_kernel_when_ready await kernel_awaitable File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async return await obj File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 365, in start_kernel await super(RemoteKernelManager, self).start_kernel(kwargs) File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 362, in _async_start_kernel raise e File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 351, in _async_start_kernel await ensure_async(self._launch_kernel(kernel_cmd, kw)) File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async return await obj File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 425, in _launch_kernel proxy = await self.process_proxy.launch_process(kernel_cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/kubeflow.py", line 49, in launch_process await super(KubeflowProcessProxy, self).launch_process(kernel_cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/container.py", line 81, in launch_process self.local_proc = self.launch_kernel(kernel_cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/processproxy.py", line 456, in launch_kernel return launch_kernel(cmd, kwargs) File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 169, in launch_kernel raise ex File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 157, in launch_kernel proc = Popen(cmd, **kwargs) TypeError: init() got an unexpected keyword argument 'kernel_id' [E 220707 10:14:46 web:2239] 500 POST /api/kernels (10.244.2.XX) 20.42ms [I 220707 10:14:49 web:2239] 200 GET /api/kernels (10.244.2.XX) 5.90ms

gaocegege commented 2 years ago

I did not reproduce it before. I will take a look ASAP but not sure if I can reproduce this time.

gaocegege commented 2 years ago

TypeError: init() got an unexpected keyword argument 'kernel_id'

It looks like a version problem related to enterprise gateway.

EugeneTorap commented 2 years ago

@gaocegege Let's try to fix the issue

andyzheung commented 1 year ago

This repo is not maintained?

gaocegege commented 1 year ago

This repo is not maintained?

The org and repository belong to Tencent Cloud. And I am not sure if there is actively maintained.

Gamattention commented 1 month ago

When going through the custom kernel example everything starts correctly in k8s, but when starting the kernel from the notebook it produced a 500 HTTP error:

image

Seems like the error is some version incompatibility with enterprise gateway. Here are the logs from the gateway:

Error: TypeError: __init__() got an unexpected keyword argument 'kernel_id'

 HTTPServerRequest(protocol='http', host='jupytergateway-elastic-with-custom-kernels.default:8888', method='POST', uri='/api/kernels', version='HTTP/1.1', remote_ip='172.17.0.1')
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/handlers.py", line 90, in post
await super(MainKernelHandler, self).post()
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/handlers.py", line 47, in post
kernel_id = await km.start_kernel(kernel_name=model["name"], path=model.get("path"))
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 156, in start_kernel
kernel_id = await super(RemoteMappingKernelManager, self).start_kernel(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 211, in start_kernel
kernel_id = await ensure_async(self.pinned_superclass.start_kernel(self, **kwargs))
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/utils.py", line 189, in ensure_async
result = await obj
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 194, in _async_start_kernel
await fut
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 165, in _add_kernel_when_ready
await kernel_awaitable
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async
return await obj
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 365, in start_kernel
await super(RemoteKernelManager, self).start_kernel(**kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 362, in _async_start_kernel
raise e
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/manager.py", line 351, in _async_start_kernel
await ensure_async(self._launch_kernel(kernel_cmd, **kw))
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/utils.py", line 34, in ensure_async
return await obj
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/kernels/remotemanager.py", line 425, in _launch_kernel
proxy = await self.process_proxy.launch_process(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/kubeflow.py", line 49, in launch_process
await super(KubeflowProcessProxy, self).launch_process(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/container.py", line 81, in launch_process
self.local_proc = self.launch_kernel(kernel_cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/enterprise_gateway/services/processproxies/processproxy.py", line 456, in launch_kernel
return launch_kernel(cmd, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 169, in launch_kernel
raise ex
File "/opt/conda/lib/python3.9/site-packages/jupyter_client/launcher.py", line 157, in launch_kernel
proc = Popen(cmd, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'kernel_id'
[E 220201 17:06:22 web:2239] 500 POST /api/kernels (172.17.0.1) 23.14ms

This seems related to this issue: jupyter-server/enterprise_gateway#1024

Seems like it's using jupyter-client 7.1.0, which might be the problem? I'll try to confirm

I encountered the same issue, have you resolved it?