search

jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
619 stars 223 forks source link

a matlab kernel launcher is required #795

Closed lazywhite closed 2 years ago

lazywhite commented 4 years ago

I installed a matlab jupyter kernel

It seems like different kernel need different launcher script, like this etc/kernel-launchers/python/scripts/launch_ipykernel.py

now I need a kernel launcher to run matlab kernel , How can I do it?

kevin-bates commented 4 years ago

Yeah, although given the MatLab kernel is python, you could probably use most of it.

What would be great is to plumb an option to specify a different kernel (e.g. --kernel=matlab). Then, start_ipython() could be start_kernel() and we add a switch on the kernel to start, with embed_kernel() being the default.

In looking at MatLab, it (and other MetaKernels) appear to be strongly dependent on configuration files. This could be a problem unless you only want to launch containerized versions since we'd need to figure out a way to get the configuration over to the node on which the kernel has landed. With containerized versions, we could bake the configuration into the image or convey the various parameters via the kernel-pod.yaml, etc. What kind of environment are you targeting for this?

I'm happy to walk you through this and answer any questions, but I'm rather tied up with a few things at the moment. Since we have yet to add support for a kernel outside of IPyKernel, IRKernel and Apache Toree (scala) this would be a great opportunity to "test those waters"! Thank you for your interest.

lazywhite commented 4 years ago

environment

k8s: v1.13.12
jupyter-enterprise-gateway: 2.1.0
matlab: 2018b (it require python<3.7, I've build a py36 version of jupyter/base-image)

my goal is building an image like elyra/kernel-py:2.1.0 and a kernelspec for gateway to launch a pod with matlab kernel just like "python_kubernetes" etc...

lazywhite commented 4 years ago

@kevin-bates I tried as you said, here is launch_ipykernel.py I add an argument in line 384, now I can boot a pod with this kernel through jupyter-gateway

but top right of notebook page show kernel status, it's always a big black dot, not white annular. and it can't process phrase like 

a = -10:0.01:10
b = sin(a)
plot(b)

please help me to figure out the wrong code.

kevin-bates commented 4 years ago

Ok. The changes look fine. I would prefer the parameter be called --kernel or --kernel_name, but this is fine for now. Got a few questions:

  1. Do python kernels still work? We'll want to have a default value for the parameter and might as well use ipykernel as the default even though it will also be handled in the else: block.
  2. You must have figured out to pass the kernel short name to the container via env KERNEL_SHORTNAME or something like that that the bootstrap-kernel.sh file processes, correct?
  3. What does the EG log show for the complete startup sequence?

The black dot indicates the front-end still believes the kernel to be starting and is not available to receive commands. The EG log, when working correctly, should show a kernel-info-request and kernel-info-reply sequence getting sent/received.

lazywhite commented 4 years ago

my answer to your question

  1. default value is not as import as making matlab kernel work :)
  2. yes
  3. logs

    gateway log

[D 2020-03-26 15:27:42.618 EnterpriseGatewayApp] Found kernel python3 in /opt/conda/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel scala_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel r_docker in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel scala_docker in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel dask_python_yarn_remote in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_python_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_r_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_tf_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_tf_gpu_docker in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_python_yarn_client in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_matlab_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_distributed in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_r_yarn_cluster in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_r_conductor_cluster in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_tf_gpu_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_python_conductor_cluster in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_r_yarn_client in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel python_tf_docker in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_scala_conductor_cluster in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_scala_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel r_kubernetes in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.619 EnterpriseGatewayApp] Found kernel spark_scala_yarn_client in /usr/local/share/jupyter/kernels [D 2020-03-26 15:27:42.620 EnterpriseGatewayApp] Found kernel python_docker in /usr/local/share/jupyter/kernels [I 200326 15:27:42 web:2246] 200 GET /api/kernelspecs (10.209.156.222) 8.76ms [D 2020-03-26 15:27:43.827 EnterpriseGatewayApp] RemoteMappingKernelManager.start_kernel: python_matlab_kubernetes, kernel_username: demo [D 2020-03-26 15:27:43.829 EnterpriseGatewayApp] Instantiating kernel 'Python on Kubernetes with Matlab-2018b' with process proxy: enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy [D 2020-03-26 15:27:43.830 EnterpriseGatewayApp] Response socket launched on '192.168.0.95:33739' using 5.0s timeout [D 2020-03-26 15:27:43.830 EnterpriseGatewayApp] Starting kernel: ['/opt/conda/bin/python', '/usr/local/share/jupyter/kernels/python_matlab_kubernetes/scripts/launch_kubernetes.py', '--kernel-shortname', 'matlab', '--RemoteProcessProxy.kernel-id', 'ec963cd2-184c-4184-b8b7-e950d0f9a478', '--RemoteProcessProxy.response-address', '192.168.0.95:33739'] [D 2020-03-26 15:27:43.830 EnterpriseGatewayApp] Launching kernel: Python on Kubernetes with Matlab-2018b with command: ['/opt/conda/bin/python', '/usr/local/share/jupyter/kernels/python_matlab_kubernetes/scripts/launch_kubernetes.py', '--kernel-shortname', 'matlab', '--RemoteProcessProxy.kernel-id', 'ec963cd2-184c-4184-b8b7-e950d0f9a478', '--RemoteProcessProxy.response-address', '192.168.0.95:33739'] [I 2020-03-26 15:27:43.846 EnterpriseGatewayApp] Created kernel namespace: demo-ec963cd2-184c-4184-b8b7-e950d0f9a478 [I 2020-03-26 15:27:44.527 EnterpriseGatewayApp] Created kernel role-binding 'kernel-controller' in namespace: demo-ec963cd2-184c-4184-b8b7-e950d0f9a478 for service account: default [D 2020-03-26 15:27:44.528 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'LC_ALL': 'en_US.UTF-8', 'LANG': 'en_US.UTF-8', 'EG_SHARED_NAMESPACE': 'False', 'HOSTNAME': 'enterprise-gateway-76c4c75b54-kn4n2', 'EG_ENABLE_TUNNELING': 'False', 'KG_PORT_RETRIES': '0', 'NB_UID': '1000', 'EG_LOG_LEVEL': 'DEBUG', 'JAVA_HOME': '/usr/lib/jvm/java-8-openjdk-amd64', 'CONDA_DIR': '/opt/conda', 'ENTERPRISE_GATEWAY_PORT_8888_TCP_PORT': '8888', 'CONDA_VERSION': '4.7.12', 'SPARK_VER': '2.4.1', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KUBERNETES_PORT_443_TCP_ADDR': '10.96.0.1', 'EG_CULL_IDLE_TIMEOUT': '3600', 'ENTERPRISE_GATEWAY_SERVICE_HOST': '10.99.80.184', 'KUBERNETES_PORT': 'tcp://10.96.0.1:443', 'PWD': '/usr/local/bin', 'HOME': '/home/jovyan', 'MINICONDA_MD5': '1c945f2b3335c7b2b15130b1b2dc5cf4', 'ENTERPRISE_GATEWAY_PORT_8888_TCP_PROTO': 'tcp', 'EG_MIRROR_WORKING_DIRS': 'False', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'DEBIAN_FRONTEND': 'noninteractive', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'EG_KERNEL_LAUNCH_TIMEOUT': '60', 'EG_SSH_PORT': '2122', 'ENTERPRISE_GATEWAY_SERVICE_PORT_HTTP': '8888', 'EG_CULL_INTERVAL': '60', 'SPARK_HOME': '/opt/spark', 'NB_USER': 'jovyan', 'EG_KERNEL_WHITELIST': "['r_kubernetes','python_kubernetes','python_tf_kubernetes','python_matlab_kubernetes','python_tf_gpu_kubernetes','scala_kubernetes','spark_r_kubernetes','spark_python_kubernetes','spark_scala_kubernetes']", 'KUBERNETES_PORT_443_TCP': 'tcp://10.96.0.1:443', 'EG_CULL_CONNECTED': 'False', 'EG_PORT_RETRIES': '0', 'KG_PORT': '8888', 'ENTERPRISE_GATEWAY_SERVICE_PORT': '8888', 'SHELL': '/bin/bash', 'ENTERPRISE_GATEWAY_PORT': 'tcp://10.99.80.184:8888', 'ENTERPRISE_GATEWAY_PORT_8888_TCP': 'tcp://10.99.80.184:8888', 'EG_PORT': '8888', 'ENTERPRISE_GATEWAY_PORT_8888_TCP_ADDR': '10.99.80.184', 'SHLVL': '0', 'LANGUAGE': 'en_US.UTF-8', 'EG_KERNEL_CLUSTER_ROLE': 'kernel-controller', 'KUBERNETES_SERVICE_PORT': '443', 'EG_NAMESPACE': 'enterprise-gateway', 'NB_GID': '100', 'KG_IP': '0.0.0.0', 'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'EG_IP': '0.0.0.0', 'KUBERNETES_SERVICE_HOST': '10.96.0.1', 'MINICONDA_VERSION': '4.7.10', 'KERNEL_LAUNCH_TIMEOUT': '40', 'KERNEL_USERNAME': 'demo', 'KERNEL_GATEWAY': '1', 'KERNEL_POD_NAME': 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', 'KERNEL_SERVICE_ACCOUNT_NAME': 'default', 'KERNEL_NAMESPACE': 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', 'KERNEL_IMAGE': 'availink/kernel-matlab:2.1.0-py36', 'KERNEL_EXECUTOR_IMAGE': 'availink/kernel-matlab:2.1.0-py36', 'KERNEL_UID': '1000', 'KERNEL_GID': '100', 'EG_MIN_PORT_RANGE_SIZE': '1000', 'EG_MAX_PORT_RANGE_RETRIES': '5', 'KERNEL_ID': 'ec963cd2-184c-4184-b8b7-e950d0f9a478', 'KERNEL_LANGUAGE': 'python', 'EG_IMPERSONATION_ENABLED': 'False'} [I 2020-03-26 15:27:44.533 EnterpriseGatewayApp] KubernetesProcessProxy: kernel launched. Kernel image: availink/kernel-matlab:2.1.0-py36, KernelID: ec963cd2-184c-4184-b8b7-e950d0f9a478, cmd: '['/opt/conda/bin/python', '/usr/local/share/jupyter/kernels/python_matlab_kubernetes/scripts/launch_kubernetes.py', '--kernel-shortname', 'matlab', '--RemoteProcessProxy.kernel-id', 'ec963cd2-184c-4184-b8b7-e950d0f9a478', '--RemoteProcessProxy.response-address', '192.168.0.95:33739']' [D 2020-03-26 15:27:45.249 EnterpriseGatewayApp] 1: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:45.761 EnterpriseGatewayApp] 2: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:46.272 EnterpriseGatewayApp] 3: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:47.016 EnterpriseGatewayApp] 4: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:48.357 EnterpriseGatewayApp] 5: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:48.867 EnterpriseGatewayApp] 6: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:49.377 EnterpriseGatewayApp] 7: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:49.888 EnterpriseGatewayApp] 8: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:50.416 EnterpriseGatewayApp] 9: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:52.405 EnterpriseGatewayApp] 10: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:53.004 EnterpriseGatewayApp] 11: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:53.514 EnterpriseGatewayApp] 12: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:54.038 EnterpriseGatewayApp] 13: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:54.551 EnterpriseGatewayApp] 14: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:55.123 EnterpriseGatewayApp] 15: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Pending', Pod IP: 'None', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:55.634 EnterpriseGatewayApp] 16: Waiting to connect to k8s pod in namespace 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478'. Name: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478', Status: 'Running', Pod IP: '192.168.0.80', KernelID: 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:57.829 EnterpriseGatewayApp] Received Payload 'MVpYPoAj/xNXozhF83DSPINZErImo7vGXJozL/Fw6RsNt3BJYHyR8iZNvD4YlM7Jn4O62GoqLWGxaHqOrLqRMMKghdj+CTc40Q/tLUQpin5JbuZvuZYpRrSL4qsAs1nYVPBfSoCuUJhPJfdzyNXGCgYe1sHhQalWxA0GqGuCocUhpCrxnRI1sQ5YGf6VapJZ7mGLk+C1b8oglwVka/q9OUufbjSqrFvt4VxZc1aXxQVOxQXJOYMwVo3AWU44kj/KaIdyAFOqMryE9o/wRBWPLHgdt6sPIacxw8wtMTAe69N7OEsB7JVtklVZoAGHGDg83v7zCrFu7U5OZnYCc9Jgbw5JckrrqBGZU7/E6hdA+1CAI5pGnUP6wXzQi5j/5FTVIX8cRR3KqxN374GlGtauCA==' [D 2020-03-26 15:27:57.829 EnterpriseGatewayApp] Decrypted Payload '{"shell_port": 59871, "iopub_port": 59997, "stdin_port": 42141, "control_port": 32979, "hb_port": 59461, "ip": "0.0.0.0", "key": "6b2715a4-b76b-4947-bd09-1e729f774274", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "11", "pgid": "8", "comm_port": 38683}' [D 2020-03-26 15:27:57.830 EnterpriseGatewayApp] Connect Info received from the launcher is as follows '{'shell_port': 59871, 'iopub_port': 59997, 'stdin_port': 42141, 'control_port': 32979, 'hb_port': 59461, 'ip': '0.0.0.0', 'key': '6b2715a4-b76b-4947-bd09-1e729f774274', 'transport': 'tcp', 'signature_scheme': 'hmac-sha256', 'kernel_name': '', 'pid': '11', 'pgid': '8', 'comm_port': 38683}' [D 2020-03-26 15:27:57.830 EnterpriseGatewayApp] Host assigned to the Kernel is: 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478' '192.168.0.80' [D 2020-03-26 15:27:57.830 EnterpriseGatewayApp] Established gateway communication to: 192.168.0.80:38683 for KernelID 'ec963cd2-184c-4184-b8b7-e950d0f9a478' [D 2020-03-26 15:27:57.830 EnterpriseGatewayApp] Updated pid to: 11 [D 2020-03-26 15:27:57.830 EnterpriseGatewayApp] Updated pgid to: 8 [D 2020-03-26 15:27:57.832 EnterpriseGatewayApp] Received connection info for KernelID 'ec963cd2-184c-4184-b8b7-e950d0f9a478' from host 'demo-ec963cd2-184c-4184-b8b7-e950d0f9a478': {'shell_port': 59871, 'iopub_port': 59997, 'stdin_port': 42141, 'control_port': 32979, 'hb_port': 59461, 'ip': '192.168.0.80', 'key': '6b2715a4-b76b-4947-bd09-1e729f774274', 'transport': 'tcp', 'signature_scheme': 'hmac-sha256', 'kernel_name': '', 'comm_port': 38683}... [D 2020-03-26 15:27:57.833 EnterpriseGatewayApp] Connecting to: tcp://192.168.0.80:32979 [D 2020-03-26 15:27:57.834 EnterpriseGatewayApp] Connecting to: tcp://192.168.0.80:59997 [I 2020-03-26 15:27:57.835 EnterpriseGatewayApp] Kernel started: ec963cd2-184c-4184-b8b7-e950d0f9a478 [D 2020-03-26 15:27:57.835 EnterpriseGatewayApp] Kernel args: {'env': {'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'KERNEL_LAUNCH_TIMEOUT': '40', 'KERNEL_USERNAME': 'demo', 'KERNEL_WORKING_DIR': '/home/ubuntu'}, 'kernel_name': 'python_matlab_kubernetes'} [I 200326 15:27:57 web:2246] 201 POST /api/kernels (10.209.156.222) 14010.47ms [D 2020-03-26 15:27:57.839 EnterpriseGatewayApp] Serving kernel resource from: /usr/local/share/jupyter/kernels/python_matlab_kubernetes [I 200326 15:27:57 web:2246] 200 GET /kernelspecs/python_matlab_kubernetes/logo-64x64.png (10.209.156.222) 3.15ms [I 200326 15:27:57 web:2246] 200 GET /api/kernels/ec963cd2-184c-4184-b8b7-e950d0f9a478 (10.209.156.222) 0.70ms [I 200326 15:27:57 web:2246] 200 GET /api/kernels/ec963cd2-184c-4184-b8b7-e950d0f9a478 (10.209.156.222) 0.46ms

## kernel-matlab pod log

ubuntu@gpu0-222:~$ kc logs -n demo-ec963cd2-184c-4184-b8b7-e950d0f9a478 demo-ec963cd2-184c-4184-b8b7-e950d0f9a478 /usr/local/bin/bootstrap-kernel.sh env: LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 HOSTNAME=demo-ec963cd2-184c-4184-b8b7-e950d0f9a478 KERNEL_NAMESPACE=demo-ec963cd2-184c-4184-b8b7-e950d0f9a478 NB_UID=1000 CONDA_DIR=/opt/conda CONDA_VERSION=4.8.2 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_PORT=tcp://10.96.0.1:443 EG_RESPONSE_ADDRESS=192.168.0.95:33739 PWD=/home/jovyan KERNEL_ID=ec963cd2-184c-4184-b8b7-e950d0f9a478 KERNEL_LANGUAGE=python HOME=/home/jovyan MINICONDA_MD5=87e77f097f6ebb5127c77662dfc3165e KUBERNETES_SERVICE_PORT_HTTPS=443 DEBIAN_FRONTEND=noninteractive KUBERNETES_PORT_443_TCP_PORT=443 NB_USER=jovyan KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 KERNEL_SPARK_CONTEXT_INIT_MODE=none KERNEL_USERNAME=demo SHELL=/bin/bash KERNEL_NAME=python_matlab_kubernetes KERNEL_SHORTNAME=matlab SHLVL=1 LANGUAGE=en_US.UTF-8 KUBERNETES_SERVICE_PORT=443 NB_GID=100 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin:/opt/conda/bin KUBERNETES_SERVICE_HOST=10.96.0.1 MINICONDAVERSION=4.8.2 =/usr/bin/env

kevin-bates commented 4 years ago

Thank you. You're right, defaults are not important right now - but do python kernels still work? I just want to make sure your general environment is working correctly.

All indications are that the launch was successful. The EG logs look correct, as do the kernel logs.

  1. Are you able to shutdown the kernel via the Notebook? You should see an indication in the kernel logs that it received the shutdown request, but this will happen immediately prior to the pod being deleted - so pod deletion is really the tell there.
  2. I'm not familiar with how the meta kernels work, but they do rely on configuration files. Is there something that can be put into the configuration that can somehow troubleshoot that the meta-kernel is working correctly?

I think this probably needs to get debugged from the Matlab level. The behavior in the current logs is very encouraging though. I suspect this will be something minor.

lazywhite commented 4 years ago

no, I can't shutdown through Kernel->Shutdown, after click, nothing happend, kernel-pod no new log

kevin-bates commented 4 years ago

You shouldn't need to rebuild the notebook image, just use one of the ones already available and invoke it with --gateway-url=<address to your EG server>. This will then redirect all kernel requests to EG.

By 'default kernel' I'm referring to the python_kubernetes kernelspec out of the box. There shouldn't be any changes to elyra/kernel-py:2.1.0 needed. I think we need to focus on getting what we know should work working first.

lazywhite commented 4 years ago

  1. You shouldn't need to rebuild the notebook image, just use one of the ones already available matlab-2018b require python < 3.7, you mean there is already a version built with python 3.6?

  2. There shouldn't be any changes to elyra/kernel-py:2.1.0 needed. In my matlab kernel image , jupyter kernelspec list return python3 and matlab two kernels if no modification to anything(like launch_ipykernel.py), how can notebook use the right matlab kernel

  3. I want to build a kernel-matlab image base on matlab_kernel , then I can use it just like elyra/kernel-py, can you describe the right way to achieve it?

kevin-bates commented 4 years ago

The kernel images, although built on notebook images, are not where notebook is running. Notebook should be running in a separate image and the version of python in it doesn't need to match the version of python required by the kernel. In traditional environments, where the kernel runs on the same node (or in the same image) as Notebook, that's not the case - the python versions must match.

The changes you made to launch_ipykernel.py reside in the kernel image. The ONLY kernel image that requires your changes is the image you're referencing in the kernelspec process-proxy stanza. The launch_ipykernel.py scripts in the existing kernel images don't no about KERNEL_SHORTNAME and their bootstrap-kernel.sh scripts won't pass the --kernel_shortname option to their launch scripts. In these cases, the env is ignored.

So you should only need to build the matlab kernel image - which you have done. And you only need place the python-matlab-kubernetes kernelspec into the EG image (or use a mount) - which you have done. So you should be able to confirm that the default kernel (python_kubernetes) works with no changes.

Perhaps it might help if you explain your configuration. Where is the notebook server running and where is the EG server running?

lazywhite commented 4 years ago

I think I have an expression error, elyra/kernel-py:2.1.0 originally depend on jupyter/base-notebook, I want to build a matlab kernel image, so I have to rebuild jupyter/base-notebook base on py36.

now I need to find out why default python3 kernel is not working

lazywhite commented 4 years ago

I've rebuild my kernel-matlab image, and default python3 kernel is working !! but now I still can't start matlab kernel correctly launch_ipykernel.py

pod error log

/usr/local/bin/bootstrap-kernel.sh env: LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 HOSTNAME=demo-11c9d079-51de-45bd-a9fd-6ad249b881c4 KERNEL_NAMESPACE=demo-11c9d079-51de-45bd-a9fd-6ad249b881c4 NB_UID=1000 CONDA_DIR=/opt/conda CONDA_VERSION=4.8.2 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1 KUBERNETES_PORT=tcp://10.96.0.1:443 EG_RESPONSE_ADDRESS=192.168.0.94:60229 PWD=/home/jovyan KERNEL_ID=11c9d079-51de-45bd-a9fd-6ad249b881c4 KERNEL_LANGUAGE=python HOME=/home/jovyan MINICONDA_MD5=87e77f097f6ebb5127c77662dfc3165e KUBERNETES_SERVICE_PORT_HTTPS=443 DEBIAN_FRONTEND=noninteractive KUBERNETES_PORT_443_TCP_PORT=443 NB_USER=jovyan KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443 KERNEL_SPARK_CONTEXT_INIT_MODE=none KERNEL_USERNAME=demo SHELL=/bin/bash KERNEL=matlab KERNEL_NAME=python_matlab_kubernetes SHLVL=1 LANGUAGE=en_US.UTF-8 KUBERNETES_SERVICE_PORT=443 XDG_CACHE_HOME=/home/jovyan/.cache/ NB_GID=100 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin:/opt/conda/bin KUBERNETES_SERVICE_HOST=10.96.0.1 MINICONDA_VERSION=4.8.2 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel matlab --RemoteProcessProxy.kernel-id 11c9d079-51de-45bd-a9fd-6ad249b881c4 --RemoteProcessProxy.response-address 192.168.0.94:60229 --RemoteProcessProxy.spark-context-initialization-mode none
[D 2020-03-27 11:10:19,563.563 launch_ipykernel] Using connection file '/tmp/kernel-11c9d079-51de-45bd-a9fd-6ad249b881c4_y_oud5yl.json'.
[I 2020-03-27 11:10:19,563.563 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 53233
[D 2020-03-27 11:10:19,564.564 launch_ipykernel] JSON Payload 'b'{"shell_port": 57809, "iopub_port": 49681, "stdin_port": 45589, "control_port": 35175, "hb_port": 39535, "ip": "0.0.0.0", "key": "ea20e38a-67cc-4ffd-b71f-fbe22286b8ac", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "12", "pgid": "9", "comm_port": 53233}'
[D 2020-03-27 11:10:19,564.564 launch_ipykernel] Encrypted Payload 'b'N4ixtCEzAfYnsj4e0f7d2h0ii/ElVwIYWygiJOkO5vLHeocU6gqfVPzB+nDuN/24vxw2HkColYnphjzWBYvjolyZoRtgMXzxMpyrjFjnOGkdPsTSltz2ChxGdKmMSOxdliNrbAM/w0K3osMrSYvSLrzIpOO5b3xejQgftAZmg/3+3L6EYEwvmsfFRPL0MF9hkZdlzuoMsBdIKri/DhjAm4APgGJ77Hh+vd7Xn6fQ0x2WKpyOkC4YA+IdiGjRYfZ7gNHxd4q+OO2IHI2jD2wO1gztxrx0cQUBh+zZaDiJdHzTUuum8WNkibEbr4uN5/6N9huKh63Qs+uSaf5YwBj4HdspaXEpO8+g0H0w784gePJXLnPy8A8OGSgdvzEW6BRd8iDDVhIC8YptjzD2/j0Z5g=='
[IPKernelApp] CRITICAL | Bad config encountered during initialization:
[IPKernelApp] CRITICAL | Unrecognized flag: '--kernel'
=================
matlab
===== enter matlab============
IPython: an enhanced interactive Python shell.

Subcommands
-----------

Subcommands are launched as `ipython-kernel cmd [args]`. For information on
using subcommand 'cmd', do: `ipython-kernel cmd -h`.

install
    Install the IPython kernel

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--debug
    set log level to logging.DEBUG (maximize logging output)
--quiet
    set log level to logging.CRITICAL (minimize logging output)
--init
    Initialize profile with default config files.  This is equivalent
    to running `ipython profile create <profile>` prior to startup.
--no-stdout
    redirect stdout to the null device
--no-stderr
    redirect stderr to the null device
--pylab
    Pre-load matplotlib and numpy for interactive use with
    the default matplotlib backend.
--autoindent
    Turn on autoindenting.
--no-autoindent
    Turn off autoindenting.
--automagic
    Turn on the auto calling of magic commands. Type %%magic at the
    IPython  prompt  for  more information.
--no-automagic
    Turn off the auto calling of magic commands.
--pdb
    Enable auto calling the pdb debugger after every exception.
--no-pdb
    Disable auto calling the pdb debugger after every exception.
--pprint
    Enable auto pretty printing of results.
--no-pprint
    Disable auto pretty printing of results.
--color-info
    IPython can display information about objects via a set of functions,
    and optionally can use colors for this, syntax highlighting
    source code and various other elements. This is on by default, but can cause
    problems with some pagers. If you see such problems, you can disable the
    colours.
--no-color-info
    Disable using colors for info related things.
--ignore-cwd
    Exclude the current working directory from sys.path
--no-ignore-cwd
    Include the current working directory in sys.path
--nosep
    Eliminate all spacing between prompts.
--matplotlib
    Configure matplotlib for interactive use with
    the default matplotlib backend.
--secure
    Use HMAC digests for authentication of messages.
    Setting this flag will generate a new UUID to use as the HMAC key.
--no-secure
    Don't authenticate messages.
--profile-dir=<Unicode> (ProfileDir.location)
    Default: ''
    Set the profile location directly. This overrides the logic used by the
    `profile` option.
--profile=<Unicode> (BaseIPythonApplication.profile)
    Default: 'default'
    The IPython profile to use.
--ipython-dir=<Unicode> (BaseIPythonApplication.ipython_dir)
    Default: ''
    The name of the IPython directory. This directory is used for logging
    configuration (through profiles), history storage, etc. The default is
    usually $HOME/.ipython. This option can also be specified through the
    environment variable IPYTHONDIR.
--log-level=<Enum> (Application.log_level)
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--config=<Unicode> (BaseIPythonApplication.extra_config_file)
    Default: ''
    Path to an extra config file to load.
    If specified, load this config file in addition to any other IPython config.
--ip=<Unicode> (IPKernelApp.ip)
    Default: ''
    Set the kernel's IP address [default localhost]. If the IP address is
    something other than localhost, then Consoles on other machines will be able
    to connect to the Kernel, so be careful!
--hb=<Int> (IPKernelApp.hb_port)
    Default: 0
    set the heartbeat port [default: random]
--shell=<Int> (IPKernelApp.shell_port)
    Default: 0
    set the shell (ROUTER) port [default: random]
--iopub=<Int> (IPKernelApp.iopub_port)
    Default: 0
    set the iopub (PUB) port [default: random]
--stdin=<Int> (IPKernelApp.stdin_port)
    Default: 0
    set the stdin (ROUTER) port [default: random]
--control=<Int> (IPKernelApp.control_port)
    Default: 0
    set the control (ROUTER) port [default: random]
-f <Unicode> (IPKernelApp.connection_file)
    Default: ''
    JSON file in which to store connection info [default: kernel-<pid>.json]
    This file will contain the IP, ports, and authentication key needed to
    connect clients to this kernel. By default, this file will be created in the
    security dir of the current profile, but can be specified by absolute path.
--transport=<CaselessStrEnum> (IPKernelApp.transport)
    Default: 'tcp'
    Choices: ['tcp', 'ipc']
--autocall=<Enum> (InteractiveShell.autocall)
    Default: 0
    Choices: (0, 1, 2)
    Make IPython automatically call any callable object even if you didn't type
    explicit parentheses. For example, 'str 43' becomes 'str(43)' automatically.
    The value can be '0' to disable the feature, '1' for 'smart' autocall, where
    it is not applied if there are no more arguments on the line, and '2' for
    'full' autocall, where all callable objects are automatically called (even
    if no arguments are present).
--colors=<CaselessStrEnum> (InteractiveShell.colors)
    Default: 'Neutral'
    Choices: ['Neutral', 'NoColor', 'LightBG', 'Linux']
    Set the color scheme (NoColor, Neutral, Linux, or LightBG).
--logfile=<Unicode> (InteractiveShell.logfile)
    Default: ''
    The name of the logfile to use.
--logappend=<Unicode> (InteractiveShell.logappend)
    Default: ''
    Start logging to the given file in append mode. Use `logfile` to specify a
    log file to **overwrite** logs to.
-c <Unicode> (InteractiveShellApp.code_to_run)
    Default: ''
    Execute the given command string.
-m <Unicode> (InteractiveShellApp.module_to_run)
    Default: ''
    Run the module as a script.
--ext=<Unicode> (InteractiveShellApp.extra_extension)
    Default: ''
    dotted module name of an IPython extension to load.
--gui=<CaselessStrEnum> (InteractiveShellApp.gui)
    Default: None
    Choices: ['asyncio', 'glut', 'gtk', 'gtk2', 'gtk3', 'osx', 'pyglet', 'qt', 'qt4', 'qt5', 'tk', 'wx', 'gtk2', 'qt4']
    Enable GUI event loop integration with any of ('asyncio', 'glut', 'gtk',
    'gtk2', 'gtk3', 'osx', 'pyglet', 'qt', 'qt4', 'qt5', 'tk', 'wx', 'gtk2',
    'qt4').
--pylab=<CaselessStrEnum> (InteractiveShellApp.pylab)
    Default: None
    Choices: ['auto', 'agg', 'gtk', 'gtk3', 'inline', 'ipympl', 'nbagg', 'notebook', 'osx', 'pdf', 'ps', 'qt', 'qt4', 'qt5', 'svg', 'tk', 'widget', 'wx']
    Pre-load matplotlib and numpy for interactive use, selecting a particular
    matplotlib backend and loop integration.
--matplotlib=<CaselessStrEnum> (InteractiveShellApp.matplotlib)
    Default: None
    Choices: ['auto', 'agg', 'gtk', 'gtk3', 'inline', 'ipympl', 'nbagg', 'notebook', 'osx', 'pdf', 'ps', 'qt', 'qt4', 'qt5', 'svg', 'tk', 'widget', 'wx']
    Configure matplotlib for interactive use with the default matplotlib
    backend.
--cache-size=<Int> (InteractiveShell.cache_size)
    Default: 1000
    Set the size of the output cache.  The default is 1000, you can change it
    permanently in your config file.  Setting it to 0 completely disables the
    caching system, and the minimum value accepted is 3 (if you provide a value
    less than 3, it is reset to 0 and a warning is issued).  This limit is
    defined because otherwise you'll spend more time re-flushing a too small
    cache than working
--ident=<CUnicode> (Session.session)
    Default: ''
    The UUID identifying this session.
--user=<Unicode> (Session.username)
    Default: 'username'
    Username for the Session. Default is your system username.
--keyfile=<Unicode> (Session.keyfile)
    Default: ''
    path to file containing execution key.

To see all available configurables, use `--help-all`
kevin-bates commented 4 years ago

Great news! The issue is that the --kernel option appears to be "flowing" from the launcher to the Matlab kernel and it doesn't recognize it as a parameter. Somehow it's picking up the various parameters that are passed to the launcher as if they are also passed to the Matlab kernel.

The only information that should "forward" to the kernel is the connection_file. Currently, all the other parameters apply to the launcher. You may want/need other parameters to "flow" in the future, so it looks like you need to figure out how to filter out parameters that only apply to the launcher vs. those that apply to the Matlab kernel (and probably python-related MetaKernels in general).

Once the parameter issue is resolved, this looks very promising. I'm fairly certain you should be able to troubleshoot the parameter issue by debugging the launch_ipykernel.py script directly and don't really need to go through the complete Kubernetes stuff.

I think if you invoke the launcher this way, it may be easier to troubleshoot from your desktop:

python launch_ipykernel.py --kernel=matlab --RemoteProcessProxy.cluster-type=none kernel-e974381e-a7b9-4497-a004-6b3af059e5e7.json

I'm attaching the dummy connection_file that I found on my system. It doesn't really matter the contents and with these parameters, the launcher will not try to contact an EG server, etc.

Note that Github doesn't allow JSON to be attached, so you can rename this kernel-e974381e-a7b9-4497-a004-6b3af059e5e7.txt with a .json extension, although that isn't really necessary in order to debug the parameter pass to the Matlab kernel.

lazywhite commented 4 years ago

@kevin-bates finally, It's working launcher.py

thx, you are my angle!

kevin-bates commented 4 years ago

Fantastic - this is great news - congratulations! Would you like to contribute your efforts to the project? I would be happy to guide you through incorporating your changes into the build such that the Matlab kernel is available in our k8s (and likely, docker) offerings.

lazywhite commented 4 years ago

I'm using a cracked version of matlab2018b, and I've hard coded volume mount part.

I think we should choose one type of "metakernel" like java kernel and publish the example code. so we can say EG support "metakernel" now.

maybe next week I'll find some time to do it.

kevin-bates commented 4 years ago

Ok. It looks like the java kernel is launched via python so as long this can be plugged into launch_ipykernel.py, we should be okay. Any required jar files would need to be appropriately baked into the image.

We should be able to "parameterize" or document the mount step. The kernel-pod.yaml template is fairly flexible now.

kevin-bates commented 2 years ago

Closing as it seems things ended on a positive note. We continue to look forward to any contributions, although that aspect of the landscape is changing with Kernel Provisioners.

SomePersonSomeWhereInTheWorld commented 6 months ago

@kevin-bates finally, It's working launcher.py

thx, you are my angle!

@kevin-bates finally, It's working launcher.py

@kevin-bates finally, It's working launcher.py

There is a 404 error not found has the file been moved?

SomePersonSomeWhereInTheWorld commented 6 months ago

@kevin-bates I tried as you said, here is launch_ipykernel.py I add an argument in line 384, now I can boot a pod with this kernel through jupyter-gateway

but top right of notebook page show kernel status, it's always a big black dot, not white annular. and it can't process phrase like

a = -10:0.01:10
b = sin(a)
plot(b)

please help me to figure out the wrong code.

404 error on the link is it available?