search

skai-x / elastic-jupyter-operator

Cloud-native way to provide elastic Jupyter Notebooks on Kubernetes. Run remote kernels, natively.
Apache License 2.0
197 stars 29 forks source link

[feat] Design Kernel CRD to ease the maintanance #4

Closed gaocegege closed 3 years ago

gaocegege commented 3 years ago

The Kernel CRD is used to:

gaocegege commented 3 years ago 
[D 2021-04-19 02:56:17.160 EnterpriseGatewayApp] Starting kernel (async): ['/opt/conda/bin/python3.7', '/usr/local/share/jupyter/kernels/python_kubernetes/scripts/launch_kubernetes.py', '--RemoteProcessProxy.kernel-id', 'c2755aa3-2cde-4a29-be65-c92d3ea8888f', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', '10.244.0.5:48791']
[D 2021-04-19 02:56:17.160 EnterpriseGatewayApp] Launching kernel: 'Python on Kubernetes' with command: ['/opt/conda/bin/python3.7', '/usr/local/share/jupyter/kernels/python_kubernetes/scripts/launch_kubernetes.py', '--RemoteProcessProxy.kernel-id', 'c2755aa3-2cde-4a29-be65-c92d3ea8888f', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', '10.244.0.5:48791']
[W 2021-04-19 02:56:17.161 EnterpriseGatewayApp] Shared namespace has been configured.  All kernels will reside in EG namespace: default
[D 2021-04-19 02:56:17.161 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'SHELL': '/bin/bash', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'EG_MIRROR_WORKING_DIRS': 'false', 'JUPYTERGATEWAY_SAMPLE_PORT_8888_TCP': 'tcp://10.96.228.154:8888', 'KUBERNETES_SERVICE_PORT': '443', 'EG_NAMESPACE': 'default', 'JUPYTERGATEWAY_SAMPLE_SERVICE_PORT_GATEWAY': '8888', 'JUPYTERGATEWAY_SAMPLE_PORT_8888_TCP_ADDR': '10.96.228.154', 'HOSTNAME': 'jupytergateway-sample-64966d448b-fccmt', 'LANGUAGE': 'en_US.UTF-8', 'EG_SHARED_NAMESPACE': 'true', 'EG_PORT': '8888', 'EG_LOG_LEVEL': 'DEBUG', 'JAVA_HOME': '/usr/lib/jvm/java-8-openjdk-amd64', 'EG_KERNEL_WHITELIST': "'r_kubernetes','python_kubernetes','python_tf_kubernetes','python_tf_gpu_kubernetes','scala_kubernetes','spark_r_kubernetes','spark_python_kubernetes','spark_scala_kubernetes'", 'NB_UID': '1000', 'JUPYTERGATEWAY_SAMPLE_PORT_8888_TCP_PORT': '8888', 'PWD': '/usr/local/bin', 'EG_CULL_IDLE_TIMEOUT': '3600', 'EG_DEFAULT_KERNEL_NAME': 'python_kubernetes', 'MINICONDA_MD5': '87e77f097f6ebb5127c77662dfc3165e', 'EG_ENABLE_TUNNELING': 'False', 'EG_KERNEL_LAUNCH_TIMEOUT': '60', 'HOME': '/home/jovyan', 'LANG': 'en_US.UTF-8', 'KUBERNETES_PORT_443_TCP': 'tcp://10.96.0.1:443', 'EG_LIST_KERNELS': 'True', 'EG_SSH_PORT': '2122', 'NB_GID': '100', 'KG_PORT': '8888', 'EG_CULL_CONNECTED': 'False', 'EG_PORT_RETRIES': '0', 'KG_IP': '0.0.0.0', 'EG_PORT_RANGE': '0..0', 'EG_CULL_INTERVAL': '60', 'JUPYTERGATEWAY_SAMPLE_SERVICE_HOST': '10.96.228.154', 'EG_IP': '0.0.0.0', 'SHLVL': '0', 'CONDA_DIR': '/opt/conda', 'MINICONDA_VERSION': '4.8.2', 'SPARK_HOME': '/opt/spark', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KG_PORT_RETRIES': '0', 'KUBERNETES_PORT_443_TCP_ADDR': '10.96.0.1', 'JUPYTERGATEWAY_SAMPLE_PORT_8888_TCP_PROTO': 'tcp', 'SPARK_VER': '2.4.6', 'JUPYTERGATEWAY_SAMPLE_PORT': 'tcp://10.96.228.154:8888', 'JUPYTERGATEWAY_SAMPLE_SERVICE_PORT': '8888', 'EG_KERNEL_IMAGE': 'ccr.ccs.tencentyun.com/kubeflow-oteam/jupyter-kernel-py:2.5.0', 'CONDA_VERSION': '4.8.2', 'NB_USER': 'jovyan', 'KUBERNETES_SERVICE_HOST': '10.96.0.1', 'LC_ALL': 'en_US.UTF-8', 'KUBERNETES_PORT': 'tcp://10.96.0.1:443', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'EG_KERNEL_CLUSTER_ROLE': 'enterprise-gateway-controller', 'DEBIAN_FRONTEND': 'noninteractive', 'KERNEL_LAUNCH_TIMEOUT': '40', 'KERNEL_USERNAME': 'jovyan', 'KERNEL_GATEWAY': '1', 'KERNEL_POD_NAME': 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f', 'KERNEL_SERVICE_ACCOUNT_NAME': 'default', 'KERNEL_NAMESPACE': 'default', 'KERNEL_IMAGE': 'elyra/kernel-py:2.5.0', 'KERNEL_EXECUTOR_IMAGE': 'elyra/kernel-py:2.5.0', 'KERNEL_UID': '1000', 'KERNEL_GID': '100', 'EG_MIN_PORT_RANGE_SIZE': '1000', 'EG_MAX_PORT_RANGE_RETRIES': '5', 'KERNEL_ID': 'c2755aa3-2cde-4a29-be65-c92d3ea8888f', 'KERNEL_LANGUAGE': 'python', 'EG_IMPERSONATION_ENABLED': 'False'}
[I 2021-04-19 02:56:17.218 EnterpriseGatewayApp] KubernetesProcessProxy: kernel launched. Kernel image: elyra/kernel-py:2.5.0, KernelID: c2755aa3-2cde-4a29-be65-c92d3ea8888f, cmd: '['/opt/conda/bin/python3.7', '/usr/local/share/jupyter/kernels/python_kubernetes/scripts/launch_kubernetes.py', '--RemoteProcessProxy.kernel-id', 'c2755aa3-2cde-4a29-be65-c92d3ea8888f', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', '10.244.0.5:48791']'
[D 2021-04-19 02:56:17.742 EnterpriseGatewayApp] 1: Waiting to connect to k8s pod in namespace 'default'. Name: '', Status: 'None', Pod IP: 'None', KernelID: 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:18.331 EnterpriseGatewayApp] 2: Waiting to connect to k8s pod in namespace 'default'. Name: '', Status: 'None', Pod IP: 'None', KernelID: 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:18.863 EnterpriseGatewayApp] 3: Waiting to connect to k8s pod in namespace 'default'. Name: 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f', Status: 'Pending', Pod IP: 'None', KernelID: 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:19.377 EnterpriseGatewayApp] 4: Waiting to connect to k8s pod in namespace 'default'. Name: 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f', Status: 'Pending', Pod IP: 'None', KernelID: 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:19.888 EnterpriseGatewayApp] 5: Waiting to connect to k8s pod in namespace 'default'. Name: 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f', Status: 'Pending', Pod IP: 'None', KernelID: 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:20.400 EnterpriseGatewayApp] 6: Waiting to connect to k8s pod in namespace 'default'. Name: 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f', Status: 'Running', Pod IP: '10.244.0.9', KernelID: 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:20.402 EnterpriseGatewayApp] Received Payload 'XeX9k3dnzdImPV2W76ema+foB3/AKwJBkInjm0mUb8BwDx7D0dn9fmmZIgC9jqLaEt3SnpZuJS1Ihq9AvU2A7fasatXUyQFm3X7a5suvzMVw8cNx6NXdXnko7mlpNHd9daEuCi9U5t94LIo3OLBzzj8BDEESCPrS/qcws47m48PdXe6uhL+OPpYXno1QDoZuuPlUFuT5l1YC/FdCbEpcmusmhfpUWmMIQ5kHxAj/KWdwf6ZAliBIndRC4x8tDsxm3YQF9zLewjEPW/EzhOjk2xh7IVzbyqu/E8g2k/c60tu5Iv2LKpJ/evqPE9ZYWa3IyfguoxEpppcFhma5i7TUP2qf8wnA7QCzOUfxVViCpkAgBbmgQbeQeaZubhUtr+/BHYle4QeOPwABfjAQVyXESA=='
[D 2021-04-19 02:56:20.420 EnterpriseGatewayApp] Decrypted Payload '{"shell_port": 44237, "iopub_port": 56865, "stdin_port": 38663, "control_port": 56197, "hb_port": 38273, "ip": "0.0.0.0", "key": "6c3079ea-6b68-4f4e-b8c9-5dc4eec13bb9", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": "10", "pgid": "8", "comm_port": 60213}'
[D 2021-04-19 02:56:20.420 EnterpriseGatewayApp] Connect Info received from the launcher is as follows '{'shell_port': 44237, 'iopub_port': 56865, 'stdin_port': 38663, 'control_port': 56197, 'hb_port': 38273, 'ip': '0.0.0.0', 'key': '6c3079ea-6b68-4f4e-b8c9-5dc4eec13bb9', 'transport': 'tcp', 'signature_scheme': 'hmac-sha256', 'kernel_name': '', 'pid': '10', 'pgid': '8', 'comm_port': 60213}'
[D 2021-04-19 02:56:20.420 EnterpriseGatewayApp] Host assigned to the Kernel is: 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f' '10.244.0.9'
[D 2021-04-19 02:56:20.420 EnterpriseGatewayApp] Established gateway communication to: 10.244.0.9:60213 for KernelID 'c2755aa3-2cde-4a29-be65-c92d3ea8888f'
[D 2021-04-19 02:56:20.444 EnterpriseGatewayApp] Received connection info for KernelID 'c2755aa3-2cde-4a29-be65-c92d3ea8888f' from host 'jovyan-c2755aa3-2cde-4a29-be65-c92d3ea8888f': {'shell_port': 44237, 'iopub_port': 56865, 'stdin_port': 38663, 'control_port': 56197, 'hb_port': 38273, 'ip': '10.244.0.9', 'key': '6c3079ea-6b68-4f4e-b8c9-5dc4eec13bb9', 'transport': 'tcp', 'signature_scheme': 'hmac-sha256', 'kernel_name': '', 'comm_port': 60213}...
[D 2021-04-19 02:56:20.444 EnterpriseGatewayApp] response socket still open, close it
[D 2021-04-19 02:56:20.451 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:56197
[D 2021-04-19 02:56:20.463 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:56865
[I 2021-04-19 02:56:20.468 EnterpriseGatewayApp] Kernel started: c2755aa3-2cde-4a29-be65-c92d3ea8888f, name: python_kubernetes
[D 2021-04-19 02:56:20.468 EnterpriseGatewayApp] Kernel args: {'env': {'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'KERNEL_LAUNCH_TIMEOUT': '40', 'KERNEL_WORKING_DIR': '/home/jovyan/work', 'KERNEL_USERNAME': 'jovyan'}, 'kernel_headers': {}, 'kernel_name': 'python_kubernetes'}
[I 2021-04-19 02:56:20.478 EnterpriseGatewayApp] Culling kernels with idle durations > 3600 seconds at 60 second intervals ...
[I 210419 02:56:20 web:2250] 201 POST /api/kernels (10.244.0.4) 5077.34ms
[I 210419 02:56:20 web:2250] 200 GET /api/kernels/c2755aa3-2cde-4a29-be65-c92d3ea8888f (10.244.0.4) 8.54ms
[I 210419 02:56:20 web:2250] 200 GET /api/kernels/c2755aa3-2cde-4a29-be65-c92d3ea8888f (10.244.0.4) 0.66ms
[D 2021-04-19 02:56:20.527 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (starting)
[D 2021-04-19 02:56:20.551 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/c2755aa3-2cde-4a29-be65-c92d3ea8888f/channels
[W 2021-04-19 02:56:20.554 EnterpriseGatewayApp] No session ID specified
[D 2021-04-19 02:56:20.555 EnterpriseGatewayApp] Requesting kernel info from c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.555 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:44237
[D 2021-04-19 02:56:20.562 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (busy)
[D 2021-04-19 02:56:20.563 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (idle)
[D 2021-04-19 02:56:20.564 EnterpriseGatewayApp] Received kernel info: {'status': 'ok', 'protocol_version': '5.3', 'implementation': 'ipython', 'implementation_version': '7.19.0', 'language_info': {'name': 'python', 'version': '3.7.6', 'mimetype': 'text/x-python', 'codemirror_mode': {'name': 'ipython', 'version': 3}, 'pygments_lexer': 'ipython3', 'nbconvert_exporter': 'python', 'file_extension': '.py'}, 'banner': "Python 3.7.6 | packaged by conda-forge | (default, Jun  1 2020, 18:57:50) \nType 'copyright', 'credits' or 'license' for more information\nIPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.\n", 'help_links': [{'text': 'Python Reference', 'url': 'https://docs.python.org/3.7'}, {'text': 'IPython Reference', 'url': 'https://ipython.org/documentation.html'}, {'text': 'NumPy Reference', 'url': 'https://docs.scipy.org/doc/numpy/reference/'}, {'text': 'SciPy Reference', 'url': 'https://docs.scipy.org/doc/scipy/reference/'}, {'text': 'Matplotlib Reference', 'url': 'https://matplotlib.org/contents.html'}, {'text': 'SymPy Reference', 'url': 'http://docs.sympy.org/latest/index.html'}, {'text': 'pandas Reference', 'url': 'https://pandas.pydata.org/pandas-docs/stable/'}]}
[I 210419 02:56:20 web:2250] 101 GET /api/kernels/c2755aa3-2cde-4a29-be65-c92d3ea8888f/channels (10.244.0.4) 25.77ms
[D 2021-04-19 02:56:20.574 EnterpriseGatewayApp] Opening websocket /api/kernels/c2755aa3-2cde-4a29-be65-c92d3ea8888f/channels
[D 2021-04-19 02:56:20.574 EnterpriseGatewayApp] Getting buffer for c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.575 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:56865
[D 2021-04-19 02:56:20.575 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:44237
[D 2021-04-19 02:56:20.576 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:56197
[D 2021-04-19 02:56:20.577 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:38663
[D 2021-04-19 02:56:20.577 EnterpriseGatewayApp] Connecting to: tcp://10.244.0.9:44237
[D 2021-04-19 02:56:20.579 EnterpriseGatewayApp] Nudge: attempt 1 on kernel c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.581 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (busy)
[D 2021-04-19 02:56:20.581 EnterpriseGatewayApp] Nudge: IOPub received: c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.582 EnterpriseGatewayApp] Nudge: resolving iopub future: c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.582 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (idle)
[D 2021-04-19 02:56:20.582 EnterpriseGatewayApp] Nudge: shell info reply received: c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.583 EnterpriseGatewayApp] Nudge: resolving shell future: c2755aa3-2cde-4a29-be65-c92d3ea8888f
[D 2021-04-19 02:56:20.588 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (busy)
[D 2021-04-19 02:56:20.590 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (idle)
[D 2021-04-19 02:56:21.120 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (busy)
[D 2021-04-19 02:56:21.122 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: execute_input
[D 2021-04-19 02:56:21.127 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: stream
[D 2021-04-19 02:56:21.129 EnterpriseGatewayApp] activity on c2755aa3-2cde-4a29-be65-c92d3ea8888f: status (idle)
[D 2021-04-19 02:56:24.965 EnterpriseGatewayApp] Websocket closed c2755aa3-2cde-4a29-be65-c92d3ea8888f:a453d2bc-168cb4491fe583c2dc5a2a96
[I 2021-04-19 02:56:24.965 EnterpriseGatewayApp] Starting buffering for c2755aa3-2cde-4a29-be65-c92d3ea8888f:a453d2bc-168cb4491fe583c2dc5a2a96
[D 2021-04-19 02:56:24.965 EnterpriseGatewayApp] Clearing buffer for c2755aa3-2cde-4a29-be65-c92d3ea8888f