search

jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
615 stars 221 forks source link

Enterprise gateway Spark Operator (Python) kernel fails to systematic restart #1258

Closed vivekbhide closed 9 months ago

vivekbhide commented 1 year ago

Details - When using the Spark Operator (Python) kernel, the kernel restart fails on selecting any of the 'Restart kernel ...' option from notebook file menu

Enterprise gateway version tested - 3.2.0

Steps to reproduce -

  1. Create a spark notebook with enterprise gateway managing the kernels to run with
  2. Run the notebook with kernel 'Spark Operator (Python)'.. Basic operation of creating spark session (hence creating spark driver and executor pods behind the scene) with some spark read files or spark sql are expected
  3. Go to Kernel menu on jupyter lab. Choose any 'Restart Kernel...' option

Expected Result -

  1. Kernel restarts with a. Any executor pod getting terminated b. Any driver prod restarting and going to 'Running' phase

Actual Results -

  1. Driver and executor pods terminates but driver pod doesn't restart.
  2. Enterprise gateway goes into infinite loop of checking the driver status, followed by erroring with 'Kernel launch timeout' exception and continuing status on the driver pod
  3. Jupyter lab UI, stuck with busy kernel status

Observation -

  1. There are couple of issues reported with spark operator regarding issues with delete and/or restart of spark application. It might be contributing to this behavior https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1566 and https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1558
  2. Issue is not always reproducible but the probability of it occurring increases when you not only start a kernel and try to restart but also run some spark operations and then try to restart

Logs - Enterprise gateway logs -

[D 2023-02-21 00:33:23.314 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:23.813 EnterpriseGatewayApp] Signal (2) sent via gateway communication port.
[D 2023-02-21 00:33:23.814 EnterpriseGatewayApp] Shutdown request sent to listener via gateway communication port.
[I 2023-02-21 00:33:23.836 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status as per delete_namespaced_custom_object -> {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Success', 'details': {'name': 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a', 'group': 'sparkoperator.k8s.io', 'kind': 'sparkapplications', 'uid': '5403dbce-b485-4284-b2ed-1fa39dbefc5e'}}
[D 2023-02-21 00:33:23.836 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status as per delete_managed_object -> True
[D 2023-02-21 00:33:23.836 EnterpriseGatewayApp] KubernetesProcessProxy.terminate_container_resources, SparkApplication: notebook.jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver, kernel ID: 72252bdb-da19-42b0-8aed-61fd424b513a has been terminated.
[D 2023-02-21 00:33:23.858 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:23.864 EnterpriseGatewayApp] activity on 72252bdb-da19-42b0-8aed-61fd424b513a: status (busy)
[D 2023-02-21 00:33:23.984 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:24.108 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:24.234 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[I 230221 00:33:24 web:2239] 200 GET /api/kernels (xx.xx.xx.xx) 5.75ms
[I 230221 00:33:24 web:2239] 200 GET /api/kernels/72252bdb-da19-42b0-8aed-61fd424b513a (xx.xx.xx.xx) 5.56ms
[D 2023-02-21 00:33:24.372 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:24.502 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:24.627 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:24.754 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:24.892 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:25.022 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:25.153 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:25.275 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:25.404 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Running'
[D 2023-02-21 00:33:25.530 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:25.566 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:25.595 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:25.596 EnterpriseGatewayApp] Instantiating kernel 'Spark Operator (Python)' with process proxy: enterprise_gateway.services.processproxies.spark_operator.SparkOperatorProcessProxy
[D 2023-02-21 00:33:25.596 EnterpriseGatewayApp] Starting kernel (async): ['/opt/conda/bin/python3.9', '/usr/local/share/jupyter/kernels/spark_python_operator/scripts/launch_custom_resource.py', '--RemoteProcessProxy.kernel-id', '72252bdb-da19-42b0-8aed-61fd424b513a', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', 'xx.xx.xx.xx:8877', '--RemoteProcessProxy.public-key', 'public-key']
[D 2023-02-21 00:33:25.596 EnterpriseGatewayApp] Launching kernel: 'Spark Operator (Python)' with command: ['/opt/conda/bin/python3.9', '/usr/local/share/jupyter/kernels/spark_python_operator/scripts/launch_custom_resource.py', '--RemoteProcessProxy.kernel-id', '72252bdb-da19-42b0-8aed-61fd424b513a', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', 'xx.xx.xx.xx:8877', '--RemoteProcessProxy.public-key', 'public-key']
[W 2023-02-21 00:33:25.596 EnterpriseGatewayApp] Shared namespace has been configured.  All kernels will reside in EG namespace: notebook
[D 2023-02-21 00:33:25.596 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'KERNEL_USERNAME': 'jovyan', 'KERNEL_LAUNCH_TIMEOUT': '300', 'KERNEL_GATEWAY': '1', 'KERNEL_POD_NAME': 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a', 'KERNEL_RESOURCE_NAME': 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a', 'KERNEL_CRD_GROUP': 'sparkoperator.k8s.io', 'KERNEL_CRD_VERSION': 'v1beta2', 'KERNEL_CRD_PLURAL': 'sparkapplications', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'KUBERNETES_SERVICE_PORT': '443', 'KUBERNETES_SERVICE_HOST': 'xx.xx.xx.xx', 'KERNEL_SERVICE_ACCOUNT_NAME': 'enterprise-gateway-sa', 'KERNEL_NAMESPACE': 'notebook', 'KERNEL_IMAGE': '<kernel_image_docker_path>', 'KERNEL_EXECUTOR_IMAGE': '<kernel_image_docker_path>', 'KERNEL_UID': '1000', 'KERNEL_GID': '100', 'EG_MIN_PORT_RANGE_SIZE': '1000', 'EG_MAX_PORT_RANGE_RETRIES': '5', 'KERNEL_ID': '72252bdb-da19-42b0-8aed-61fd424b513a', 'KERNEL_LANGUAGE': 'python', 'EG_IMPERSONATION_ENABLED': 'False'}
[I 2023-02-21 00:33:25.600 EnterpriseGatewayApp] SparkOperatorProcessProxy: kernel launched. Kernel image: <kernel_image_docker_path>, KernelID: 72252bdb-da19-42b0-8aed-61fd424b513a, cmd: '['/opt/conda/bin/python3.9', '/usr/local/share/jupyter/kernels/spark_python_operator/scripts/launch_custom_resource.py', '--RemoteProcessProxy.kernel-id', '72252bdb-da19-42b0-8aed-61fd424b513a', '--RemoteProcessProxy.port-range', '0..0', '--RemoteProcessProxy.response-address', 'xx.xx.xx.xx:8877', '--RemoteProcessProxy.public-key', 'public-key']'
[D 2023-02-21 00:33:26.129 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:26.129 EnterpriseGatewayApp] 1: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:26.657 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:26.657 EnterpriseGatewayApp] 2: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:27.186 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:27.186 EnterpriseGatewayApp] 3: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:27.716 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:27.716 EnterpriseGatewayApp] 4: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:28.244 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:28.244 EnterpriseGatewayApp] 5: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:28.768 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:28.768 EnterpriseGatewayApp] 6: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:29.296 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:29.296 EnterpriseGatewayApp] 7: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:29.822 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:29.822 EnterpriseGatewayApp] 8: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:30.348 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:30.349 EnterpriseGatewayApp] 9: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:30.878 EnterpriseGatewayApp] container jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver status in get_container_status -> 'Failed'
[D 2023-02-21 00:33:30.878 EnterpriseGatewayApp] 10: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'Failed', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:31.398 EnterpriseGatewayApp] 11: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:31.915 EnterpriseGatewayApp] 12: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:32.437 EnterpriseGatewayApp] 13: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:32.958 EnterpriseGatewayApp] 14: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:33.477 EnterpriseGatewayApp] 15: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:34.001 EnterpriseGatewayApp] 16: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[I 230221 00:33:34 web:2239] 200 GET /api/kernels (xx.xx.xx.xx) 6.25ms
[I 230221 00:33:34 web:2239] 200 GET /api/kernels/72252bdb-da19-42b0-8aed-61fd424b513a (xx.xx.xx.xx) 5.63ms
[D 2023-02-21 00:33:34.524 EnterpriseGatewayApp] 17: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:35.047 EnterpriseGatewayApp] 18: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:35.571 EnterpriseGatewayApp] 19: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:36.093 EnterpriseGatewayApp] 20: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:36.615 EnterpriseGatewayApp] 21: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:37.140 EnterpriseGatewayApp] 22: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:37.664 EnterpriseGatewayApp] 23: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:38.186 EnterpriseGatewayApp] 24: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:38.710 EnterpriseGatewayApp] 25: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:39.230 EnterpriseGatewayApp] 26: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:39.753 EnterpriseGatewayApp] 27: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:40.274 EnterpriseGatewayApp] 28: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:40.796 EnterpriseGatewayApp] 29: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:41.318 EnterpriseGatewayApp] 30: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:41.837 EnterpriseGatewayApp] 31: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:42.359 EnterpriseGatewayApp] 32: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:42.880 EnterpriseGatewayApp] 33: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:43.401 EnterpriseGatewayApp] 34: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:43.918 EnterpriseGatewayApp] 35: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:44.437 EnterpriseGatewayApp] 36: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[I 230221 00:33:44 web:2239] 200 GET /api/kernels (xx.xx.xx.xx) 6.06ms
[I 230221 00:33:44 web:2239] 200 GET /api/kernels/72252bdb-da19-42b0-8aed-61fd424b513a (xx.xx.xx.xx) 6.03ms
[D 2023-02-21 00:33:44.961 EnterpriseGatewayApp] 37: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:45.484 EnterpriseGatewayApp] 38: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:46.008 EnterpriseGatewayApp] 39: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:46.534 EnterpriseGatewayApp] 40: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:47.053 EnterpriseGatewayApp] 41: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:47.578 EnterpriseGatewayApp] 42: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:48.103 EnterpriseGatewayApp] 43: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:48.626 EnterpriseGatewayApp] 44: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:49.148 EnterpriseGatewayApp] 45: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:49.672 EnterpriseGatewayApp] 46: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:49.770 EnterpriseGatewayApp] Polling every 60 seconds for kernels idle > 172800 seconds...
[D 2023-02-21 00:33:49.771 EnterpriseGatewayApp] kernel_id=72252bdb-da19-42b0-8aed-61fd424b513a, kernel_name=spark_python_operator, last_activity=2023-02-21 00:33:23.864032+00:00
[D 2023-02-21 00:33:50.193 EnterpriseGatewayApp] 47: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:50.715 EnterpriseGatewayApp] 48: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:51.233 EnterpriseGatewayApp] 49: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:51.753 EnterpriseGatewayApp] 50: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:52.276 EnterpriseGatewayApp] 51: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:52.795 EnterpriseGatewayApp] 52: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:53.315 EnterpriseGatewayApp] 53: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:53.842 EnterpriseGatewayApp] 54: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:54.365 EnterpriseGatewayApp] 55: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[I 230221 00:33:54 web:2239] 200 GET /api/kernels (xx.xx.xx.xx) 6.32ms
[I 230221 00:33:54 web:2239] 200 GET /api/kernels/72252bdb-da19-42b0-8aed-61fd424b513a (xx.xx.xx.xx) 6.21ms
[D 2023-02-21 00:33:54.889 EnterpriseGatewayApp] 56: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:55.412 EnterpriseGatewayApp] 57: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:55.932 EnterpriseGatewayApp] 58: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:56.449 EnterpriseGatewayApp] 59: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:56.972 EnterpriseGatewayApp] 60: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:57.494 EnterpriseGatewayApp] 61: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:58.016 EnterpriseGatewayApp] 62: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:58.539 EnterpriseGatewayApp] 63: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:59.062 EnterpriseGatewayApp] 64: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:33:59.585 EnterpriseGatewayApp] 65: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:00.109 EnterpriseGatewayApp] 66: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:00.633 EnterpriseGatewayApp] 67: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:01.154 EnterpriseGatewayApp] 68: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:01.686 EnterpriseGatewayApp] 69: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:02.205 EnterpriseGatewayApp] 70: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:03.445 EnterpriseGatewayApp] 71: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:03.965 EnterpriseGatewayApp] 72: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[D 2023-02-21 00:34:04.486 EnterpriseGatewayApp] 73: Waiting to connect to k8s sparkapplication in namespace 'notebook'. Name: 'jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver', Status: 'None', Pod IP: 'None', KernelID: '72252bdb-da19-42b0-8aed-61fd424b513a'
[I 230221 00:34:04 web:2239] 200 GET /api/kernels/72252bdb-da19-42b0-8aed-61fd424b513a (xx.xx.xx.xx) 5.90ms
[I 230221 00:34:04 web:2239] 200 GET /api/kernels (xx.xx.xx.xx) 5.91ms
`

Spark operator logs -

`I0221 00:33:23.837600      10 metrics.go:125] Decrementing spark_app_running_count with labels map[app_type:Unknown] metricVal to 0
I0221 00:33:23.837622      10 sparkapp_metrics.go:183] Application is deleted. Decreasing Running Count for Executor jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-1.
I0221 00:33:23.837627      10 metrics.go:125] Decrementing spark_app_executor_running_count with labels map[app_type:Unknown] metricVal to 3
I0221 00:33:23.837631      10 sparkapp_metrics.go:183] Application is deleted. Decreasing Running Count for Executor jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-2.
I0221 00:33:23.837634      10 metrics.go:125] Decrementing spark_app_executor_running_count with labels map[app_type:Unknown] metricVal to 2
I0221 00:33:23.837639      10 controller.go:873] Deleting pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver in namespace notebook
I0221 00:33:23.857191      10 event.go:282] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"notebook", Name:"jovyan-72252bdb-da19-42b0-8aed-61fd424b513a", UID:"5403dbce-b485-4284-b2ed-1fa39dbefc5e", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"406292198", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationDeleted' SparkApplication jovyan-72252bdb-da19-42b0-8aed-61fd424b513a was deleted
I0221 00:33:23.861644      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver updated in namespace notebook.
I0221 00:33:23.994753      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-1 updated in namespace notebook.
I0221 00:33:24.055212      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-2 updated in namespace notebook.
I0221 00:33:24.358846      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-2 updated in namespace notebook.
I0221 00:33:24.665771      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-1 updated in namespace notebook.
I0221 00:33:25.406640      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver updated in namespace notebook.
I0221 00:33:25.444782      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-2 updated in namespace notebook.
I0221 00:33:25.454333      10 spark_pod_eventhandler.go:77] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-2 deleted in namespace notebook.
I0221 00:33:25.706927      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-1 updated in namespace notebook.
I0221 00:33:25.728818      10 spark_pod_eventhandler.go:77] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-032ba38671626041-exec-1 deleted in namespace notebook.
I0221 00:33:25.932676      10 controller.go:188] SparkApplication notebook/jovyan-72252bdb-da19-42b0-8aed-61fd424b513a was added, enqueuing it for submission
I0221 00:33:25.932700      10 controller.go:267] Starting processing key: "notebook/jovyan-72252bdb-da19-42b0-8aed-61fd424b513a"
I0221 00:33:25.932749      10 sparkui.go:288] Creating a service jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-ui-svc for the Spark UI for application jovyan-72252bdb-da19-42b0-8aed-61fd424b513a
I0221 00:33:25.932940      10 event.go:282] Event(v1.ObjectReference{Kind:"SparkApplication", Namespace:"notebook", Name:"jovyan-72252bdb-da19-42b0-8aed-61fd424b513a", UID:"1b586c07-1102-4b75-9692-55d5df5e256f", APIVersion:"sparkoperator.k8s.io/v1beta2", ResourceVersion:"406292242", FieldPath:""}): type: 'Normal' reason: 'SparkApplicationAdded' SparkApplication jovyan-72252bdb-da19-42b0-8aed-61fd424b513a was added, enqueuing it for submission
E0221 00:33:26.013256      10 controller.go:686] failed to create UI service for SparkApplication notebook/jovyan-72252bdb-da19-42b0-8aed-61fd424b513a: services "jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-ui-svc" already exists
I0221 00:33:26.013316      10 submission.go:65] spark-submit arguments: [/opt/spark/bin/spark-submit --master k8s://https://xx.xx.xx.xx:443 --deploy-mode cluster --conf spark.kubernetes.namespace=notebook --conf spark.app.name=jovyan-72252bdb-da19-42b0-8aed-61fd424b513a --conf spark.kubernetes.driver.pod.name=jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver --conf spark.kubernetes.container.image=<kernel_image_docker_path> --conf spark.kubernetes.container.image.pullPolicy=Always --conf spark.kubernetes.pyspark.pythonVersion=3 --conf spark.kubernetes.submission.waitAppCompletion=false --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/app-name=jovyan-72252bdb-da19-42b0-8aed-61fd424b513a --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.driver.label.sparkoperator.k8s.io/submission-id=7942f7e8-957e-472a-b761-6db1bf1d1888 --conf spark.driver.cores=1 --conf spark.driver.memory=2g --conf spark.kubernetes.authenticate.driver.serviceAccountName=enterprise-gateway-sa --conf spark.driver.extraJavaOptions=-Djava.security.krb5.conf=/secrets/conf/krb5.conf --conf spark.kubernetes.driver.label.kernel_id=72252bdb-da19-42b0-8aed-61fd424b513a --conf spark.kubernetes.driver.label.component=kernel --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/app-name=jovyan-72252bdb-da19-42b0-8aed-61fd424b513a --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/launched-by-spark-operator=true --conf spark.kubernetes.executor.label.sparkoperator.k8s.io/submission-id=7942f7e8-957e-472a-b761-6db1bf1d1888 --conf spark.executor.instances=2 --conf spark.kubernetes.executor.container.image=<kernel_image_docker_path> --conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.kubernetes.authenticate.executor.serviceAccountName=enterprise-gateway-sa --conf spark.kubernetes.executor.deleteOnTermination=true --conf spark.executor.extraJavaOptions=-Djava.security.krb5.conf=/secrets/conf/krb5.conf local:///usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 72252bdb-da19-42b0-8aed-61fd424b513a --spark-context-initialization-mode none --response-address xx.xx.xx.xx:8877 --port-range 0..0 --public-key public-key]
I0221 00:33:28.104116      10 webhook.go:247] Serving admission request
I0221 00:33:28.104765      10 webhook.go:568] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver in namespace notebook is subject to mutation
I0221 00:33:28.213514      10 webhook.go:247] Serving admission request
I0221 00:33:28.213939      10 webhook.go:568] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver in namespace notebook is subject to mutation
W0221 00:33:28.295791      10 submission.go:75] trying to resubmit an already submitted SparkApplication notebook/jovyan-72252bdb-da19-42b0-8aed-61fd424b513a
I0221 00:33:28.295828      10 controller.go:274] Ending processing key: "notebook/jovyan-72252bdb-da19-42b0-8aed-61fd424b513a"
I0221 00:33:30.953371      10 spark_pod_eventhandler.go:58] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver updated in namespace notebook.
I0221 00:33:30.985616      10 spark_pod_eventhandler.go:77] Pod jovyan-72252bdb-da19-42b0-8aed-61fd424b513a-driver deleted in namespace notebook.
kevin-bates commented 1 year ago

I'm not able to reproduce this, but it sounds like @lresende can per a side conversation we had.

lresende commented 1 year ago

@vivekbhide please give it a try on latest code in the main branch and let me know if things are better for you.

lresende commented 9 months ago

Closing this as I believe this has been resolved. @vivekbhide please re-open if you still see the issues