search

jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
623 stars 222 forks source link

enterprise-gateway does not connect to k8s kernel when istio is configured #1168

Open tafaust opened 2 years ago

tafaust commented 2 years ago

Hello enterprise-gateway team!

I checked all other issues in the jupyter-server organization and googled a lot without luck yet.

Description

I am trying to run enterprise-gateway in my kubernetes cluster to be able to run remote kernels. I installed enterprise-gateway via helm with the chart from this repository. I extended the helm chart with my own helm chart which does install a namespace with istio labels before installing enterprise-gateway. My values-override.yaml looks as follows:

# https://jupyter-enterprise-gateway.readthedocs.io/en/latest/operators/deploy-kubernetes.html#helm-configuration-parameters
enterprise-gateway:
  # Enterprise Gateway image name and tag to use.
  image: elyra/enterprise-gateway:dev
  # Enterprise Gateway image pull policy.
  imagePullPolicy: IfNotPresent
  # Log output level.
  logLevel: DEBUG
  # Whether to mirror working directories.
  mirrorWorkingDirs: false
  # Optional authorization token passed in all requests (see --EnterpriseGatewayApp.auth_token)
  authToken: ""

  # K8s Enterprise Gateway Service
  service:
    type: "ClusterIP"
    ports:
      # The primary port on which Enterprise Gateway is servicing requests.
      - name: "http"
        port: 8888
        targetPort: 8888
      # The port on which Enterprise Gateway will receive kernel connection info responses.
      - name: "http-response"
        port: 8877
        targetPort: 8877

  # https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/kubernetes/helm/enterprise-gateway/templates/ingress.yaml
  ingress:
    enabled: false

  kernel:
    # Kernel cluster role created by this chart.
    clusterRole: kernel-controller
    # Will start kernels in the same namespace as EG if True.
    shareGatewayNamespace: false
    # Timeout for kernel launching in seconds.
    launchTimeout: 120
    # Timeout for an idle kernel before its culled in seconds. Default is 1 hour.
    cullIdleTimeout: 3600
    # List of kernel names that are available for use.
    allowedKernels:
      - python_kubernetes
      - spark_python_operator
  #    - r_kubernetes
  #    - python_tf_kubernetes
  #    - python_tf_gpu_kubernetes
  #    - scala_kubernetes
  #    - spark_r_kubernetes
  #    - spark_python_kubernetes
  #    - spark_scala_kubernetes
    # Default kernel name should be something from the whitelist
    defaultKernelName: spark_python_operator

  # Kernel Image Puller (daemonset)
  # https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/kubernetes/helm/enterprise-gateway/templates/daemonset.yaml
  kip:
    enabled: false

For testing purposes, I ran kubefwd for all namespaces to be able to communicate with the enterprise-gateway service. I can successfully call the enterprise-gateway REST endpoints from the CLI, e.g.

curl http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs | jq yields

this JSON response. 
{
  "default": "spark_python_operator",
  "kernelspecs": {
    "python_kubernetes": {
      "name": "python_kubernetes",
      "spec": {
        "argv": [
          "python",
          "/usr/local/share/jupyter/kernels/python_kubernetes/scripts/launch_kubernetes.py",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}"
        ],
        "env": {},
        "display_name": "Python on Kubernetes",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy",
            "config": {
              "image_name": "elyra/kernel-py:dev"
            }
          },
          "debugger": true
        }
      },
      "resources": {
        "logo-64x64": "/kernelspecs/python_kubernetes/logo-64x64.png"
      }
    },
    "spark_python_operator": {
      "name": "spark_python_operator",
      "spec": {
        "argv": [
          "python",
          "/usr/local/share/jupyter/kernels/spark_python_operator/scripts/launch_custom_resource.py",
          "--RemoteProcessProxy.kernel-id",
          "{kernel_id}",
          "--RemoteProcessProxy.port-range",
          "{port_range}",
          "--RemoteProcessProxy.response-address",
          "{response_address}",
          "--RemoteProcessProxy.public-key",
          "{public_key}"
        ],
        "env": {},
        "display_name": "Spark Operator (Python)",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "process_proxy": {
            "class_name": "enterprise_gateway.services.processproxies.spark_operator.SparkOperatorProcessProxy",
            "config": {
              "image_name": "elyra/kernel-spark-py:dev",
              "executor_image_name": "elyra/kernel-spark-py:dev"
            }
          }
        }
      },
      "resources": {
        "logo-64x64": "/kernelspecs/spark_python_operator/logo-64x64.png"
      }
    }
  }
}

However, when I try to spawn a remote kernel via CLI

curl -X POST -i 'http://enterprise-gateway.ns-jupyter:8888/api/kernels' --data '{ "name": "python_kubernetes", "env": { "KERNEL_USERNAME": "jovyan" }}'

or via jupyter lab

python3 -m jupyterlab --debug \
  --gateway-url=http://enterprise-gateway.ns-jupyter:8888 \
  --GatewayClient.http_user=guest \
  --GatewayClient.http_pwd=guest-password \
  --GatewayClient.request_timeout=240.0 \
  --GatewayClient.connect_timeout=240.0

it spawns the remote kernel

remote kernel logs
/usr/local/bin/bootstrap-kernel.sh env: SHELL=/bin/bash 
KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 KERNEL_NAME=python_kubernetes HOSTNAME=jovyan-8fb19249-6000-4847-9f2e-cdd3e70d7eb8 LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none KERNEL_ID=8fb19249-6000-4847-9f2e-cdd3e70d7eb8 NB_UID=1000 PWD=/home/jovyan RESPONSE_ADDRESS=10.100.44.42:8877 MINICONDA_MD5=87e77f097f6ebb5127c77662dfc3165e HOME=/home/jovyan LANG=en_US.UTF-8 KUBERNETES_PORT_443_TCP=tcp://172.20.77.1:443 NB_GID=100 XDG_CACHE_HOME=/home/jovyan/.cache/ SHLVL=0 CONDA_DIR=/opt/conda MINICONDA_VERSION=4.8.2 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=172.20.77.1 PORT_RANGE=0..0 KERNEL_USERNAME=jovyan KERNEL_LANGUAGE=python CONDA_VERSION=4.8.2 NB_USER=jovyan KUBERNETES_SERVICE_HOST=172.20.77.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=tcp://172.20.77.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfiWkzCCMl/VFI8J2042RvWh13bSihVo+xp6HQnnQ8YWO5MsyW/nelzcMa2eBJWB+Yg/IQ/0q6BRog7oqDpUNbUxwGSzU3TyBYeRQCtXynR/EjFNyswE6gQrg15GbFxwmz4nfMkKXtlpItLrslcUqVY+wlUd+sdbJe9YMLp3REwIDAQAB DEBIAN_FRONTEND=noninteractive KERNEL_NAMESPACE=jovyan-8fb19249-6000-4847-9f2e-cdd3e70d7eb8 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 8fb19249-6000-4847-9f2e-cdd3e70d7eb8 --port-range 0..0 --response-address 10.100.44.42:8877 --public-key MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfiWkzCCMl/VFI8J2042RvWh13bSihVo+xp6HQnnQ8YWO5MsyW/nelzcMa2eBJWB+Yg/IQ/0q6BRog7oqDpUNbUxwGSzU3TyBYeRQCtXynR/EjFNyswE6gQrg15GbFxwmz4nfMkKXtlpItLrslcUqVY+wlUd+sdbJe9YMLp3REwIDAQAB --spark-context-initialization-mode none
[D 2022-10-07 15:09:13,424.424 launch_ipykernel] Using connection file '/tmp/kernel-8fb19249-6000-4847-9f2e-cdd3e70d7eb8_3kfui2q3.json'.
[I 2022-10-07 15:09:13,426.426 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 59305
[D 2022-10-07 15:09:13,427.427 launch_ipykernel] JSON Payload 'b'{"shell_port": 44887, "iopub_port": 38565, "stdin_port": 60485, "control_port": 34723, "hb_port": 59659, "ip": "0.0.0.0", "key": "f50f4038-8f8e-40eb-8a70-20d0fa8469db", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": 9, "pgid": 7, "comm_port": 59305, "kernel_id": "8fb19249-6000-4847-9f2e-cdd3e70d7eb8"}'
[D 2022-10-07 15:09:13,441.441 launch_ipykernel] Encrypted Payload 'b'eyJ2ZXJzaW9uIjogMSwgImtleSI6ICJUbUlTbkxheGNiMVE0c2Uwc2Zka09zalhGOWJhdXROTmtidHNvbVdlUzBFN3M4Zk1Jcm9idFgxaXBSTlVTS1ZBNklXd0ZDeCtVT0JDNkR0aXlYR1BxN3RqTkpZTVVrYlpFaUEyZVhCUGN3bjhneW5adHF2ZmVTWTZkMThJL1k0MkRVN0lmblcwU3B3T0VIbFVZdE5VYVdLR2o5VGxlMXVIQ3R2SktVd1M2enc9IiwgImNvbm5faW5mbyI6ICJiWjNnOStkT3psZHRoTEJ3cGVwcGR1bnVCZ3UwdDhiMWVOR3hGRGprTFpuUitKb044YVFrb0V1NEFSenhRMDlDamZQektzTFNHUjQ3bnNTeUEzOGwyTklNMUY2cTE3LzJXSjVWNHJ3bk5KaGs5N3Vtdlo4ZHg0blc4dlpua2hrT3lvVnE3RExrdlUrb0Q4dHl2aDNTWHpGa3EyTTVaZHFRZXFSMHpvWTMvcWhKT2MzSDY0ckJwcHhaTzZJcTF6WDJEd2tZVk9oU3BuL1IrLzRvRGxoNWNYck55QXFBeUhPNUZGaXJjYUdWZmVPRlFzUXFKZ2VaTVdKaHFyazE4aFI2Ylo2b3dtVGovQ3JtRXN1YUIzaXRVZnU5VjJIcGluaWVLbzdaZ2t4dVF2dEgrcHJ2dzR6bkdObldOVEhrQUh6MUw2QTJkakVpRlhHNGYvbzdIc1lpb0IwSUZKVGk1elpzMmMvU00yQmlvZW5Vd1JSeFhoelZid01pdEV2L0s2KzVEdzI4MHgwajdHc3Q5OTF0eHR5VXNQUC9Qa1ZweGRuMjJYTGxmMG9sZVhlVG5DSTd5ZkZCLzJBM2dhZENZN3ZxNVpTRXlDdWt1QVppTjIxdzFiZVZKQT09In0

but enterprise-gateway is unable to connect to the remote kernel as can be seen from the enterprise-gateway pod logs.

Reproduce

  1. Deploy the helm chart with the provided values-override.yaml from the previous section.
  2. Invoke the CLI or jupyter lab command from the Description section and start a 'python_kubernetes' kernel.

Expected behavior

I would expect that enterprise-gateway receives the connection information from the remote kernel so that enterprise-gateway does not timeout.

Context

$ python3 -m jupyter --version
Selected Jupyter core packages...
IPython          : 8.5.0
ipykernel        : 6.16.0
ipywidgets       : 8.0.2
jupyter_client   : 7.3.5
jupyter_core     : 4.11.1
jupyter_server   : 1.19.1
jupyterlab       : 3.4.8
nbclient         : 0.6.8
nbconvert        : 7.1.0
nbformat         : 5.6.1
notebook         : 6.4.12
qtconsole        : 5.3.2
traitlets        : 5.4.0
Command Line Output 
[D 2022-10-07 17:22:23.377 ServerApp] Searching ['/Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter', '/Users/tahesse/.jupyter', '/Users/tahesse/Library/Python/3.9/etc/jupyter', '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /etc/jupyter
[D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter
[D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Users/tahesse/Library/Python/3.9/etc/jupyter
[D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Users/tahesse/.jupyter
[D 2022-10-07 17:22:23.377 ServerApp] Looking for jupyter_config in /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter
[D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /etc/jupyter
[D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter
[D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Users/tahesse/Library/Python/3.9/etc/jupyter
[D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Users/tahesse/.jupyter
[D 2022-10-07 17:22:23.378 ServerApp] Looking for jupyter_server_config in /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter
[D 2022-10-07 17:22:23.380 ServerApp] Paths used for configuration of jupyter_server_config: 
        /etc/jupyter/jupyter_server_config.json
[D 2022-10-07 17:22:23.380 ServerApp] Paths used for configuration of jupyter_server_config: 
        /usr/local/etc/jupyter/jupyter_server_config.json
[D 2022-10-07 17:22:23.380 ServerApp] Paths used for configuration of jupyter_server_config: 
        /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/jupyter_server_config.json
[D 2022-10-07 17:22:23.381 ServerApp] Paths used for configuration of jupyter_server_config: 
        /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.d/jupyterlab.json
        /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.d/nbclassic.json
        /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.d/notebook_shim.json
        /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_server_config.json
[D 2022-10-07 17:22:23.394 ServerApp] Paths used for configuration of jupyter_server_config: 
        /Users/tahesse/.jupyter/jupyter_server_config.json
[D 2022-10-07 17:22:23.394 ServerApp] Paths used for configuration of jupyter_server_config: 
        /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter/jupyter_server_config.json
[D 2022-10-07 17:22:23.425 LabApp] Config changed: {'NotebookApp': {}, 'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': }, 'GatewayClient': {'url': 'http://enterprise-gateway.ns-jupyter:8888', 'http_user': 'guest', 'http_pwd': 'guest-password', 'request_timeout': 240.0, 'connect_timeout': 240.0}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[I 2022-10-07 17:22:23.426 ServerApp] jupyterlab | extension was successfully linked.
[D 2022-10-07 17:22:23.433 NotebookApp] Config changed: {'NotebookApp': {}, 'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': }, 'GatewayClient': {'url': 'http://enterprise-gateway.ns-jupyter:8888', 'http_user': 'guest', 'http_pwd': 'guest-password', 'request_timeout': 240.0, 'connect_timeout': 240.0}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[I 2022-10-07 17:22:23.433 ServerApp] nbclassic | extension was successfully linked.
[D 2022-10-07 17:22:23.655 ServerApp] Paths used for configuration of jupyter_notebook_config: 
        /Users/tahesse/.jupyter/jupyter_notebook_config.json
[D 2022-10-07 17:22:23.655 ServerApp] Paths used for configuration of jupyter_notebook_config: 
        /etc/jupyter/jupyter_notebook_config.json
[D 2022-10-07 17:22:23.655 ServerApp] Paths used for configuration of jupyter_notebook_config: 
        /usr/local/etc/jupyter/jupyter_notebook_config.json
[D 2022-10-07 17:22:23.656 ServerApp] Paths used for configuration of jupyter_notebook_config: 
        /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/jupyter_notebook_config.json
[D 2022-10-07 17:22:23.656 ServerApp] Paths used for configuration of jupyter_notebook_config: 
        /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
        /Users/tahesse/Library/Python/3.9/etc/jupyter/jupyter_notebook_config.json
[D 2022-10-07 17:22:23.666 ServerApp] Paths used for configuration of jupyter_notebook_config: 
        /Users/tahesse/.jupyter/jupyter_notebook_config.json
[I 2022-10-07 17:22:23.666 ServerApp] notebook_shim | extension was successfully linked.
[D 2022-10-07 17:22:23.667 ServerApp] Config changed: {'GatewayClient': {'url': 'http://enterprise-gateway.ns-jupyter:8888', 'http_user': 'guest', 'http_pwd': 'guest-password', 'request_timeout': 240.0, 'connect_timeout': 240.0}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'NotebookApp': {}, 'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': }}
[D 2022-10-07 17:22:23.667 ServerApp] Raising open file limit: soft 256->4096; hard 9223372036854775807->9223372036854775807
[I 2022-10-07 17:22:23.753 ServerApp] notebook_shim | extension was successfully loaded.
[I 2022-10-07 17:22:23.754 LabApp] JupyterLab extension loaded from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyterlab
[I 2022-10-07 17:22:23.754 LabApp] JupyterLab application directory is /Users/tahesse/Library/Python/3.9/share/jupyter/lab
[I 2022-10-07 17:22:23.757 ServerApp] jupyterlab | extension was successfully loaded.
[I 2022-10-07 17:22:23.802 ServerApp] nbclassic | extension was successfully loaded.
[I 2022-10-07 17:22:23.803 ServerApp] The port 8888 is already in use, trying another port.
[I 2022-10-07 17:22:23.804 ServerApp] Serving notebooks from local directory: /Users/tahesse/Code/fincloud-ml/infra/jupyter-k8s/jupyter
[I 2022-10-07 17:22:23.804 ServerApp] Jupyter Server 1.19.1 is running at:
[I 2022-10-07 17:22:23.804 ServerApp] http://localhost:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf
[I 2022-10-07 17:22:23.804 ServerApp]  or http://127.0.0.1:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf
[I 2022-10-07 17:22:23.804 ServerApp] Kernels will be managed by the Gateway server running at:
[I 2022-10-07 17:22:23.804 ServerApp] http://enterprise-gateway.ns-jupyter:8888
[I 2022-10-07 17:22:23.804 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-10-07 17:22:23.818 ServerApp] 

    To access the server, open this file in a browser:
        file:///Users/tahesse/Library/Jupyter/runtime/jpserver-90417-open.html
    Or copy and paste one of these URLs:
        http://localhost:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf
     or http://127.0.0.1:8889/lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf
[D 2022-10-07 17:22:25.395 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:25.404 ServerApp] Paths used for configuration of page_config:
/etc/jupyter/labconfig/page_config.json
[D 2022-10-07 17:22:25.405 ServerApp] Paths used for configuration of page_config:
/usr/local/etc/jupyter/labconfig/page_config.json
[D 2022-10-07 17:22:25.406 ServerApp] Paths used for configuration of page_config:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/page_config.json
[D 2022-10-07 17:22:25.406 ServerApp] Paths used for configuration of page_config:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/page_config.json
[D 2022-10-07 17:22:25.406 ServerApp] Paths used for configuration of page_config:
/Users/tahesse/.jupyter/labconfig/page_config.json
[D 2022-10-07 17:22:25.464 LabApp] Using contents: services/contents
[D 2022-10-07 17:22:25.485 LabApp] 200 GET /lab?token=f8eca77899eac00ea665cfc352b982b793d5d882e1641bdf (127.0.0.1) 93.99ms
[D 2022-10-07 17:22:25.508 ServerApp] Path main.191f751bb4b62e169926.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/main.191f751bb4b62e169926.js
[D 2022-10-07 17:22:25.542 ServerApp] 200 GET /static/lab/main.191f751bb4b62e169926.js?v=191f751bb4b62e169926 (127.0.0.1) 34.56ms
[D 2022-10-07 17:22:25.559 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/remoteEntry.1116b848a6f7126bf3e9.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/remoteEntry.1116b848a6f7126bf3e9.js
[D 2022-10-07 17:22:25.578 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/remoteEntry.1116b848a6f7126bf3e9.js (127.0.0.1) 19.93ms
[D 2022-10-07 17:22:25.580 ServerApp] Path jupyterlab_pygments/static/remoteEntry.aa1060b2d1221f8e5688.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/jupyterlab_pygments/static/remoteEntry.aa1060b2d1221f8e5688.js
[D 2022-10-07 17:22:25.584 ServerApp] 200 GET /lab/extensions/jupyterlab_pygments/static/remoteEntry.aa1060b2d1221f8e5688.js (127.0.0.1) 25.10ms
[D 2022-10-07 17:22:25.586 ServerApp] Path favicons/favicon.ico served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/favicons/favicon.ico
[D 2022-10-07 17:22:25.594 ServerApp] 200 GET /static/favicons/favicon.ico (127.0.0.1) 8.39ms
[D 2022-10-07 17:22:25.606 ServerApp] Path 4155.784ca1752696680bf373.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4155.784ca1752696680bf373.js
[D 2022-10-07 17:22:25.615 ServerApp] 200 GET /static/lab/4155.784ca1752696680bf373.js?v=784ca1752696680bf373 (127.0.0.1) 20.06ms
[D 2022-10-07 17:22:25.615 ServerApp] Path 2719.3fe9e6fa24f270d8b8d1.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/2719.3fe9e6fa24f270d8b8d1.js
[D 2022-10-07 17:22:25.657 ServerApp] 200 GET /static/lab/2719.3fe9e6fa24f270d8b8d1.js?v=3fe9e6fa24f270d8b8d1 (127.0.0.1) 61.41ms
[D 2022-10-07 17:22:25.658 ServerApp] Path 9473.098fe5e3f0187b3dc1fe.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/9473.098fe5e3f0187b3dc1fe.js
[D 2022-10-07 17:22:25.709 ServerApp] 200 GET /static/lab/9473.098fe5e3f0187b3dc1fe.js?v=098fe5e3f0187b3dc1fe (127.0.0.1) 112.91ms
[D 2022-10-07 17:22:25.710 ServerApp] Path 6700.3c95b4eec62b695c6f74.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/6700.3c95b4eec62b695c6f74.js
[D 2022-10-07 17:22:25.721 ServerApp] 200 GET /static/lab/6700.3c95b4eec62b695c6f74.js?v=3c95b4eec62b695c6f74 (127.0.0.1) 116.92ms
[D 2022-10-07 17:22:25.722 ServerApp] Path 1036.b407e23ec5843f2b574d.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1036.b407e23ec5843f2b574d.js
[D 2022-10-07 17:22:25.759 ServerApp] 200 GET /static/lab/1036.b407e23ec5843f2b574d.js?v=b407e23ec5843f2b574d (127.0.0.1) 153.75ms
[D 2022-10-07 17:22:25.761 ServerApp] Path jlab_core.716b79be8019e5bdc801.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/jlab_core.716b79be8019e5bdc801.js
[D 2022-10-07 17:22:25.860 ServerApp] 200 GET /static/lab/jlab_core.716b79be8019e5bdc801.js?v=716b79be8019e5bdc801 (127.0.0.1) 244.69ms
[D 2022-10-07 17:22:25.864 ServerApp] Path 1033.890eeae47460e241b1e9.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1033.890eeae47460e241b1e9.js
[D 2022-10-07 17:22:25.890 ServerApp] 200 GET /static/lab/1033.890eeae47460e241b1e9.js?v=890eeae47460e241b1e9 (127.0.0.1) 28.88ms
[D 2022-10-07 17:22:25.892 ServerApp] Path 4570.53adcb6f69939da383ff.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4570.53adcb6f69939da383ff.js
[D 2022-10-07 17:22:25.906 ServerApp] 200 GET /static/lab/4570.53adcb6f69939da383ff.js?v=53adcb6f69939da383ff (127.0.0.1) 44.50ms
[D 2022-10-07 17:22:25.906 ServerApp] Path 807.96166378e2efe232d81e.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/807.96166378e2efe232d81e.js
[D 2022-10-07 17:22:25.978 ServerApp] 200 GET /static/lab/807.96166378e2efe232d81e.js?v=96166378e2efe232d81e (127.0.0.1) 115.60ms
[D 2022-10-07 17:22:25.978 ServerApp] Path 7294.f71c2889fedcd71bd1ee.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7294.f71c2889fedcd71bd1ee.js
[D 2022-10-07 17:22:25.984 ServerApp] 200 GET /static/lab/7294.f71c2889fedcd71bd1ee.js?v=f71c2889fedcd71bd1ee (127.0.0.1) 121.53ms
[D 2022-10-07 17:22:25.985 ServerApp] Path 3935.4159b022aa6d82e44127.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3935.4159b022aa6d82e44127.js
[D 2022-10-07 17:22:26.032 ServerApp] 200 GET /static/lab/3935.4159b022aa6d82e44127.js?v=4159b022aa6d82e44127 (127.0.0.1) 168.47ms
[D 2022-10-07 17:22:26.032 ServerApp] Path 5096.96faf5ddac346a9ad9c3.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/5096.96faf5ddac346a9ad9c3.js
[D 2022-10-07 17:22:26.056 ServerApp] 200 GET /static/lab/5096.96faf5ddac346a9ad9c3.js?v=96faf5ddac346a9ad9c3 (127.0.0.1) 165.94ms
[D 2022-10-07 17:22:26.063 ServerApp] Path 1142.074d125bb59f5a332666.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1142.074d125bb59f5a332666.js
[D 2022-10-07 17:22:26.072 ServerApp] 200 GET /static/lab/1142.074d125bb59f5a332666.js?v=074d125bb59f5a332666 (127.0.0.1) 13.72ms
[D 2022-10-07 17:22:26.074 ServerApp] Path 7730.7e3a9fb140d2d55a51fc.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7730.7e3a9fb140d2d55a51fc.js
[D 2022-10-07 17:22:26.076 ServerApp] 200 GET /static/lab/7730.7e3a9fb140d2d55a51fc.js?v=7e3a9fb140d2d55a51fc (127.0.0.1) 17.94ms
[D 2022-10-07 17:22:26.077 ServerApp] Path 3308.3a94151c0d57440646ab.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3308.3a94151c0d57440646ab.js
[D 2022-10-07 17:22:26.087 ServerApp] 200 GET /static/lab/3308.3a94151c0d57440646ab.js?v=3a94151c0d57440646ab (127.0.0.1) 27.34ms
[D 2022-10-07 17:22:26.087 ServerApp] Path 4631.96a143e70f005fef7b59.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4631.96a143e70f005fef7b59.js
[D 2022-10-07 17:22:26.097 ServerApp] 200 GET /static/lab/4631.96a143e70f005fef7b59.js?v=96a143e70f005fef7b59 (127.0.0.1) 35.20ms
[D 2022-10-07 17:22:26.097 ServerApp] Path 7755.d506a1d9dadf30b1e490.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7755.d506a1d9dadf30b1e490.js
[D 2022-10-07 17:22:26.104 ServerApp] 200 GET /static/lab/7755.d506a1d9dadf30b1e490.js?v=d506a1d9dadf30b1e490 (127.0.0.1) 41.44ms
[D 2022-10-07 17:22:26.104 ServerApp] Path 6080.39703b8cc7bd284ce1bd.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/6080.39703b8cc7bd284ce1bd.js
[D 2022-10-07 17:22:26.131 ServerApp] 200 GET /static/lab/6080.39703b8cc7bd284ce1bd.js?v=39703b8cc7bd284ce1bd (127.0.0.1) 57.94ms
[D 2022-10-07 17:22:26.136 ServerApp] Path 2040.6489997c35a5d7b0b100.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/2040.6489997c35a5d7b0b100.js
[D 2022-10-07 17:22:26.148 ServerApp] 200 GET /static/lab/2040.6489997c35a5d7b0b100.js?v=6489997c35a5d7b0b100 (127.0.0.1) 16.17ms
[D 2022-10-07 17:22:26.150 ServerApp] Path 1057.a0608544097dd22a589e.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1057.a0608544097dd22a589e.js
[D 2022-10-07 17:22:26.154 ServerApp] 200 GET /static/lab/1057.a0608544097dd22a589e.js?v=a0608544097dd22a589e (127.0.0.1) 19.07ms
[D 2022-10-07 17:22:26.154 ServerApp] Path 3502.c915d993b864739407e3.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3502.c915d993b864739407e3.js
[D 2022-10-07 17:22:26.171 ServerApp] 200 GET /static/lab/3502.c915d993b864739407e3.js?v=c915d993b864739407e3 (127.0.0.1) 35.42ms
[D 2022-10-07 17:22:26.172 ServerApp] Path 3029.8636761a48c8422ce37f.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3029.8636761a48c8422ce37f.js
[D 2022-10-07 17:22:26.181 ServerApp] 200 GET /static/lab/3029.8636761a48c8422ce37f.js?v=8636761a48c8422ce37f (127.0.0.1) 45.05ms
[D 2022-10-07 17:22:26.181 ServerApp] Path 3219.8a3701725aaa457179f6.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3219.8a3701725aaa457179f6.js
[D 2022-10-07 17:22:26.187 ServerApp] 200 GET /static/lab/3219.8a3701725aaa457179f6.js?v=8a3701725aaa457179f6 (127.0.0.1) 50.77ms
[D 2022-10-07 17:22:26.187 ServerApp] Path 4151.fd7ca59b64f0adc97032.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4151.fd7ca59b64f0adc97032.js
[D 2022-10-07 17:22:26.191 ServerApp] 200 GET /static/lab/4151.fd7ca59b64f0adc97032.js?v=fd7ca59b64f0adc97032 (127.0.0.1) 41.75ms
[D 2022-10-07 17:22:26.192 ServerApp] Path 1249.047c76b5ea96a41605db.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1249.047c76b5ea96a41605db.js
[D 2022-10-07 17:22:26.202 ServerApp] 200 GET /static/lab/1249.047c76b5ea96a41605db.js?v=047c76b5ea96a41605db (127.0.0.1) 10.68ms
[D 2022-10-07 17:22:26.204 ServerApp] Path 1358.9ab4f57e8227ce75f427.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/1358.9ab4f57e8227ce75f427.js
[D 2022-10-07 17:22:26.213 ServerApp] 200 GET /static/lab/1358.9ab4f57e8227ce75f427.js?v=9ab4f57e8227ce75f427 (127.0.0.1) 21.61ms
[D 2022-10-07 17:22:26.214 ServerApp] Path 4429.c4f083ef6b6e29345fd4.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/4429.c4f083ef6b6e29345fd4.js
[D 2022-10-07 17:22:26.219 ServerApp] 200 GET /static/lab/4429.c4f083ef6b6e29345fd4.js?v=c4f083ef6b6e29345fd4 (127.0.0.1) 27.66ms
[D 2022-10-07 17:22:26.220 ServerApp] Path 466.239a0c009f33f1cb7cab.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/466.239a0c009f33f1cb7cab.js
[D 2022-10-07 17:22:26.226 ServerApp] 200 GET /static/lab/466.239a0c009f33f1cb7cab.js?v=239a0c009f33f1cb7cab (127.0.0.1) 33.70ms
[D 2022-10-07 17:22:26.226 ServerApp] Path 3496.ecb0e7fcc54191234ae6.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/3496.ecb0e7fcc54191234ae6.js
[D 2022-10-07 17:22:26.230 ServerApp] 200 GET /static/lab/3496.ecb0e7fcc54191234ae6.js?v=ecb0e7fcc54191234ae6 (127.0.0.1) 37.58ms
[D 2022-10-07 17:22:26.230 ServerApp] Path 126.1e2fb998804b27c72a3e.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/126.1e2fb998804b27c72a3e.js
[D 2022-10-07 17:22:26.244 ServerApp] 200 GET /static/lab/126.1e2fb998804b27c72a3e.js?v=1e2fb998804b27c72a3e (127.0.0.1) 40.89ms
[D 2022-10-07 17:22:26.246 ServerApp] Path 911.0c08f040896753efc653.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/911.0c08f040896753efc653.js
[D 2022-10-07 17:22:26.252 ServerApp] 200 GET /static/lab/911.0c08f040896753efc653.js?v=0c08f040896753efc653 (127.0.0.1) 7.23ms
[D 2022-10-07 17:22:26.252 ServerApp] Path 714.ff3007d0b781b5538482.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/714.ff3007d0b781b5538482.js
[D 2022-10-07 17:22:26.301 ServerApp] 200 GET /static/lab/714.ff3007d0b781b5538482.js?v=ff3007d0b781b5538482 (127.0.0.1) 55.57ms
[D 2022-10-07 17:22:26.301 ServerApp] Path 7796.098f091848a062457e84.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/static/7796.098f091848a062457e84.js
[D 2022-10-07 17:22:26.308 ServerApp] 200 GET /static/lab/7796.098f091848a062457e84.js?v=098f091848a062457e84 (127.0.0.1) 62.26ms
[D 2022-10-07 17:22:26.364 ServerApp] Path jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js
[D 2022-10-07 17:22:26.366 ServerApp] 200 GET /lab/extensions/jupyterlab_pygments/static/568.1e2faa2ba0bbe59c4780.js?v=1e2faa2ba0bbe59c4780 (127.0.0.1) 2.50ms
[D 2022-10-07 17:22:26.368 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js
[D 2022-10-07 17:22:26.375 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340 (127.0.0.1) 8.76ms
[D 2022-10-07 17:22:26.376 ServerApp] Path jupyterlab_pygments/static/747.8eb3ddccc7ec4987bff9.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/jupyterlab_pygments/static/747.8eb3ddccc7ec4987bff9.js
[D 2022-10-07 17:22:26.388 ServerApp] 200 GET /lab/extensions/jupyterlab_pygments/static/747.8eb3ddccc7ec4987bff9.js?v=8eb3ddccc7ec4987bff9 (127.0.0.1) 20.69ms
[D 2022-10-07 17:22:26.391 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/644.11f638668109f1562dd1.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/644.11f638668109f1562dd1.js
[D 2022-10-07 17:22:26.405 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/644.11f638668109f1562dd1.js?v=11f638668109f1562dd1 (127.0.0.1) 16.39ms
[D 2022-10-07 17:22:26.405 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/291.f707f721e4b3a5489ee0.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/291.f707f721e4b3a5489ee0.js
[D 2022-10-07 17:22:26.430 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/291.f707f721e4b3a5489ee0.js?v=f707f721e4b3a5489ee0 (127.0.0.1) 41.09ms
[D 2022-10-07 17:22:26.431 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/595.e3c9c115ecf5763f080b.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/595.e3c9c115ecf5763f080b.js
[D 2022-10-07 17:22:26.446 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/595.e3c9c115ecf5763f080b.js?v=e3c9c115ecf5763f080b (127.0.0.1) 56.37ms
[D 2022-10-07 17:22:26.447 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/113.dd66397047ecb9a605cf.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/113.dd66397047ecb9a605cf.js
[D 2022-10-07 17:22:26.469 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/113.dd66397047ecb9a605cf.js?v=dd66397047ecb9a605cf (127.0.0.1) 79.03ms
[D 2022-10-07 17:22:26.471 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/699.d004d23061da64f55a9a.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/699.d004d23061da64f55a9a.js
[D 2022-10-07 17:22:26.480 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/699.d004d23061da64f55a9a.js?v=d004d23061da64f55a9a (127.0.0.1) 10.04ms
[D 2022-10-07 17:22:26.480 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/965.8b60ee098074f7ed452d.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/965.8b60ee098074f7ed452d.js
[D 2022-10-07 17:22:26.482 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/965.8b60ee098074f7ed452d.js?v=8b60ee098074f7ed452d (127.0.0.1) 11.65ms
[D 2022-10-07 17:22:26.483 ServerApp] Path @jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js served from /Users/tahesse/Library/Python/3.9/share/jupyter/labextensions/@jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js
[D 2022-10-07 17:22:26.493 ServerApp] 200 GET /lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js?v=bcbea9feb6e7c4da7530 (127.0.0.1) 10.99ms
[D 2022-10-07 17:22:26.510 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:26.511 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:26.511 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:26.512 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:26.512 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:26.513 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:26.513 ServerApp] Request list kernel specs at: http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs
[D 2022-10-07 17:22:26.844 LabApp] 200 GET /lab/api/settings?1665156146509 (127.0.0.1) 333.67ms
[D 2022-10-07 17:22:26.845 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels
[D 2022-10-07 17:22:26.847 ServerApp] 200 GET /api/sessions?1665156146510 (127.0.0.1) 334.97ms
[D 2022-10-07 17:22:26.854 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:26.857 ServerApp] 200 GET /api/terminals?1665156146510 (127.0.0.1) 3.08ms
[D 2022-10-07 17:22:26.998 ServerApp] 200 GET /api/kernels?1665156146510 (127.0.0.1) 485.42ms
[I 2022-10-07 17:22:27.015 ServerApp] Default kernel name on Gateway server (spark_python_operator) differs from Notebook server (python3).  Updating to Gateway server's value.
[D 2022-10-07 17:22:27.016 ServerApp] 200 GET /api/kernelspecs?1665156146507 (127.0.0.1) 506.02ms
[D 2022-10-07 17:22:27.019 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:27.020 ServerApp] Request list kernel specs at: http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs
[D 2022-10-07 17:22:27.196 ServerApp] 200 GET /api/kernelspecs?1665156147018 (127.0.0.1) 176.54ms
[D 2022-10-07 17:22:27.908 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:27.913 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:27.914 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:27.915 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:27.916 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:27.916 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.082 ServerApp] 200 GET /lab/api/translations/en?1665156147899 (127.0.0.1) 174.89ms
[D 2022-10-07 17:22:28.106 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.121 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.133 ServerApp] 200 GET /lab/api/listings/@jupyterlab/extensionmanager-extension/listings.json?1665156148109 (127.0.0.1) 17.92ms
[D 2022-10-07 17:22:28.134 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.136 ServerApp] 200 GET /api/contents?content=1&1665156148112 (127.0.0.1) 2.67ms
[D 2022-10-07 17:22:28.137 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.137 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.138 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.141 LabApp] 200 GET /lab/api/settings/@jupyterlab/mainmenu-extension:plugin?1665156148125 (127.0.0.1) 4.86ms
[D 2022-10-07 17:22:28.142 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.142 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.143 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.143 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.144 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.144 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.145 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.145 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.146 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.148 LabApp] 200 GET /lab/api/settings/@jupyterlab/htmlviewer-extension:plugin?1665156148126 (127.0.0.1) 6.30ms
[D 2022-10-07 17:22:28.149 LabApp] 200 GET /lab/api/settings/@jupyterlab/cell-toolbar-extension:plugin?1665156148124 (127.0.0.1) 5.84ms
[D 2022-10-07 17:22:28.150 LabApp] 200 GET /lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1665156148124 (127.0.0.1) 6.10ms
[D 2022-10-07 17:22:28.153 LabApp] 200 GET /lab/api/settings/@jupyterlab/docmanager-extension:plugin?1665156148125 (127.0.0.1) 7.22ms
[D 2022-10-07 17:22:28.154 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.154 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.154 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.155 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.155 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.155 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.157 LabApp] 200 GET /lab/api/settings/@jupyterlab/notebook-extension:panel?1665156148126 (127.0.0.1) 3.22ms
[D 2022-10-07 17:22:28.157 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.158 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:28.160 LabApp] Node v16.16.0

[D 2022-10-07 17:22:28.295 LabApp] Yarn configuration loaded.
[D 2022-10-07 17:22:29.200 ServerApp] 200 GET /lab/api/translations/?1665156148154 (127.0.0.1) 1042.68ms
[D 2022-10-07 17:22:29.201 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[I 2022-10-07 17:22:29.384 LabApp] Build is up to date
[D 2022-10-07 17:22:29.389 LabApp] 200 GET /lab/api/build?1665156148103 (127.0.0.1) 1283.76ms
[D 2022-10-07 17:22:29.389 ServerApp] Path @jupyterlab/theme-light-extension/index.css served from /Users/tahesse/Library/Python/3.9/share/jupyter/lab/themes/@jupyterlab/theme-light-extension/index.css
[D 2022-10-07 17:22:29.406 ServerApp] 200 GET /lab/api/themes/@jupyterlab/theme-light-extension/index.css (127.0.0.1) 22.31ms
[D 2022-10-07 17:22:29.407 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.407 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.408 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.409 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.409 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.409 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.409 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.410 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.410 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.410 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.414 LabApp] 200 GET /lab/api/settings/@jupyterlab/filebrowser-extension:widget?1665156148190 (127.0.0.1) 7.30ms
[D 2022-10-07 17:22:29.417 LabApp] 200 GET /lab/api/settings/@jupyterlab/csvviewer-extension:csv?1665156148190 (127.0.0.1) 8.02ms
[D 2022-10-07 17:22:29.417 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.418 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.418 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.419 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.430 LabApp] 200 GET /lab/api/workspaces/default?1665156148237 (127.0.0.1) 13.14ms
[D 2022-10-07 17:22:29.439 LabApp] 200 GET /lab/api/settings/@jupyterlab/application-extension:context-menu?1665156148246 (127.0.0.1) 21.62ms
[D 2022-10-07 17:22:29.440 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.441 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.442 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.442 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.442 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.443 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.448 LabApp] 200 GET /lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1665156148268 (127.0.0.1) 8.37ms
[D 2022-10-07 17:22:29.451 LabApp] 200 GET /lab/api/settings/@jupyterlab/csvviewer-extension:tsv?1665156148191 (127.0.0.1) 9.58ms
[D 2022-10-07 17:22:29.452 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.453 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.453 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.454 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.454 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.455 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.463 LabApp] 200 GET /lab/api/settings/@jupyterlab/fileeditor-extension:plugin?1665156148191 (127.0.0.1) 10.64ms
[D 2022-10-07 17:22:29.474 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.474 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.475 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.678 LabApp] 200 GET /lab/api/settings?1665156149472 (127.0.0.1) 204.78ms
[D 2022-10-07 17:22:29.679 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.680 ServerApp] 200 GET /api/sessions?1665156149473 (127.0.0.1) 0.86ms
[D 2022-10-07 17:22:29.680 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.681 ServerApp] Paths used for configuration of default_setting_overrides:
/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.681 ServerApp] Paths used for configuration of default_setting_overrides:
/usr/local/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.682 ServerApp] Paths used for configuration of default_setting_overrides:
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.682 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/Library/Python/3.9/etc/jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.682 ServerApp] Paths used for configuration of default_setting_overrides:
/Users/tahesse/.jupyter/labconfig/default_setting_overrides.json
[D 2022-10-07 17:22:29.684 LabApp] 200 GET /lab/api/settings/@jupyterlab/shortcuts-extension:shortcuts?1665156149575 (127.0.0.1) 4.63ms
[D 2022-10-07 17:22:29.700 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[I 2022-10-07 17:22:29.701 ServerApp] Request start kernel: kernel_id=None, path=''
[D 2022-10-07 17:22:29.701 ServerApp] Request new kernel at: http://enterprise-gateway.ns-jupyter:8888/api/kernels
[D 2022-10-07 17:22:29.808 ServerApp] Using contents: services/contents
[D 2022-10-07 17:22:29.852 ServerApp] Path favicon.ico served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/favicon.ico
[D 2022-10-07 17:22:29.857 ServerApp] Path style/bootstrap.min.css served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/style/bootstrap.min.css
[D 2022-10-07 17:22:29.866 ServerApp] Path style/bootstrap-theme.min.css served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/style/bootstrap-theme.min.css
[D 2022-10-07 17:22:29.875 ServerApp] Path style/index.css served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/style/index.css
[D 2022-10-07 17:22:29.880 ServerApp] Path logo/logo.png served from /Users/tahesse/Library/Python/3.9/lib/python/site-packages/jupyter_server/static/logo/logo.png
[W 2022-10-07 17:22:29.882 ServerApp] 404 GET /static/lab/jlab_core.716b79be8019e5bdc801.js.map?v=716b79be8019e5bdc801 (127.0.0.1) 74.79ms referer=None
[D 2022-10-07 17:22:29.926 ServerApp] 200 GET /api/nbconvert?1665156148162 (127.0.0.1) 519.28ms
[D 2022-10-07 17:22:29.992 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:29.996 LabApp] 204 PUT /lab/api/workspaces/default?1665156149991 (127.0.0.1) 3.99ms
[D 2022-10-07 17:22:29.997 ServerApp] 200 GET /api/nbconvert?1665156148161 (127.0.0.1) 796.41ms
[D 2022-10-07 17:22:30.008 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:30.009 ServerApp] 200 GET /api/contents?content=1&1665156150007 (127.0.0.1) 1.86ms
[D 2022-10-07 17:22:36.935 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:36.937 ServerApp] 200 GET /api/terminals?1665156156927 (127.0.0.1) 2.82ms
[D 2022-10-07 17:22:37.007 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:37.008 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels
[D 2022-10-07 17:22:37.145 ServerApp] 200 GET /api/kernels?1665156157004 (127.0.0.1) 138.73ms
[D 2022-10-07 17:22:39.707 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:39.709 ServerApp] 200 GET /api/sessions?1665156159699 (127.0.0.1) 3.15ms
[D 2022-10-07 17:22:40.018 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:40.026 ServerApp] 200 GET /api/contents?content=1&1665156160015 (127.0.0.1) 7.99ms
[D 2022-10-07 17:22:46.954 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:46.956 ServerApp] 200 GET /api/terminals?1665156166948 (127.0.0.1) 2.80ms
[D 2022-10-07 17:22:47.156 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:47.157 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels
[D 2022-10-07 17:22:47.293 ServerApp] 200 GET /api/kernels?1665156167151 (127.0.0.1) 136.77ms
[D 2022-10-07 17:22:49.720 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:49.723 ServerApp] 200 GET /api/sessions?1665156169716 (127.0.0.1) 4.12ms
[D 2022-10-07 17:22:50.041 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:50.048 ServerApp] 200 GET /api/contents?content=1&1665156170034 (127.0.0.1) 7.76ms
[D 2022-10-07 17:22:56.973 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:56.976 ServerApp] 200 GET /api/terminals?1665156176966 (127.0.0.1) 3.96ms
[D 2022-10-07 17:22:57.304 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:57.306 ServerApp] Request list kernels: http://enterprise-gateway.ns-jupyter:8888/api/kernels
[D 2022-10-07 17:22:57.447 ServerApp] 200 GET /api/kernels?1665156177299 (127.0.0.1) 142.16ms
[D 2022-10-07 17:22:59.735 ServerApp] Accepting token-authenticated connection from 127.0.0.1
[D 2022-10-07 17:22:59.738 ServerApp] 200 GET /api/sessions?1665156179727 (127.0.0.1) 4.94ms

If I can provide any more information, please let me know!

Thank you for your work on enterprise-gateway, and thanks for any help / pointers in the right direction! :)

welcome[bot] commented 2 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

tafaust commented 2 years ago

I should also point out that I tracked the error down to this line in the jupyter-server project: https://github.com/jupyter-server/jupyter_server/blob/7d2154a1e243f80ed5fc4c067fd022e32f3fc8f0/jupyter_server/gateway/managers.py#L70

kevin-bates commented 2 years ago

Hi @tahesse - thanks for opening this issue and the great details! I will try to take a look into this next week but if anyone else wants to look into it, that would be great!

The kernel logs look okay and given EG never appears to receive the kernel connection information, implies there's something amiss between the kernel pod and the EG pod. I'm assuming they're running within the same network - correct?

10.100.44.42 (used by the kernel pod) looks like it might be a public IP while this information implies an internal IP of 172.20.43.27:

[D 2022-10-07 15:09:11.198 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'SHELL': '/bin/bash', 'KUBERNETES_SERVICE_PORT_HTTPS': '443', 'EG_MIRROR_WORKING_DIRS': 'False', 'KUBERNETES_SERVICE_PORT': '443', 'ENTERPRISE_GATEWAY_PORT_8877_TCP': 'tcp://172.20.43.27:8877', 'EG_NAMESPACE': 'ns-jupyter', 'ENTERPRISE_GATEWAY_SERVICE_PORT_HTTP': '8888', 'HOSTNAME': 'enterprise-gateway-8559c987dd-9lv2p', 'LANGUAGE': 'en_US.UTF-8', 'EG_SHARED_NAMESPACE': 'False', 'EG_PORT': '8888', 'EG_LOG_LEVEL': 'DEBUG', 'JAVA_HOME': '/usr/lib/jvm/java-8-openjdk-amd64', 'NB_UID': '1000', 'ENTERPRISE_GATEWAY_SERVICE_HOST': '172.20.43.27', 'EG_ALLOWED_KERNELS': '"python_kubernetes","spark_python_operator"', 'PWD': '/usr/local/bin', 'ENTERPRISE_GATEWAY_PORT_8877_TCP_PROTO': 'tcp', 'EG_CULL_IDLE_TIMEOUT': '3600', 'EG_DEFAULT_KERNEL_NAME': 'spark_python_operator', 'ENTERPRISE_GATEWAY_SERVICE_PORT_HTTP_RESPONSE': '8877', 'ENTERPRISE_GATEWAY_PORT_8888_TCP_PORT': '8888', 'EG_ENABLE_TUNNELING': 'False', 'ENTERPRISE_GATEWAY_PORT_8888_TCP_ADDR': '172.20.43.27', 'EG_KERNEL_LAUNCH_TIMEOUT': '120', 'HOME': '/home/jovyan', 'LANG': 'en_US.UTF-8', 'KUBERNETES_PORT_443_TCP': 'tcp://172.20.77.1:443', 'ENTERPRISE_GATEWAY_PORT_8877_TCP_PORT': '8877', 'EG_LIST_KERNELS': 'True', 'EG_SSH_PORT': '2122', 'NB_GID': '100', 'EG_RESPONSE_PORT': '8877', 'ENTERPRISE_GATEWAY_PORT_8888_TCP': 'tcp://172.20.43.27:8888', 'KG_PORT': '8888', 'EG_CULL_CONNECTED': 'False', 'EG_PORT_RETRIES': '0', 'KG_IP': '0.0.0.0', 'ENTERPRISE_GATEWAY_PORT_8877_TCP_ADDR': '172.20.43.27', 'EG_CULL_INTERVAL': '60', 'EG_IP': '0.0.0.0', 'SHLVL': '0', 'CONDA_DIR': '/opt/conda', 'ENTERPRISE_GATEWAY_SERVICE_PORT': '8888', 'SPARK_HOME': '/opt/spark', 'KUBERNETES_PORT_443_TCP_PROTO': 'tcp', 'KG_PORT_RETRIES': '0', 'KUBERNETES_PORT_443_TCP_ADDR': '172.20.77.1', 'SPARK_VER': '3.2.1', 'ENTERPRISE_GATEWAY_PORT': 'tcp://172.20.43.27:8888', 'NB_USER': 'jovyan', 'KUBERNETES_SERVICE_HOST': '172.20.77.1', 'ENTERPRISE_GATEWAY_PORT_8888_TCP_PROTO': 'tcp', 'LC_ALL': 'en_US.UTF-8', 'KUBERNETES_PORT': 'tcp://172.20.77.1:443', 'KUBERNETES_PORT_443_TCP_PORT': '443', 'PATH': '/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'EG_KERNEL_CLUSTER_ROLE': 'kernel-controller', 'DEBIAN_FRONTEND': 'noninteractive', 'KERNEL_USERNAME': 'jovyan', 'KERNEL_GATEWAY': '1', 'KERNEL_POD_NAME': 'jovyan-8fb19249-6000-4847-9f2e-cdd3e70d7eb8', 'KERNEL_SERVICE_ACCOUNT_NAME': 'default', 'KERNEL_NAMESPACE': 'jovyan-8fb19249-6000-4847-9f2e-cdd3e70d7eb8', 'KERNEL_IMAGE': 'elyra/kernel-py:dev', 'KERNEL_EXECUTOR_IMAGE': 'elyra/kernel-py:dev', 'KERNEL_UID': '1000', 'KERNEL_GID': '100', 'EG_MIN_PORT_RANGE_SIZE': '1000', 'EG_MAX_PORT_RANGE_RETRIES': '5', 'KERNEL_ID': '8fb19249-6000-4847-9f2e-cdd3e70d7eb8', 'KERNEL_LANGUAGE': 'python', 'EG_IMPERSONATION_ENABLED': 'False'}

Since it appears it's finding the public local IP you might try setting the env EG_PROHIBITED_LOCAL_IPS to 10.100.44.42 and I suspect it will likely find 172.20.43.27. That's what that env is intended for - to help mitigate ambiguities.

I will try to deploy your helm chart in my environment upon my return next week although I suspect this issue stems from the cluster's configuration more than EG and probably won't reproduce the issue (but we'll see).

tafaust commented 2 years ago

@kevin-bates thank you for the reply!

I'm assuming they're running within the same network - correct?

I'm 99% sure because they run in the same Kubernetes Cluster in different namespace (it does work for other services though). I also did connect via ssh to the remote kernel pod and was able to communicate with the REST API.

10.100.44.42 is indeed an IP address in our subnet (it should be the node IP). 172.20.43.27 does look like a local (container/pod) IP, is that the intended behavior?

I will give EG_PROHIBITED_LOCAL_IPS: '10.100.44.42' a try. TYSM for the reply @kevin-bates!

I will let you know about the outcome.

kevin-bates commented 2 years ago

10.100.44.42 is indeed an IP address in our subnet (it should be the node IP). 172.20.43.27 does look like a local (container/pod) IP, is that the intended behavior?

Yes, internal is preferred unless the kernel is running in an external network.

tafaust commented 2 years ago

@kevin-bates I started my local jupyterlab with:

EG_PROHIBITED_LOCAL_IPS='10.100.*.*' python3 -m jupyterlab --debug \
  --gateway-url=http://enterprise-gateway.ns-jupyter:8888 \
  --GatewayClient.http_user=guest \
  --GatewayClient.http_pwd=guest-password \
  --GatewayClient.request_timeout=240.0 \
  --GatewayClient.connect_timeout=240.0

My remote kernel logs:

/usr/local/bin/bootstrap-kernel.sh env: SHELL=/bin/bash KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 KERNEL_NAME=python_kubernetes HOSTNAME=guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3 LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none KERNEL_ID=51985e3e-f3e8-4a34-a4e2-69d44c201ce3 NB_UID=1000 PWD=/home/jovyan RESPONSE_ADDRESS=10.100.44.42:8877 MINICONDA_MD5=87e77f097f6ebb5127c77662dfc3165e HOME=/home/jovyan LANG=en_US.UTF-8 KUBERNETES_PORT_443_TCP=tcp://172.20.77.1:443 NB_GID=100 XDG_CACHE_HOME=/home/jovyan/.cache/ SHLVL=0 CONDA_DIR=/opt/conda MINICONDA_VERSION=4.8.2 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=172.20.77.1 PORT_RANGE=0..0 KERNEL_USERNAME=guest KERNEL_LANGUAGE=python CONDA_VERSION=4.8.2 NB_USER=jovyan KUBERNETES_SERVICE_HOST=172.20.77.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=tcp://172.20.77.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfiWkzCCMl/VFI8J2042RvWh13bSihVo+xp6HQnnQ8YWO5MsyW/nelzcMa2eBJWB+Yg/IQ/0q6BRog7oqDpUNbUxwGSzU3TyBYeRQCtXynR/EjFNyswE6gQrg15GbFxwmz4nfMkKXtlpItLrslcUqVY+wlUd+sdbJe9YMLp3REwIDAQAB DEBIAN_FRONTEND=noninteractive KERNEL_NAMESPACE=guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3 _=/usr/bin/env
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 51985e3e-f3e8-4a34-a4e2-69d44c201ce3 --port-range 0..0 --response-address 10.100.44.42:8877 --public-key MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEfiWkzCCMl/VFI8J2042RvWh13bSihVo+xp6HQnnQ8YWO5MsyW/nelzcMa2eBJWB+Yg/IQ/0q6BRog7oqDpUNbUxwGSzU3TyBYeRQCtXynR/EjFNyswE6gQrg15GbFxwmz4nfMkKXtlpItLrslcUqVY+wlUd+sdbJe9YMLp3REwIDAQAB --spark-context-initialization-mode none
[D 2022-10-10 06:45:21,298.298 launch_ipykernel] Using connection file '/tmp/kernel-51985e3e-f3e8-4a34-a4e2-69d44c201ce3_kpviwi08.json'.
[I 2022-10-10 06:45:21,300.300 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 59223
[D 2022-10-10 06:45:21,301.301 launch_ipykernel] JSON Payload 'b'{"shell_port": 44921, "iopub_port": 46351, "stdin_port": 38143, "control_port": 32833, "hb_port": 43811, "ip": "0.0.0.0", "key": "b6f18ebe-f585-4a45-9897-6f347c3f6ae3", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": 9, "pgid": 7, "comm_port": 59223, "kernel_id": "51985e3e-f3e8-4a34-a4e2-69d44c201ce3"}'
[D 2022-10-10 06:45:21,348.348 launch_ipykernel] Encrypted Payload 'b'eyJ2ZXJzaW9uIjogMSwgImtleSI6ICJYd1ZyU2MwUWFoSlIwZHJ3YWNmaThaYTYzTWtQM3ZkTGtEMnl2b0NJc0I5SUMyOTlSU3A4c2w2N1d3VGxXSzBtME1ERXpjU3VVdzJVMjltL0R3aWhTNVVpaDFmZk9JaU5RRGxwcDhkKzdDSHM2c3ZXZnE0S29TOWYrMjlxYjl0WDlGdmVXRXNXbXlCc1hWeTZVTDZRZG90QUJXc29SUGE2YzI4UVc2SGlGUXM9IiwgImNvbm5faW5mbyI6ICJaaXFmeFl3UUxobW5HdUxGY0N4S285SFZVSEcrcFYzS3RWaTg5UDdkQnF0bi9EeWMvclo2eUVLaHhkSWpRSXR1Sm9URDZuTzFEN3FDN2pCVFhWTmZ4akRGNjlCYXBnUWVQVzFrOXN5dnRWK0lBTDM1MnpzWFhKeWgxZFE4ZUFyM2F1Mm1tWUFRMVExRzZvbG5kSTlBS2hrSk5KRWo4SC9QVE5zWU9lMFpPZUtpMlF1YTk4QmNRZ3dSaGgzSGpXTE92ZmJBejdBelYvREdzN0hZYjVZSERDUmVuNk1iaElBV21Za1ZQV21mMjB1VlorK2kwSVg5eUFBVDZ2YUR6UWkvcnNEWVNHQ2dUOVhDQ0o5Uk9venBDTXp4NkJuNXJ3Ly9qWGgzNGZqTjRkSGdOa0RMWEtWMWx1QUpDaTJDZ2pUWUxJL2loTkVWeTFwVWl5cnlneUJadG0vdys0eHlJd0F3ay9nK0ZVTjlQaC9sUG92MDNoVnJZTUdOa1JrdjJoVm9vNTA2dVFMbE1kR04zY3dIT294TFdYTk5qZWxUVXBsUkNsUGV4UHFTRS9XMEZ0U0dyYWRNTDVMSExzUEI3TmFmNi9uSjloYmNKN2hvUEtNRWZUSHJ3QT09In0='

and enterprise-gateway logs:

[D 2022-10-10 06:49:08.493 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:09.022 EnterpriseGatewayApp] 417: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:09.048 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:09.578 EnterpriseGatewayApp] 418: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:09.606 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:10.131 EnterpriseGatewayApp] 419: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:10.158 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:10.689 EnterpriseGatewayApp] 420: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:10.712 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:11.234 EnterpriseGatewayApp] 421: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:11.255 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:11.779 EnterpriseGatewayApp] 422: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:11.807 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:12.336 EnterpriseGatewayApp] 423: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:12.363 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:12.896 EnterpriseGatewayApp] 424: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:12.925 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:13.448 EnterpriseGatewayApp] 425: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:13.475 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:13.999 EnterpriseGatewayApp] 426: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:14.027 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:14.558 EnterpriseGatewayApp] 427: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:14.586 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:15.121 EnterpriseGatewayApp] 428: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:15.151 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:15.687 EnterpriseGatewayApp] 429: Waiting to connect to k8s pod in namespace 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3'. Name: 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3', Status: 'Running', Pod IP: '10.100.18.194', KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3'
[D 2022-10-10 06:49:15.715 EnterpriseGatewayApp] Waiting for KernelID '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' to send connection info from host 'guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3' - retrying...
[D 2022-10-10 06:49:16.268 EnterpriseGatewayApp] KubernetesProcessProxy.terminate_container_resources, pod: guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3.guest-51985e3e-f3e8-4a34-a4e2-69d44c201ce3, kernel ID: 51985e3e-f3e8-4a34-a4e2-69d44c201ce3 has been terminated.
[E 2022-10-10 06:49:16.275 EnterpriseGatewayApp] KernelID: '51985e3e-f3e8-4a34-a4e2-69d44c201ce3' launch timeout due to: Waited too long (238.0s) to get connection file
[E 221010 06:49:16 web:2239] 500 POST /api/kernels (127.0.0.1) 238663.10ms

I don't quite understand why enterprise-gateway still tries to work with the remote IP address in 10.100.0.0/16.

Thank you for the information and help so far!

tafaust commented 2 years ago

I should also point out that I tracked the error down to this line in the jupyter-server project: https://github.com/jupyter-server/jupyter_server/blob/7d2154a1e243f80ed5fc4c067fd022e32f3fc8f0/jupyter_server/gateway/managers.py#L70

Just to be follow up, I am currently stuck at this point. grafik

Is there any way that I can potentially start a kernel locally to debug it? I am currently investigating how this could be further debugged.

kevin-bates commented 2 years ago

EG_PROHIBITED_LOCAL_IPS='10.100..' python3 -m jupyterlab --debug \ --gateway-url=http://enterprise-gateway.ns-jupyter:8888 \ --GatewayClient.http_user=guest \ --GatewayClient.http_pwd=guest-password \ --GatewayClient.request_timeout=240.0 \ --GatewayClient.connect_timeout=240.0

This is setting the env EG_PROHIBITED_LOCAL_IPS only in the jupyter lab process - where it does not apply. You need to restart your EG process (Kubernetes Pod) with this env set. If you deploy using the helm charts, then add the following entry to the env stanza in the deployment.yaml file:

        - name: EG_PROHIBITED_LOCAL_IPS
          value: "10.100.*.*"

(I'm not certain whether the quotes are necessary or not.)

You can docker exec into the EG pod and confirm its env prior to launching a kernel.

Is there any way that I can potentially start a kernel locally to debug it?

You could try using python_distributed and set EG_REMOTE_HOSTS to "localhost". This will use the DistributedProcessProxy (not the KubernetesProcessProxy), but I believe the EG_PROHIBITED_LOCAL_IPS portion of things is still similar - although the K8s env introduces other networks (like the internal docker networks) that DistributedProcessProxy does not.

Your primary issue wrt this last exercise is that you're not setting the env into the appropriate process. I recommend sticking with the K8s env for a bit longer since the symptoms are somewhat specific to that env.

tafaust commented 2 years ago

Thanks @kevin-bates. Setting the EG_PROHIBITED_LOCAL_IPS on the deployment.yaml did work but prevents all connections to/from enterprise-gateway.

I have enterprise-gateway with istio deployed, I will remove it and retry. If that doesn't help, I will set EG_REMOTE_HOSTS to localhost. I will get back with the results tomorrow.

Meanwhile, would an extraEnv configuration option (as it is done in the jupyter helm file) for enterprise-gateway be a thing? I can quickly craft a PR on that.

kevin-bates commented 2 years ago

Setting the EG_PROHIBITED_LOCAL_IPS on the deployment.yaml did work but prevents all connections to/from enterprise-gateway.

Hmm - this should not have any bearing on the accessibility of EG from applications. Could you clarify what you mean by prevents all connections to/from enterprise-gateway?

I have enterprise-gateway with istio deployed

Hmm, might istio be preventing the response to port 8877 in the first place? That port number is configurable and perhaps something that you need to configure in your ingress.

I will set EG_REMOTE_HOSTS to localhost.

I'm not sure how useful this experiment will be and may not be worth the effort.

Meanwhile, would an extraEnv configuration option (as it is done in the jupyter helm file) for enterprise-gateway be a thing? I can quickly craft a PR on that.

Could you please clarify what you mean by this as well? Typically k8s deployments are performed via helm or some other form of yaml and you're free to add whatever you want - so some details would be helpful.

tafaust commented 2 years ago

Hmm - this should not have any bearing on the accessibility of EG from applications. Could you clarify what you mean by prevents all connections to/from enterprise-gateway?

I cannot communicate with the REST API from within the cluster:

curl -vvv http://enterprise-gateway.ns-jupyter:8888/api/kernelspecs                                                                         7 ↵ tahesse@gauss
*   Trying 127.1.41.1:8888...
* connect to 127.1.41.1 port 8888 failed: Connection refused
* Failed to connect to enterprise-gateway.ns-jupyter port 8888 after 10 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to enterprise-gateway.ns-jupyter port 8888 after 10 ms: Connection refused

Hmm, might istio be preventing the response to port 8877 in the first place? That port number is configurable and perhaps something that you need to configure in your ingress.

I have no ingress running for enterprise-gateway; is an ingress mandatory? I am testing with kubefwd and depend on cluster internal service resolution via service-name.namespace.svc.cluster.local as described in https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/.

Could you please clarify what you mean by this as well? Typically k8s deployments are performed via helm or some other form of yaml and you're free to add whatever you want - so some details would be helpful.

Sure! For the Jupyterhub k8s deployment (via helm), there is this option in the values.yaml:

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/values.yaml#L76

which is used as such in the hub deployment:

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/templates/hub/deployment.yaml#L208-L210

The advantage is that an operator can directly add the env vars in the values-override.yaml instead of modifying the deployment.yaml (especially nice, when extending the helm chart).

kevin-bates commented 2 years ago

Hmm - this should not have any bearing on the accessibility of EG from applications. Could you clarify what you mean by prevents all connections to/from enterprise-gateway?

I cannot communicate with the REST API from within the cluster:

EG_PROHIBITED_LOCAL_IPS is only used within the process proxies and should not affect access to EG itself. I'm curious how you determined this worked if you can't trigger the creation of a kernel?

Could you please clarify what you mean by this as well? Typically k8s deployments are performed via helm or some other form of yaml and you're free to add whatever you want - so some details would be helpful.

The advantage is that an operator can directly add the env vars in the values-override.yaml instead of modifying the deployment.yaml (especially nice, when extending the helm chart).

I see, yes, that is helpful. A PR would be great!

I have no ingress running for enterprise-gateway; is an ingress mandatory? I am testing with kubefwd and depend on cluster internal service resolution via service-name.namespace.svc.cluster.local as described in https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/.

No, ingress is not mandatory. Just some form of a reverse proxy is recommended and it looks like you're using Hub. I'm not familiar with kubefwd but wondering if what you're doing is exacerbating this ability for the kernel pods to communicate their connection information back to EG for whatever reason.

tafaust commented 2 years ago

EG_PROHIBITED_LOCAL_IPS is only used within the process proxies and should not affect access to EG itself. I'm curious how you determined this worked if you can't trigger the creation of a kernel?

Sorry, I hope that I can clear up the confusion. kubefwd does forward all service connections to my host so that I can communicate as if my host is part of the cluster network. I have to restart kubefwd after redeployment of enterprise-gateway because it does not refresh connections (it tries to but fails). So this was really an issue on my side because I forgot to restart kubefwd. Sorry about that!

Just some form of a reverse proxy is recommended and it looks like you're using Hub.

That is my ultimate goal but I first try to make it work with jupyterlab because the development/fix cycle is faster and there are less moving parts.

I'm not familiar with kubefwd but wondering if what you're doing is exacerbating this ability for the kernel pods to communicate their connection information back to EG for whatever reason.

We use kubefwd consistently during development when developing towards cluster infrastructure. https://github.com/txn2/kubefwd

I see, yes, that is helpful. A PR would be great!

I will craft one after I got it running. :)

Update on my side: I know have added

        - name: EG_PROHIBITED_LOCAL_IPS
          value: '10.100.*.*'
        - name: EG_RESPONSE_ADDRESS
          value: '172.20.71.6:8877'

to the EG deployment.yaml and it seems that the response_address is not propagated to the kernel as I'd expect from looking at

whereas the environmental variables exist in the enterprise-gateway pod:

jovyan@enterprise-gateway-6bc565d956-gb4s2:/usr/local/bin$ printenv | grep EG_RESPONSE
EG_RESPONSE_ADDRESS=172.20.71.6:8877
EG_RESPONSE_PORT=8877
Kernel pod logs 
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 4b6d95f4-241d-4644-b622-f6ff4b54814a --port-range 0..0 --response-address 10.100.35.193:8877 --public-key MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDSQP9YFtzoY1v+VwYXd09x/fNEDSFIASwjoAoNA5jiOAKQujgw/xxBge1SnovvlGDjOFkkuK1bfRvECYnHafM98hRGlRVGXzbbw5d6hDHUXQMdXgh1JQJFAV8vMI6o3Sqm3ZJRodYuUDvPbbJRNhSbQEEVuzZN5R5p382gxUUFTQIDAQAB --spark-context-initialization-mode none
/usr/local/bin/bootstrap-kernel.sh env: SHELL=/bin/bash KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 KERNEL_NAME=python_kubernetes HOSTNAME=guest-4b6d95f4-241d-4644-b622-f6ff4b54814a LANGUAGE=en_US.UTF-8 KERNEL_SPARK_CONTEXT_INIT_MODE=none GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_SERVICE_PORT_SPARK_DRIVER_UI_PORT=4040 KERNEL_ID=4b6d95f4-241d-4644-b622-f6ff4b54814a NB_UID=1000 GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_SERVICE_PORT=4040 GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_PORT_4040_TCP_PROTO=tcp PWD=/home/jovyan RESPONSE_ADDRESS=10.100.35.193:8877 GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_PORT_4040_TCP=tcp://172.20.18.60:4040 MINICONDA_MD5=87e77f097f6ebb5127c77662dfc3165e HOME=/home/jovyan LANG=en_US.UTF-8 KUBERNETES_PORT_443_TCP=tcp://172.20.77.1:443 NB_GID=100 GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_PORT_4040_TCP_PORT=4040 XDG_CACHE_HOME=/home/jovyan/.cache/ SHLVL=0 CONDA_DIR=/opt/conda MINICONDA_VERSION=4.8.2 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=172.20.77.1 PORT_RANGE=0..0 KERNEL_USERNAME=guest KERNEL_LANGUAGE=python CONDA_VERSION=4.8.2 NB_USER=jovyan KUBERNETES_SERVICE_HOST=172.20.77.1 LC_ALL=en_US.UTF-8 KUBERNETES_PORT=tcp://172.20.77.1:443 KUBERNETES_PORT_443_TCP_PORT=443 PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/bin GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_PORT=tcp://172.20.18.60:4040 GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_PORT_4040_TCP_ADDR=172.20.18.60 PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDSQP9YFtzoY1v+VwYXd09x/fNEDSFIASwjoAoNA5jiOAKQujgw/xxBge1SnovvlGDjOFkkuK1bfRvECYnHafM98hRGlRVGXzbbw5d6hDHUXQMdXgh1JQJFAV8vMI6o3Sqm3ZJRodYuUDvPbbJRNhSbQEEVuzZN5R5p382gxUUFTQIDAQAB GUEST_1F146A87_8304_41D7_9193_8584C02CF412_UI_SVC_SERVICE_HOST=172.20.18.60 DEBIAN_FRONTEND=noninteractive KERNEL_NAMESPACE=ns-spark-apps _=/usr/bin/env
[D 2022-10-12 08:39:46,466.466 launch_ipykernel] Using connection file '/tmp/kernel-4b6d95f4-241d-4644-b622-f6ff4b54814a_v5j3dkkk.json'.
[I 2022-10-12 08:39:46,470.470 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 39427
Traceback (most recent call last):
File "/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py", line 616, in 
connection_file, response_addr, lower_port, upper_port, kernel_id, public_key
File "/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py", line 269, in return_connection_info
s.connect((response_ip, response_port))
ConnectionRefusedError: [Errno 111] Connection refused

I will test if dropping envoy proxies does help.

tafaust commented 2 years ago

I can confirm that the issue is due to istio, i.e. the envoy proxy sidecars. I guess that the hotfix for now is to not deploy it with istio.

@kevin-bates Is there interest from the enterprise-gateway maintainer side to support istio?

kevin-bates commented 2 years ago

I can confirm that the issue is due to istio

Great news.

Is there interest from the enterprise-gateway maintainer side to support istio?

We are always interested in supporting configurations our users need. That said, I don't think any of the current maintainers have the bandwidth and/or resources to take this on - so istio's support would need to come in the form of a contribution.

Regarding your previous troubleshooting, setting the env EG_RESPONSE_ADDRESS isn't going to do anything but set that env value in the EG pod. Nothing uses that env. Instead, you should set EG_RESPONSE_IP and EG_RESPONSE_PORT (although the latter is the default of 8877 so its initialization is not necessary). However, since using EG_PROHIBITED_LOCAL_IPS to mask out 10.100.*.* is still producing a response IP of 10.100.n.n indicates that EG cannot find any other local IPs and needs to fallback to the one it finds.

When you got this working by removing Istio from the equation, what kind of response address was computed?

tafaust commented 2 years ago

We are always interested in supporting configurations our users need. That said, I don't think any of the current maintainers have the bandwidth and/or resources to take this on - so istio's support would need to come in the form of a contribution.

Maybe https://github.com/splunk/jupyterhub-istio-proxy can serve as a blueprint for implementation. I have a tight schedule but maybe I can pour some time into it. I will have a look at https://jupyter-enterprise-gateway.readthedocs.io/en/latest/contributors/devinstall.html and try to come up with a PR that extends the proxies ability to support istio's envoy sidecar reverse proxies.

Do you want to keep this issue open or start a separate issue for the istio service mesh extension?

I also noticed that if enterprise-gateway runs outside of the istio service mesh, it won't even start python based spark-operator kernels (when spark-operator is running in the service mesh). It won't start a spark driver.

However, since using EG_PROHIBITED_LOCAL_IPS to mask out 10.100.. is still producing a response IP of 10.100.n.n indicates that EG cannot find any other local IPs and needs to fallback to the one it finds.

That fallback is a really nice to know! Thank you! :)

When you got this working by removing Istio from the equation, what kind of response address was computed?

It is an IP within the same /16 subnet, despite the EG_PROHIBITED_LOCAL_IPS set to 10.100.*.*:

+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id 443cbb3d-f70f-44a5-8670-3ca57938ccd6 --port-range 0..0 --response-address 10.100.21.198:8877 --public-key MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCrzyqh/7jryyFFvLJ20XDI1rsGdatlROT7in70oJCfR2F6FEhwdexv1cVleM6OTTN8NLbvZnUPk+lOKuYxrfNqjJO9wqEd27hM/MtbYPvL5e5v92LH5xiaagWdI7KQfWQfH1t3vnZ4PtoJsxb45ZQvIiDg0vSMjw8NxWhDZpeOxwIDAQAB --spark-context-initialization-mode none
kevin-bates commented 2 years ago

try to come up with a PR that extends the proxies ability to support istio's envoy sidecar reverse proxies.

I don't know anything about Istio, but this implies it gets involved in intra-cluster communications (between pods). Is that correct?

I was hoping this could be something that is configured (either via helm) or within the kernel-pod.yaml used to launch the kernel pods and not require "source code" changes.

tafaust commented 2 years ago

I don't know anything about Istio, but this implies it gets involved in intra-cluster communications (between pods). Is that correct?

Yes, istio is basically spawns a sidecar when instructed and handles communication between pods through envoy proxies (thus allowing for secure and traceable transmission).

I was hoping this could be something that is configured (either via helm) or within the kernel-pod.yaml used to launch the kernel pods and not require "source code" changes.

I can promise that I'll look for the least invasive solution that makes enterprise-gateway with istio work. I'm also puzzled why it doesn't work in the first place because the communication between pods is merely tunneled through the proxies AFAIU.

I currently look into solutions that target the helm/declarations:

kevin-bates commented 2 years ago

Let's keep this issue open. I've gone ahead and amended the title to include the istio context. Thank you for your help.

tafaust commented 2 years ago

@kevin-bates I tried to set

annotations:
  proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }
  traffic.sidecar.istio.io/excludeOutboundPorts: "8877"

on the kernel pod but without success. I will try to debug the network connection from enterprise-gateway through envoy proxies to the kernel pod and vice versa and get back with the results.

tafaust commented 2 years ago

@kevin-bates I did some tests on a clean cluster with EG-unrelated pods/services and monitored the istio traffic. AFAIU istio does pod to pod communication via pod -> svc (apparently only Endpoints according to https://discuss.istio.io/t/503-between-pod-to-pod-communication-1-5-1/6121/15) -> pod becausae istio does not have the information to route between the pods (contrary to kubedns) it seems (for whatever reason).

It makes sense given that the launch_kubernetes.py script is able to communicate with the kubernetes api (probably ClusterIP) when it creates the kernel pod. However, it means that EG_PROHIBITED_LOCAL_IPS set to 10.100.*.* had no effect (in the istio scenario).

Hence, my proposal for now is to add another k8s resource (Service) to the kubernetes deployments and make it configurable, i.e. only deploy the Service when istio is configured (or we could additionally attempt to auto-detect it by looking at the enterprise-gateway namespace annotations).

What do you think? I'll test it meanwhile in my cluster and post some code to my ideas asap.

EDIT: https://istio.io/latest/docs/ops/deployment/requirements/#pod-requirements specifically states:

To be part of a mesh, Kubernetes pods must satisfy the following requirements:

  • Service association: A pod must belong to at least one Kubernetes service even if the pod does NOT expose any port. If a pod belongs to multiple Kubernetes services, the services cannot use the same port number for different protocols, for instance HTTP and TCP.

Is there any way to nicely debug enterprise-gateway during development? I've worked myself through https://jupyter-enterprise-gateway.readthedocs.io/en/latest/contributors/system-architecture.html and https://jupyter-enterprise-gateway.readthedocs.io/en/latest/contributors/devinstall.html but the development process is still kind of slow. make run-dev throws an error ERROR: jupyter_enterprise_gateway-*.whl is not a valid wheel filename. whereas I've not done any building before and was not able to find anything in the Makefile.

kevin-bates commented 2 years ago

Hi @tahesse.

What do you think? I'll test it meanwhile in my cluster and post some code to my ideas asap.

I think a configurable approach is ideal, one that we can easily document and enable via helm deployments - thank you.

Service association: A pod must belong to at least one Kubernetes service even if the pod does NOT expose any port.

What does it mean to "belong to a Kubernetes service"? I'm assuming this implies the Service and the (kernel) pod must reside in the same namespace. Since kernel pods are primarily run in namespaces outside of EG's, does this imply that each launch of the kernel will result in the creation of its own service? And, if folks are specifying their own kernel namespace (via KERNEL_NAMESPACE), then we would only create a service if one doesn't already exist?

Is there any way to nicely debug enterprise-gateway during development?

Sorry for the hassles here. I use a Mac and run Rancher Desktop for my k8s development. My typically iteration is:

  1. Make source changes.
  2. Run make clean dist enterprise-gateway to build the elyra/enterprise-gateway:dev image (clean probably isn't necessary, but I'm paranoid. :smile:)
  3. If I need to make a launcher change, build the appropriate kernel image make kernel-py to build elyra/kernel-py:dev
  4. Uninstall the previous deployment: helm delete enterprise-gateway -n enterprise-gateway
  5. Deploy the updates from the github directory: helm upgrade --install enterprise-gateway etc/kubernetes/helm/enterprise-gateway -n enterprise-gateway
  6. Test and Repeat

I create aliases for the helm deployments...

alias eg_deploy='helm upgrade --install  enterprise-gateway etc/kubernetes/helm/enterprise-gateway -n enterprise-gateway'
alias eg_remove='helm delete enterprise-gateway -n enterprise-gateway'

and another to tail the EG logs...

alias eg_logs='kubectl logs -f deployment.apps/enterprise-gateway -n enterprise-gateway'

If others have an easier workflow, I'd love to hear from you.

tafaust commented 2 years ago

What does it mean to "belong to a Kubernetes service"?

I think they refer to association through label and selector between pod/deployment and service.

Thanks for posting your workflow, that helps a lot for sure! I'm running my stuff on silicon Mac M1 which is quite the hassle with arm VS amd arch. My Kubernetes (EKS) was partitioned for. multi-tenant use using Loft. Roughly the same as Rancher I think, that is k8s in docker (kind).

TYSM!

So far, I had to modify few python files to get some "istio_enabled" pivot which can be leveraged to spawn an additional headless k8s service, I'll do some testing tomorrow with the workflow steps you posted. Thanks again!

tafaust commented 2 years ago

What do you think about having a kubernetes Service for the kernel pods at all times? And in the case of istio, i.e. service discovery outside of kube-proxy, make the Service headless?

Read kubernetes best practices for more information: https://kubernetes.io/docs/concepts/configuration/overview/#services

So far, my changes do not seem to be working with the headless service. As far as I understand, the process hangs at

grafik

I don't quite understand if the socket connection might be an issue and moreover, it looks like there are socket connections on both sides which is quite confusing.

I am not sure whether that is due to the missing ports (https://istio.io/latest/docs/ops/configuration/traffic-management/traffic-routing/#headless-services) which are generated within the launch_ipython.py (there is no chance I can know the ports before they're dynamically generated) or due to the underlying communication (route issues like missing routes for the envoy proxies).

Needs more investigation...

kevin-bates commented 2 years ago

What do you think about having a kubernetes Service for the kernel pods at all times?

You might be interested in #1181. This enables the ability to add your Service details into the kernel pod template.

Regarding the inability of the server to receive the connection information from the kernel pod, that's odd. However, one of the intentions of the single response address PR was to expose the response port outside of the EG service, but, assuming your kernel pods are within the same cluster, that shouldn't be necessary.

I'm sorry I'm not familiar with Istio.

tafaust commented 2 years ago

RE. #1181 the ownerReferences is really nice! While testing, I was deleting the services manually (postponing the cleanup until I get something working). I adapted the service part with the ownerReference. Thanks!

Regarding the inability of the server to receive the connection information from the kernel pod, that's odd. However, one of the intentions of the single response address PR was to expose the response port outside of the EG service, but, assuming your kernel pods are within the same cluster, that shouldn't be necessary.

Fair enough, the introduced change regarding single response address sounds good and I see the problem with istio here.

I will now try to replace all IP with service (i.e. going through DNS rather than IP) communications. Somehow enterprise-gateway is picking up on the envoy proxies but envoy reverse proxies block the communication it seems. Normally, istio should allow that communication to happen if it bypasses the envoy reverse proxies.

govindhi commented 7 months ago

Hi, Any update here how to get enterprise gateway work with istio? If any one has solved can you please provide details?

tafaust commented 2 months ago

Hi, Any update here how to get enterprise gateway work with istio? If any one has solved can you please provide details?

Hey, maybe https://github.com/kubeflow/spark-operator/issues/1652 provides some help to you? I left my old job and am not working on this issue anymore.