search

GoogleCloudPlatform / kubeflow-distribution

Blueprints for Deploying Kubeflow on Google Cloud Platform and Anthos
Apache License 2.0
80 stars 63 forks source link

Issues connecting to notebook kernel (websockets) #403

Open willshiao opened 1 year ago

willshiao commented 1 year ago

Hi,

I was able to deploy a KubeFlow instance successfully, and I can spin up new JupyterLab instances and access them. However, I encounter an issue when I actually attempt to connect to the kernel or run anything in the notebook. I get browser console errors like the following:

WebSocket connection to wss://kubeflow.endpoints.myproject.cloud.goog/notebook/my-namespace/test-instance/api/events/subscribe failed

Everything works fine if I directly kubectl port-forward to the notebook pod. This leads me to think that an issue with websockets is occurring in either the istio service or ingress. How should I go about verifying this hypothesis and fixing this? I've searched online and found several issues with websockets and various services, but none that seem directly relevant. I believe I'm running the latest version of KubeFlow (not sure how to check the exact version).

Thanks!

gkcalat commented 1 year ago

Hi @willshiao! Could you please try the latest release v1.7.0?