Closed natalytvinova closed 8 months ago
syncronize-issues-to-jira[bot]
commented
8 months ago Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5372.
This message was autogenerated
kimwnasptd
commented
8 months ago @natalytvinova this looks like an issue with networking.
First of all, if you open you browser's dev tools and go on the network tab, what's the response of the request that tries to hit /notebook/admin/uat-2/lab? Is it a 404?
Then, could you provide us with the output of kubectl get virtualservices -n admin -o yaml? If it's a 404 then most probably it's related to the Istio Gateway and the VirtualService that registers that this path should be redirected to the Notebook's K8s Service
natalytvinova
commented
8 months ago Hi @kimwnasptd
In the dev-tools the only thing failing is:
GET http://my.domain.com/api/metrics 405 (Method Not Allowed)
Meanwhile the main request returns 200:
Request URL:
http://my.domain.com/jupyter/api/namespaces/admin/notebooks
Request Method:
GET
Status Code:
200 OKBut what I also get in jupyter-ui pod logs (container jupyter-ui):
2024-02-26T06:56:25.274Z [pebble] Check "up" failure 7921 (threshold 3): received non-20x status code 401
2024-02-26T06:56:55.266Z [jupyter-ui] 2024-02-26 06:56:55,265 | kubeflow.kubeflow.crud_backend.errors.handlers | ERROR | HTTP Exception handled: 401 Unauthorized: No user detected.
2024-02-26T06:56:55.269Z [pebble] Check "up" failure 7922 (threshold 3): received non-20x status code 401
2024-02-26T06:56:55.269Z [jupyter-ui] 127.0.0.1 - - [26/Feb/2024:06:56:55 +0000] "GET / HTTP/1.1" 401 69 "-" "Go-http-client/1.1"Here is a virtualservices yaml:
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: "2024-02-23T12:31:43Z"
generation: 1
name: notebook-admin-no-vol
namespace: admin
ownerReferences:
- apiVersion: kubeflow.org/v1beta1
blockOwnerDeletion: true
controller: true
kind: Notebook
name: no-vol
uid: 2ae69814-3d1d-4c01-ad35-0bcf5de0b3fe
resourceVersion: "62773663"
uid: 213c97c8-2f77-468b-8816-125817ed86e3
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- headers:
request:
set: {}
match:
- uri:
prefix: /notebook/admin/no-vol/
rewrite:
uri: /notebook/admin/no-vol/
route:
- destination:
host: no-vol.admin.svc.cluster.local
port:
number: 80
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: "2024-02-23T13:08:25Z"
generation: 1
name: notebook-admin-tyest
namespace: admin
ownerReferences:
- apiVersion: kubeflow.org/v1beta1
blockOwnerDeletion: true
controller: true
kind: Notebook
name: tyest
uid: 0517acfc-8600-41ea-96c3-d3ff4f7b6e6d
resourceVersion: "62802573"
uid: 5edb6893-6605-4f8f-8e86-1295c1a76956
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- headers:
request:
set: {}
match:
- uri:
prefix: /notebook/admin/tyest/
rewrite:
uri: /notebook/admin/tyest/
route:
- destination:
host: tyest.admin.svc.cluster.local
port:
number: 80
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: "2024-02-23T08:29:00Z"
generation: 1
name: notebook-admin-uat
namespace: admin
ownerReferences:
- apiVersion: kubeflow.org/v1beta1
blockOwnerDeletion: true
controller: true
kind: Notebook
name: uat
uid: 145059af-7eae-4b00-9d74-ea41e5bee7a8
resourceVersion: "62607516"
uid: 5e935a22-f789-4c71-82cf-7723dea404cb
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- headers:
request:
set: {}
match:
- uri:
prefix: /notebook/admin/uat/
rewrite:
uri: /notebook/admin/uat/
route:
- destination:
host: uat.admin.svc.cluster.local
port:
number: 80
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: "2024-02-23T08:33:10Z"
generation: 1
name: notebook-admin-uat-2
namespace: admin
ownerReferences:
- apiVersion: kubeflow.org/v1beta1
blockOwnerDeletion: true
controller: true
kind: Notebook
name: uat-2
uid: 59f63b8e-8f41-414a-980e-e68db67eeea5
resourceVersion: "62610487"
uid: f9a4ebb4-59b1-4e9a-b6a3-bc00e62b3968
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- headers:
request:
set: {}
match:
- uri:
prefix: /notebook/admin/uat-2/
rewrite:
uri: /notebook/admin/uat-2/
route:
- destination:
host: uat-2.admin.svc.cluster.local
port:
number: 80
kind: List
metadata:
resourceVersion: ""After a debug session with @kimwnasptd we found that kubeflow-gateway gateway name was actually kubeflow instead of kubeflow-gateway and some of the virtualservices we looking at the actual kubeflow gateway like kubeflow-dashboard.
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: "2024-02-01T16:24:31Z"
generation: 2
labels:
app.juju.is/created-by: istio-pilot
app.kubernetes.io/instance: istio-pilot-kubeflow
kubernetes-resource-handler-scope: ingress
name: kubeflow-dashboard
namespace: kubeflow
resourceVersion: "59745310"
uid: 33b236f3-b1dc-447d-a5bc-d1442ff71da4
spec:
gateways:
- kubeflow/kubeflow
hosts:
- '*'
http:
- match:
- uri:
prefix: /
rewrite:
uri: /
route:
- destination:
host: kubeflow-dashboard.kubeflow.svc.cluster.local
port:
number: 8082But others like the notebooks were using kubeflow-gateway
ubuntu@infra-1-medma:~$ kubectl get virtualservices -n admin -o yaml
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
creationTimestamp: "2024-02-23T12:31:43Z"
generation: 1
name: notebook-admin-no-vol
namespace: admin
ownerReferences:
- apiVersion: kubeflow.org/v1beta1
blockOwnerDeletion: true
controller: true
kind: Notebook
name: no-vol
uid: 2ae69814-3d1d-4c01-ad35-0bcf5de0b3fe
resourceVersion: "62773663"
uid: 213c97c8-2f77-468b-8816-125817ed86e3
spec:
gateways:
- kubeflow/kubeflow-gateway
hosts:
- '*'
http:
- headers:
request:
set: {}
match:
- uri:
prefix: /notebook/admin/no-vol/
rewrite:
uri: /notebook/admin/no-vol/
route:
- destination:
host: no-vol.admin.svc.cluster.local
port:
number: 80Juju config of the default-gateway for istio-pilot was set to "kubeflow". After changing it to "kubeflow-gateway" all the names were aligned and the access got restored
DnPlas
commented
8 months ago Hey @natalytvinova thanks for raising this, and thanks @kimwnasptd for the follow up.
Just to add more context and prevent this issue form happening in the future. The istio-pilot charm has a configuration option called default-gateway which is used for naming the Gateway resource, that is, whenever you deploy both istio-operators, the Gateway that gets created receives the name that you set in that option. By default, this value is istio-gateway.
In a Kubeflow deployment, many applications assume the Gateway name to be kubeflow-gateway, and that is why in the CKF bundle defintion, we set it to that value. It is very important that this value doesn't change in your CKF deployment because while some applications will be able to catch the change (like the dashboard did), user workloads (like the notebook you created) won't be able to catch this change and will just assume that there is a Gateway in the kubeflow namespace called kubeflow-gateway. This behaviour is the same upstream, we cannot really do much about it, btw.
That being said, this doesn't seem like an issue, but rather a misconfiguration, so I'm closing it. Feel free to re-open if you think this is still an issue.
Bug Description
Hi team,
I successfully created a Jupyter Notebook, logs are presented bellow. But unfortunately I get this when trying to connect to it:
To Reproduce
Environment
Kubeflow bundle 1.8 Juju 3.1.7 Charmed Kubernetes 1.28
Relevant Log Output
Additional Context
No response