search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
104 stars 50 forks source link

Issue with kubeflow UI #450

Closed soumen-ghosh closed 2 years ago

soumen-ghosh commented 2 years ago

I've installaed kubeflow-lite bundle through juju. And followed steps mentioned in https://discourse.ubuntu.com/t/install-kubeflow-anywhere/20207 to configure the UI.

Say, the external ip of istio-ingressgateway service is 10.64.140.43

juju config dex-auth public-url=http://10.64.140.43.nip.io ## I've tried both with xip.io or nip.io (https://charmed-kubeflow.io/docs/install) juju config oidc-gatekeeper public-url=http://10.64.140.43.nip.io ## I've tried both with xip.io or nip.io

But in the browser the page does not open. I have even tried opening the browser on the master node and tried - that too does not work.

How do I check what the issue could be? Can I choose some other domain instead of xip.io or nip.io in case those are getting blocked by IT security? What other option (like sslip.io) can I try to access the UI from any machine?

Regards, ~ Soumen

EMCP commented 2 years ago

you might be experiencing this issue , https://github.com/canonical/bundle-kubeflow/issues/438 . some have gotten further with the full juju deploy kubeflow --trust

ca-scribner commented 2 years ago

setting the dex and oidc-gatekeeper public-url's looks right. To debug, I'd look at (and you can post here if you look) juju status and kubectl get all -A. What I'm wondering is if all applications for kubeflow have deployed successfully (in Active state and their associated pods are running normally).

If any are not as you'd expect, I suggest looking at juju debug-log -i unit-that-is-having-trouble or the kubectl logs pod-that-is-having-trouble.

If it is like @EMCP wonders where some applications are stuck in an imagePullBackoff, please respond back as I am not sure why that's happening.

psakamoori commented 2 years ago

Seeing same issue of not able to access live pages on dashboard

Screenshot from 2022-06-03 00-31-34

===== logs

psakamoori@pvlab:~/.kube$ juju status Model Controller Cloud/Region Version SLA Timestamp kubeflow microk8s-localhost microk8s/localhost 2.9.31 unsupported 00:30:28-07:00

App Version Status Scale Charm Channel Rev Address Exposed Message admission-webhook res:oci-image@fc124ea active 1 admission-webhook stable 12 10.152.183.101 no
argo-controller res:oci-image@0eec3c1 active 1 argo-controller stable 55 no
dex-auth res:oci-image@a74f783 waiting 2/1 dex-auth 2.28/stable 78 10.152.183.221 no Waiting for leadership envoy res:oci-image@b4adee5 active 1 envoy stable 6 10.152.183.204 no
istio-ingressgateway res:oci-image@aae58cf active 1 istio-gateway 1.5/stable 40 no
istio-pilot res:oci-image@87fc646 active 1 istio-pilot 1.5/stable 61 10.152.183.124 no
jupyter-controller res:oci-image@62a1ccf active 1 jupyter-controller stable 61 no
jupyter-ui res:oci-image@5536a2d active 1 jupyter-ui stable 21 10.152.183.102 no
kfp-api res:oci-image@81e784a active 1 kfp-api stable 33 10.152.183.155 no
kfp-db mariadb/server:10.3 active 1 charmed-osm-mariadb-k8s stable 35 10.152.183.181 no ready kfp-persistence res:oci-image@1012943 active 1 kfp-persistence stable 29 no
kfp-profile-controller res:oci-image@14ec522 active 1 kfp-profile-controller stable 16 10.152.183.232 no
kfp-schedwf res:oci-image@34e7e9e active 1 kfp-schedwf stable 32 no
kfp-ui res:oci-image@b67a29c active 1 kfp-ui stable 32 10.152.183.156 no
kfp-viewer res:oci-image@c208ebd active 1 kfp-viewer stable 31 no
kfp-viz res:oci-image@13c46cf active 1 kfp-viz stable 28 10.152.183.210 no
kubeflow-dashboard res:oci-image@858a90f active 1 kubeflow-dashboard stable 64 10.152.183.35 no
kubeflow-profiles res:profile-image@f4450cf active 1 kubeflow-profiles stable 57 10.152.183.242 no
kubeflow-roles waiting 1 kubeflow-roles stable 1 10.152.183.159 no installing agent kubeflow-volumes res:oci-image@fedee0e active 1 kubeflow-volumes stable 11 10.152.183.44 no
metacontroller-operator active 1 metacontroller-operator stable 2 10.152.183.11 no
minio res:oci-image@1755999 active 1 minio stable 57 10.152.183.9 no
mlmd res:oci-image@e2cb9ce active 1 mlmd stable 10 10.152.183.48 no
oidc-gatekeeper res:oci-image@4e7f8dd active 1 oidc-gatekeeper stable 57 10.152.183.168 no
seldon-controller-manager res:oci-image@047f2fc active 1 seldon-core stable 52 10.152.183.20 no
training-operator active 1 training-operator stable 6 10.152.183.130 no

Unit Workload Agent Address Ports Message admission-webhook/0 active idle 10.1.131.75 4443/TCP
argo-controller/0 active idle 10.1.131.118
dex-auth/8 error idle 10.1.131.122 5556/TCP hook failed: "oidc-client-relation-broken" dex-auth/9 waiting idle 10.1.122.194 5556/TCP Waiting for leadership envoy/0 active idle 10.1.131.111 9901/TCP,9090/TCP
istio-ingressgateway/0 active idle 10.1.131.116 15020/TCP,80/TCP,443/TCP,15029/TCP,15030/TCP,15031/TCP,15032/TCP,15443/TCP,15011/TCP,8060/TCP,853/TCP
istio-pilot/0 active idle 10.1.131.86 8080/TCP,15010/TCP,15012/TCP,15017/TCP
jupyter-controller/0 active idle 10.1.131.84
jupyter-ui/0 active idle 10.1.131.89 5000/TCP
kfp-api/0 active idle 10.1.131.119 8888/TCP,8887/TCP
kfp-db/0 active idle 10.1.131.88 3306/TCP ready kfp-persistence/0 active idle 10.1.131.120
kfp-profile-controller/0 active idle 10.1.131.117 80/TCP
kfp-schedwf/0 active idle 10.1.131.93
kfp-ui/0 active idle 10.1.131.121 3000/TCP
kfp-viewer/0 active idle 10.1.131.97
kfp-viz/0 active idle 10.1.131.101 8888/TCP
kubeflow-dashboard/0 active idle 10.1.131.114 8082/TCP
kubeflow-profiles/0 active idle 10.1.131.104 8080/TCP,8081/TCP
kubeflow-roles/0 maintenance idle 10.1.131.98
kubeflow-volumes/0 active idle 10.1.131.105 5000/TCP
metacontroller-operator/0 active idle 10.1.131.100
minio/0 active idle 10.1.131.109 9000/TCP
mlmd/0 active idle 10.1.131.112 8080/TCP
oidc-gatekeeper/2 active idle 10.1.122.195 8080/TCP
seldon-controller-manager/0 active idle 10.1.131.110 8080/TCP,4443/TCP
training-operator/0 active idle 10.1.131.108

psakamoori@pvlab:~/.kube$ microk8s kubectl get all -A NAMESPACE NAME READY STATUS RESTARTS AGE admin pod/ml-pipeline-visualizationserver-569ccd5d86-5snq7 1/1 Running 0 2d10h kubeflow pod/minio-operator-0 1/1 Running 0 2d11h kubeflow pod/kfp-db-operator-0 1/1 Running 0 2d11h kubeflow pod/kfp-profile-controller-7998fb4d7b-66lzr 1/1 Running 0 2d11h kubeflow pod/kfp-schedwf-operator-0 1/1 Running 0 2d11h kubeflow pod/kfp-viewer-operator-0 1/1 Running 0 2d11h kubeflow pod/admission-webhook-operator-0 1/1 Running 0 2d11h kubeflow pod/kfp-api-operator-0 1/1 Running 0 2d11h kubeflow pod/envoy-operator-0 1/1 Running 0 2d11h kubeflow pod/kubeflow-profiles-operator-0 1/1 Running 0 2d11h kubeflow pod/argo-controller-operator-0 1/1 Running 0 2d11h kubeflow pod/jupyter-controller-operator-0 1/1 Running 0 2d11h kubeflow pod/kubeflow-dashboard-5b97bc47fd-4r4h4 1/1 Running 0 2d11h kubeflow pod/kfp-viewer-f7fc56dbd-p4pcr 1/1 Running 0 2d11h kubeflow pod/jupyter-ui-operator-0 1/1 Running 0 2d11h kubeflow pod/minio-0 1/1 Running 0 2d11h kubeflow pod/mlmd-operator-0 1/1 Running 0 2d11h kubeflow pod/kfp-persistence-operator-0 1/1 Running 0 2d11h kubeflow pod/jupyter-ui-6669f65f4f-qkcgt 1/1 Running 0 2d11h kubeflow pod/kfp-persistence-655f446d47-7vbhl 1/1 Running 0 2d11h kubeflow pod/kfp-viz-operator-0 1/1 Running 0 2d11h kubeflow pod/istio-ingressgateway-operator-0 1/1 Running 0 2d11h kubeflow pod/kfp-profile-controller-operator-0 1/1 Running 0 2d11h kubeflow pod/envoy-675b94f98f-5mh4q 1/1 Running 0 2d11h controller-microk8s-localhost pod/modeloperator-5fd54dccc8-hhsjc 1/1 Running 0 2d11h kubeflow pod/jupyter-controller-7587c447dd-4qhc2 1/1 Running 0 2d11h kubeflow pod/oidc-gatekeeper-operator-0 1/1 Running 0 2d11h admin pod/ml-pipeline-ui-artifact-77dfb58d8b-prd9z 1/1 Running 0 2d10h kubeflow pod/kfp-schedwf-566479d99c-5xsqf 1/1 Running 0 2d11h kubeflow pod/kubeflow-profiles-6767c8ff5c-hwsc6 2/2 Running 0 2d11h kubeflow pod/kubeflow-dashboard-operator-0 1/1 Running 0 2d11h kubeflow pod/dex-auth-operator-0 1/1 Running 0 2d11h kubeflow pod/seldon-controller-manager-operator-0 1/1 Running 0 2d11h kubeflow pod/admission-webhook-56bff57bb9-nwzc8 1/1 Running 0 2d11h kubeflow pod/metacontroller-operator-charm-0 1/1 Running 0 2d11h kubeflow pod/kfp-ui-operator-0 1/1 Running 0 2d11h kubeflow pod/istio-pilot-operator-0 1/1 Running 0 2d11h kubeflow pod/modeloperator-599b895d88-q67wh 1/1 Running 0 2d11h kubeflow pod/kubeflow-volumes-6bbd577bbc-vrdq8 1/1 Running 0 2d11h kubeflow pod/kubeflow-volumes-operator-0 1/1 Running 0 2d11h kube-system pod/calico-kube-controllers-f7868dd95-bqx4b 1/1 Running 0 2d12h ingress pod/nginx-ingress-microk8s-controller-zz2sk 1/1 Running 0 2d12h kube-system pod/coredns-7f9c69c78c-brsjl 1/1 Running 1 2d12h controller-microk8s-localhost pod/controller-0 2/2 Running 2 2d11h kubeflow pod/kfp-api-d6775464d-4cdx5 1/1 Running 0 2d11h kubeflow pod/mlmd-0 1/1 Running 0 2d11h kubeflow pod/kubeflow-roles-0 1/1 Running 1 2d11h kubeflow pod/metacontroller-operator-0 1/1 Running 1 2d11h kubeflow pod/kfp-viz-cb599cfb5-xv9bp 1/1 Running 0 2d11h kubeflow pod/kfp-ui-7f877ccdd8-prghv 1/1 Running 0 2d11h kubeflow pod/training-operator-0 2/2 Running 0 2d11h kubeflow pod/istio-ingressgateway-56b7b475f7-v54pp 1/1 Running 0 2d11h kubeflow pod/istio-pilot-d6f78d9f-wzh9z 1/1 Running 0 2d11h kubeflow pod/kfp-db-0 1/1 Running 0 2d11h kubeflow pod/argo-controller-5644cf7766-sbthm 1/1 Running 6 2d11h kube-system pod/hostpath-provisioner-566686b959-62j5k 1/1 Running 8 2d12h kubeflow pod/seldon-controller-manager-76f96c7b95-shsqr 1/1 Running 12 2d11h kube-system pod/calico-node-8jdq7 1/1 Running 0 73m kube-system pod/calico-node-t5jmx 1/1 Running 2 72m ingress pod/nginx-ingress-microk8s-controller-vgpv5 1/1 Running 2 71m kubeflow pod/oidc-gatekeeper-78f4d754f7-bjrp2 1/1 Running 0 20m kubeflow pod/dex-auth-7d786b6fb7-97wrv 1/1 Running 0 21m

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.152.183.1 443/TCP 7d3h kube-system service/kube-dns ClusterIP 10.152.183.10 53/UDP,53/TCP,9153/TCP 3d8h controller-microk8s-localhost service/controller-service ClusterIP 10.152.183.179 17070/TCP 2d11h controller-microk8s-localhost service/modeloperator ClusterIP 10.152.183.63 17071/TCP 2d11h kubeflow service/modeloperator ClusterIP 10.152.183.138 17071/TCP 2d11h kubeflow service/admission-webhook-operator ClusterIP 10.152.183.177 30666/TCP 2d11h kubeflow service/argo-controller-operator ClusterIP 10.152.183.158 30666/TCP 2d11h kubeflow service/dex-auth-operator ClusterIP 10.152.183.51 30666/TCP 2d11h kubeflow service/admission-webhook ClusterIP 10.152.183.101 4443/TCP 2d11h kubeflow service/envoy-operator ClusterIP 10.152.183.109 30666/TCP 2d11h kubeflow service/istio-ingressgateway-operator ClusterIP 10.152.183.167 30666/TCP 2d11h kubeflow service/istio-pilot-operator ClusterIP 10.152.183.145 30666/TCP 2d11h kubeflow service/jupyter-controller-operator ClusterIP 10.152.183.113 30666/TCP 2d11h kubeflow service/jupyter-ui-operator ClusterIP 10.152.183.29 30666/TCP 2d11h kubeflow service/dex-auth ClusterIP 10.152.183.221 5556/TCP 2d11h kubeflow service/kfp-api-operator ClusterIP 10.152.183.250 30666/TCP 2d11h kubeflow service/kfp-db-operator ClusterIP 10.152.183.82 30666/TCP 2d11h kubeflow service/kfp-persistence-operator ClusterIP 10.152.183.83 30666/TCP 2d11h kubeflow service/istio-pilot ClusterIP 10.152.183.124 8080/TCP,15010/TCP,15012/TCP,15017/TCP 2d11h kubeflow service/kfp-profile-controller-operator ClusterIP 10.152.183.180 30666/TCP 2d11h kubeflow service/kfp-db ClusterIP 10.152.183.181 3306/TCP 2d11h kubeflow service/kfp-db-endpoints ClusterIP None 2d11h kubeflow service/jupyter-ui ClusterIP 10.152.183.102 5000/TCP 2d11h kubeflow service/kfp-schedwf-operator ClusterIP 10.152.183.57 30666/TCP 2d11h kubeflow service/kfp-ui-operator ClusterIP 10.152.183.144 30666/TCP 2d11h kubeflow service/kfp-viewer-operator ClusterIP 10.152.183.19 30666/TCP 2d11h kubeflow service/kfp-viz-operator ClusterIP 10.152.183.98 30666/TCP 2d11h kubeflow service/kubeflow-dashboard-operator ClusterIP 10.152.183.238 30666/TCP 2d11h kubeflow service/kubeflow-profiles-operator ClusterIP 10.152.183.205 30666/TCP 2d11h kubeflow service/kubeflow-roles ClusterIP 10.152.183.159 65535/TCP 2d11h kubeflow service/kubeflow-roles-endpoints ClusterIP None 2d11h kubeflow service/kubeflow-volumes-operator ClusterIP 10.152.183.112 30666/TCP 2d11h kubeflow service/metacontroller-operator ClusterIP 10.152.183.11 65535/TCP 2d11h kubeflow service/metacontroller-operator-endpoints ClusterIP None 2d11h kubeflow service/kfp-viz ClusterIP 10.152.183.210 8888/TCP 2d11h kubeflow service/minio-operator ClusterIP 10.152.183.214 30666/TCP 2d11h kubeflow service/mlmd-operator ClusterIP 10.152.183.12 30666/TCP 2d11h kubeflow service/kubeflow-profiles ClusterIP 10.152.183.242 8080/TCP,8081/TCP 2d11h kubeflow service/kubeflow-volumes ClusterIP 10.152.183.44 5000/TCP 2d11h kubeflow service/oidc-gatekeeper-operator ClusterIP 10.152.183.122 30666/TCP 2d11h kubeflow service/training-operator ClusterIP 10.152.183.130 65535/TCP 2d11h kubeflow service/training-operator-endpoints ClusterIP None 2d11h kubeflow service/seldon-controller-manager-operator ClusterIP 10.152.183.206 30666/TCP 2d11h kubeflow service/minio ClusterIP 10.152.183.9 9000/TCP 2d11h kubeflow service/minio-endpoints ClusterIP None 2d11h kubeflow service/metadata-grpc-service ClusterIP 10.152.183.48 8080/TCP 2d11h kubeflow service/seldon-controller-manager ClusterIP 10.152.183.20 8080/TCP,4443/TCP 2d11h kubeflow service/mlmd ClusterIP 10.152.183.55 8080/TCP 2d11h kubeflow service/mlmd-endpoints ClusterIP None 2d11h kubeflow service/envoy ClusterIP 10.152.183.204 9901/TCP,9090/TCP 2d11h kubeflow service/kubeflow-dashboard ClusterIP 10.152.183.35 8082/TCP 2d11h kubeflow service/oidc-gatekeeper ClusterIP 10.152.183.168 8080/TCP 2d11h kubeflow service/istio-ingressgateway LoadBalancer 10.152.183.120 15020:31181/TCP,80:32483/TCP,443:32426/TCP,15029:31528/TCP,15030:30059/TCP,15031:30906/TCP,15032:31151/TCP,15443:32164/TCP,15011:32367/TCP,8060:30094/TCP,853:31931/TCP 2d11h kubeflow service/kfp-profile-controller ClusterIP 10.152.183.232 80/TCP 2d11h kubeflow service/ml-pipeline ClusterIP 10.152.183.155 8887/TCP,8888/TCP 2d11h kubeflow service/kfp-api ClusterIP 10.152.183.207 8888/TCP,8887/TCP 2d11h kubeflow service/kfp-ui ClusterIP 10.152.183.156 3000/TCP 2d11h kubeflow service/seldon-webhook-service ClusterIP 10.152.183.36 4443/TCP 2d11h admin service/ml-pipeline-visualizationserver ClusterIP 10.152.183.13 8888/TCP 2d10h admin service/ml-pipeline-ui-artifact ClusterIP 10.152.183.64 80/TCP 2d10h

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system daemonset.apps/calico-node 2 2 2 2 2 kubernetes.io/os=linux 7d3h ingress daemonset.apps/nginx-ingress-microk8s-controller 2 2 2 2 2 3d7h

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE admin deployment.apps/ml-pipeline-visualizationserver 1/1 1 1 2d10h controller-microk8s-localhost deployment.apps/modeloperator 1/1 1 1 2d11h admin deployment.apps/ml-pipeline-ui-artifact 1/1 1 1 2d10h kube-system deployment.apps/calico-kube-controllers 1/1 1 1 7d3h kube-system deployment.apps/coredns 1/1 1 1 3d8h kubeflow deployment.apps/modeloperator 1/1 1 1 2d11h kube-system deployment.apps/hostpath-provisioner 1/1 1 1 3d8h kubeflow deployment.apps/kfp-viz 1/1 1 1 2d11h kubeflow deployment.apps/kfp-viewer 1/1 1 1 2d11h kubeflow deployment.apps/envoy 1/1 1 1 2d11h kubeflow deployment.apps/kfp-schedwf 1/1 1 1 2d11h kubeflow deployment.apps/kfp-persistence 1/1 1 1 2d11h kubeflow deployment.apps/jupyter-controller 1/1 1 1 2d11h kubeflow deployment.apps/kfp-profile-controller 1/1 1 1 2d11h kubeflow deployment.apps/kubeflow-volumes 1/1 1 1 2d11h kubeflow deployment.apps/admission-webhook 1/1 1 1 2d11h kubeflow deployment.apps/kfp-ui 1/1 1 1 2d11h kubeflow deployment.apps/jupyter-ui 1/1 1 1 2d11h kubeflow deployment.apps/kubeflow-dashboard 1/1 1 1 2d11h kubeflow deployment.apps/argo-controller 1/1 1 1 2d11h kubeflow deployment.apps/istio-ingressgateway 1/1 1 1 2d11h kubeflow deployment.apps/kfp-api 1/1 1 1 2d11h kubeflow deployment.apps/istio-pilot 1/1 1 1 2d11h kubeflow deployment.apps/kubeflow-profiles 1/1 1 1 2d11h kubeflow deployment.apps/seldon-controller-manager 1/1 1 1 2d11h kubeflow deployment.apps/oidc-gatekeeper 1/1 1 1 2d11h kubeflow deployment.apps/dex-auth 1/1 1 1 2d11h

NAMESPACE NAME DESIRED CURRENT READY AGE admin replicaset.apps/ml-pipeline-visualizationserver-569ccd5d86 1 1 1 2d10h kubeflow replicaset.apps/kfp-profile-controller-7998fb4d7b 1 1 1 2d11h kubeflow replicaset.apps/kfp-viz-cb599cfb5 1 1 1 2d11h kubeflow replicaset.apps/kubeflow-dashboard-5b97bc47fd 1 1 1 2d11h kubeflow replicaset.apps/kfp-viewer-f7fc56dbd 1 1 1 2d11h kubeflow replicaset.apps/jupyter-ui-6669f65f4f 1 1 1 2d11h kubeflow replicaset.apps/kfp-api-d6775464d 1 1 1 2d11h kubeflow replicaset.apps/kfp-persistence-655f446d47 1 1 1 2d11h kubeflow replicaset.apps/envoy-675b94f98f 1 1 1 2d11h controller-microk8s-localhost replicaset.apps/modeloperator-5fd54dccc8 1 1 1 2d11h kubeflow replicaset.apps/jupyter-controller-7587c447dd 1 1 1 2d11h admin replicaset.apps/ml-pipeline-ui-artifact-77dfb58d8b 1 1 1 2d10h kubeflow replicaset.apps/kfp-schedwf-566479d99c 1 1 1 2d11h kubeflow replicaset.apps/kubeflow-profiles-6767c8ff5c 1 1 1 2d11h kube-system replicaset.apps/calico-kube-controllers-f7868dd95 1 1 1 2d12h kubeflow replicaset.apps/istio-pilot-d6f78d9f 1 1 1 2d11h kubeflow replicaset.apps/kfp-ui-7f877ccdd8 1 1 1 2d11h kubeflow replicaset.apps/admission-webhook-56bff57bb9 1 1 1 2d11h kubeflow replicaset.apps/istio-ingressgateway-56b7b475f7 1 1 1 2d11h kube-system replicaset.apps/coredns-7f9c69c78c 1 1 1 2d12h kubeflow replicaset.apps/modeloperator-599b895d88 1 1 1 2d11h kubeflow replicaset.apps/kubeflow-volumes-6bbd577bbc 1 1 1 2d11h kube-system replicaset.apps/hostpath-provisioner-566686b959 1 1 1 2d12h kubeflow replicaset.apps/argo-controller-5644cf7766 1 1 1 2d11h kubeflow replicaset.apps/seldon-controller-manager-76f96c7b95 1 1 1 2d11h kubeflow replicaset.apps/oidc-gatekeeper-78f4d754f7 1 1 1 20m kubeflow replicaset.apps/dex-auth-7d786b6fb7 1 1 1 21m

NAMESPACE NAME READY AGE kubeflow statefulset.apps/minio-operator 1/1 2d11h controller-microk8s-localhost statefulset.apps/controller 1/1 2d11h kubeflow statefulset.apps/kfp-db-operator 1/1 2d11h kubeflow statefulset.apps/kfp-schedwf-operator 1/1 2d11h kubeflow statefulset.apps/kfp-viewer-operator 1/1 2d11h kubeflow statefulset.apps/admission-webhook-operator 1/1 2d11h kubeflow statefulset.apps/kfp-api-operator 1/1 2d11h kubeflow statefulset.apps/envoy-operator 1/1 2d11h kubeflow statefulset.apps/kubeflow-profiles-operator 1/1 2d11h kubeflow statefulset.apps/argo-controller-operator 1/1 2d11h kubeflow statefulset.apps/jupyter-controller-operator 1/1 2d11h kubeflow statefulset.apps/jupyter-ui-operator 1/1 2d11h kubeflow statefulset.apps/minio 1/1 2d11h kubeflow statefulset.apps/mlmd-operator 1/1 2d11h kubeflow statefulset.apps/kfp-persistence-operator 1/1 2d11h kubeflow statefulset.apps/kfp-viz-operator 1/1 2d11h kubeflow statefulset.apps/istio-ingressgateway-operator 1/1 2d11h kubeflow statefulset.apps/kfp-profile-controller-operator 1/1 2d11h kubeflow statefulset.apps/oidc-gatekeeper-operator 1/1 2d11h kubeflow statefulset.apps/kfp-db 1/1 2d11h kubeflow statefulset.apps/kubeflow-dashboard-operator 1/1 2d11h kubeflow statefulset.apps/dex-auth-operator 1/1 2d11h kubeflow statefulset.apps/seldon-controller-manager-operator 1/1 2d11h kubeflow statefulset.apps/mlmd 1/1 2d11h kubeflow statefulset.apps/metacontroller-operator-charm 1/1 2d11h kubeflow statefulset.apps/kfp-ui-operator 1/1 2d11h kubeflow statefulset.apps/istio-pilot-operator 1/1 2d11h kubeflow statefulset.apps/kubeflow-volumes-operator 1/1 2d11h kubeflow statefulset.apps/training-operator 1/1 2d11h kubeflow statefulset.apps/kubeflow-roles 1/1 2d11h kubeflow statefulset.apps/metacontroller-operator 1/1 2d11h

=========== Followed below command =====

$ microk8s kubectl patch role -n kubeflow istio-ingressgateway-operator -p '{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"name":"istio-ingressgateway-operator"},"rules":[{"apiGroups":[""],"resources":[""],"verbs":["*"]}]}' role.rbac.authorization.k8s.io/istio-ingressgateway-operator patched

$ juju config dex-auth public-url=http://10.152.183.35:8082 $ juju config oidc-gatekeeper public-url=http://10.152.183.35:8082

Appreciate any inputs on this...

DomFleischmann commented 2 years ago

@psakamoori Your issue is an entirely different one, you are setting the kubeflow dasboard IP as the public-url. Ingress, authentication is all routed through istio and therefore you have to point to the istio-ingressgateway IP. In your case this should be the command:

juju config dex-auth public-url=http://10.152.183.120.nip.io
juju config oidc-gatekeeper public-url=http://10.152.183.120.nip.io

Given that 10.152.183.120 is the loadbalancer IP of the istio-ingressgateway service.

psakamoori commented 2 years ago

@DomFleischmann Thank you for the details. Corrected the juju config as above...and I am seeing below logs...not sure of why dex-auth is waiting....and dex-aut/8* error because of "hook failed: "oidc-client-relation-broken"

Screenshot from 2022-06-03 10-37-10

Screenshot from 2022-06-03 10-37-56

Screenshot from 2022-06-03 10-38-17

Screenshot from 2022-06-03 10-38-44

Still seeing dashboard not showing "Notebook server" page...

Please kindly suggest if I should create a new issue ticket...

thank you

DomFleischmann commented 2 years ago

You can solve the dex-auth error with the following command: juju resolve dex-auth --no-retry You might have to execute it multiple times until the unit with errors disappears.

Regarding the dashboard not working, please make sure to access the UI with the same IP you set in the public-url config.

I also agree that this is not the right place to discuss this as this is a separate issue, please try and follow the steps I told you and if it keeps failing please file a new issue.

ca-scribner commented 2 years ago

I think this original issue can be closed, but if I've misunderstood please reopen