Kubeflow dashboard access failure after Kubeflow 1.7 -> 1.8 Upgrade

selcem-artan commented 8 months ago

Bug Description

Initially, Kubeflow 1.7 was deployed and UI access was verified. Later with upgrade to Kubeflow 1.8, dashboard access does not work.

$ microk8s kubectl -n kubeflow get svc istio-ingressgateway-workload -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 172.27.80.162

$ juju config oidc-gatekeeper public-url http://172.27.80.162.nip.io

ubuntu@node06:~$ juju config dex-auth public-url http://172.27.80.162.nip.io

$ curl -v http://172.27.80.162.nip.io

Trying 172.27.80.162:80...
Connected to 172.27.80.162.nip.io (172.27.80.162) port 80 (#0)

GET / HTTP/1.1 Host: 172.27.80.162.nip.io User-Agent: curl/7.81.0 Accept: /
Mark bundle as not supporting multiuse < HTTP/1.1 404 Not Found < date: Wed, 21 Feb 2024 12:18:25 GMT < server: istio-envoy < content-length: 0 <
Connection #0 to host 172.27.80.162.nip.io left intact

To Reproduce

Charm versions as below;

dex-auth 2.36/stable istio-gateway 1.17/stable istio-pilot 1.17/stable

Environment

Juju v3.1, Microk8s v1.26, Kubeflow v1.8

Relevant Log Output

kubectl -n kubeflow logs oidc-gatekeeper-0 -c oidc-authservice returns;

2024-02-21T12:16:50.655Z [oidc-authservice] time="2024-02-21T12:16:50Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: oidc: issuer did not match the issuer returned by provider, expected \"http://172.27.80.162.nip.io/dex\" got \"http:///dex\""
2024-02-21T12:17:00.658Z [oidc-authservice] time="2024-02-21T12:17:00Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: oidc: issuer did not match the issuer returned by provider, expected \"http://172.27.80.162.nip.io/dex\" got \"http:///dex\""
2024-02-21T12:17:10.661Z [oidc-authservice] time="2024-02-21T12:17:10Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: oidc: issuer did not match the issuer returned by provider, expected \"http://172.27.80.162.nip.io/dex\" got \"http:///dex\""
2024-02-21T12:17:20.664Z [oidc-authservice] time="2024-02-21T12:17:20Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: oidc: issuer did not match the issuer returned by provider, expected \"http://172.27.80.162.nip.io/dex\" got \"http:///dex\""

Additional Context

No response

syncronize-issues-to-jira[bot] commented 8 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5359.

This message was autogenerated

NohaIhab commented 8 months ago

hi @selcem-artan the logs you've attached are different from what we discussed offline, from these logs authservice can reach dex, but looks like dex url is not configured correctly in this case, hence the http:///dex\ Can you show the dex-auth configuration now with juju config dex-auth public-url?

selcem-artan commented 8 months ago

$ juju config dex-auth public-url http://172.27.80.162.nip.io

selcem-artan commented 8 months ago

Hello Noha,

I realized a lot of Virtualservices is missing in upgraded KUbeflow 1.8 compared to freshly installed Kubeflow 1.8

FReshly installed KUbeflow 1.8: NAME GATEWAYS HOSTS AGE dex-auth ["kubeflow/kubeflow-gateway"] [""] 12m envoy ["kubeflow/kubeflow-gateway"] [""] 12m jupyter-ui ["kubeflow/kubeflow-gateway"] [""] 12m katib-ui ["kubeflow/kubeflow-gateway"] [""] 12m kfp-ui ["kubeflow/kubeflow-gateway"] [""] 12m kubeflow-dashboard ["kubeflow/kubeflow-gateway"] [""] 12m kubeflow-volumes ["kubeflow/kubeflow-gateway"] [""] 12m oidc-gatekeeper ["kubeflow/kubeflow-gateway"] [""] 12m tensorboards-web-app ["kubeflow/kubeflow-gateway"] ["*"] 12m

Upgraded to 1.8: ubuntu@node06:~$ microk8s kubectl get virtualservice -nkubeflow NAME GATEWAYS HOSTS AGE kfp-ui ["kubeflow/istio-gateway"] [""] 6d6h envoy ["kubeflow/istio-gateway"] [""] 6d5h tensorboards-web-app ["kubeflow/istio-gateway"] [""] 7h32m dex-auth ["kubeflow/istio-gateway"] [""] 149m

selcem-artan commented 8 months ago

Problems were resolved, after fixing istio relations of below missing components. THanks a lot for yout help

ubuntu@node06:~$ microk8s kubectl get virtualservice -nkubeflow NAME GATEWAYS HOSTS AGE kfp-ui ["kubeflow/istio-gateway"] [""] 6d6h envoy ["kubeflow/istio-gateway"] [""] 6d5h tensorboards-web-app ["kubeflow/istio-gateway"] [""] 7h41m dex-auth ["kubeflow/istio-gateway"] [""] 159m kubeflow-dashboard ["kubeflow/istio-gateway"] [""] 3m51s kubeflow-volumes ["kubeflow/istio-gateway"] [""] 2m46s oidc-gatekeeper ["kubeflow/istio-gateway"] [""] 93s jupyter-ui ["kubeflow/istio-gateway"] [""] 24s

canonical / bundle-kubeflow