search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
104 stars 50 forks source link

Dex-auth failing to start on k8s 1.21 #347

Closed asbalderson closed 3 years ago

asbalderson commented 3 years ago

When deploying the kueflow bundle on k8s 1.20 there are no problems; but when switching to 1.21, the dex-auth charm is failing to come up:

on both k8s versions the cs:kubeflow bundle is deployed alongside an overlay setting the user and password for dex-auth to "admin"

the juju logs show: DEBUG:root:dex-auth/0 workload status is error since 2021-04-20 22:02:09+00:00 and the logs for the dex-auth pod shows: 2021-04-28T04:04:21.053911834Z stderr F time="2021-04-28T04:04:21Z" level=info msg="config using log level: debug" 2021-04-28T04:04:21.053958204Z stderr F time="2021-04-28T04:04:21Z" level=info msg="config issuer: /dex" 2021-04-28T04:04:21.054245138Z stderr F failed to initialize storage: failed to inspect service account token: jwt claim "kubernetes.io/serviceaccount/namespace" not found

(thats the whole log)

I can provide a crashdump with more logs around the pods running in the k8s 1.21 if needed.

brendanobra commented 3 years ago

I'm also encountering this. I believe it is because the pod is configured to mount a serviceaccount token that was never created:

      name: kube-api-access-mwpfn

...

    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-mwpfn
      readOnly: true

kubectl -n kubeflow get secret | grep kube-api-access-mwpfn returns nothing ^^^

brendanobra commented 3 years ago

I believe this is the underlying issue: https://github.com/dexidp/dex/issues/2082 So not really a kubeflow issue

DomFleischmann commented 3 years ago

This issue has been fixed and is available on the latest kubeflow bundles.