search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
104 stars 50 forks source link

pipelines-api, oidc-gatekeeper, and dex-auth in CrashBackoffLoop after deploy #254

Closed jhobbs closed 3 years ago

jhobbs commented 4 years ago

This is with juju 2.8.5, deploying version 9af36dc8d1b3ee5d090cb36fea2f797c52858816 of this repo on microk8s v1.19.2.

ubuntu@rufflet:~$ microk8s.kubectl get pods --all-namespaces
NAMESPACE          NAME                                           READY   STATUS             RESTARTS   AGE
kube-system        calico-kube-controllers-847c8c99d-pjdbd        1/1     Running            0          20h
kube-system        coredns-86f78bb79c-lx89z                       1/1     Running            0          20h
kube-system        calico-node-9frzm                              1/1     Running            0          20h
kube-system        metrics-server-8bbfb4bdb-cvswm                 1/1     Running            0          20h
kube-system        kubernetes-dashboard-7ffd448895-rd8d7          1/1     Running            0          20h
kube-system        dashboard-metrics-scraper-6c4568dc68-5psvm     1/1     Running            0          20h
kube-system        hostpath-provisioner-5c65fbdb4f-xrmpw          1/1     Running            0          20h
controller-micro   controller-0                                   2/2     Running            2          20h
controller-micro   modeloperator-5d9fd66694-zp69j                 1/1     Running            0          20h
metallb-system     speaker-lljfj                                  1/1     Running            0          19h
metallb-system     controller-559b68bfd8-55hxn                    1/1     Running            0          19h
kubeflow           modeloperator-66d8cd69c6-84srb                 1/1     Running            0          76m
kubeflow           ambassador-operator-0                          1/1     Running            0          75m
kubeflow           argo-controller-operator-0                     1/1     Running            0          75m
kubeflow           argo-ui-operator-0                             1/1     Running            0          74m
kubeflow           argo-ui-54d5cfcf67-8gg62                       1/1     Running            0          74m
kubeflow           katib-ui-operator-0                            1/1     Running            0          74m
kubeflow           dex-auth-operator-0                            1/1     Running            0          74m
kubeflow           jupyter-controller-operator-0                  1/1     Running            0          74m
kubeflow           jupyter-web-operator-0                         1/1     Running            0          73m
kubeflow           katib-controller-operator-0                    1/1     Running            0          73m
kubeflow           ambassador-8bb997fbc-4vvhh                     1/1     Running            0          75m
kubeflow           katib-ui-566d5b6d9d-zcm5m                      1/1     Running            0          73m
kubeflow           katib-db-operator-0                            1/1     Running            0          73m
kubeflow           katib-db-manager-operator-0                    1/1     Running            0          73m
kubeflow           jupyter-controller-6bf6576545-z2h4s            1/1     Running            0          73m
kubeflow           katib-controller-75fc7c4677-rl5j6              1/1     Running            0          73m
kubeflow           katib-db-0                                     1/1     Running            0          73m
kubeflow           metadata-ui-operator-0                         1/1     Running            0          72m
kubeflow           pipelines-db-operator-0                        1/1     Running            0          72m
kubeflow           katib-db-manager-6478487499-stkl7              1/1     Running            0          73m
kubeflow           pipelines-persistence-operator-0               1/1     Running            0          72m
kubeflow           pipelines-visualization-operator-0             1/1     Running            0          72m
kubeflow           pytorch-operator-operator-0                    1/1     Running            0          72m
kubeflow           metadata-db-operator-0                         1/1     Running            0          72m
kubeflow           metadata-api-operator-0                        1/1     Running            0          72m
kubeflow           pipelines-visualization-5df6fc7f9b-4qct6       1/1     Running            0          72m
kubeflow           pytorch-operator-7b7fd58bcb-tktl8              1/1     Running            0          72m
kubeflow           minio-operator-0                               1/1     Running            0          72m
kubeflow           pipelines-db-0                                 1/1     Running            0          72m
kubeflow           pipelines-api-operator-0                       1/1     Running            0          71m
kubeflow           pipelines-viewer-operator-0                    1/1     Running            0          71m
kubeflow           kubeflow-profiles-operator-0                   1/1     Running            0          71m
kubeflow           metadata-db-0                                  1/1     Running            0          72m
kubeflow           minio-0                                        1/1     Running            0          71m
kubeflow           argo-controller-558457b4f5-mfl2b               1/1     Running            0          71m
kubeflow           metadata-envoy-operator-0                      1/1     Running            0          71m
kubeflow           pipelines-viewer-78f6989fc-2f26g               1/1     Running            0          71m
kubeflow           metadata-api-7558d4c879-9wc5b                  1/1     Running            1          71m
kubeflow           metadata-grpc-operator-0                       1/1     Running            0          71m
kubeflow           oidc-gatekeeper-operator-0                     1/1     Running            0          70m
kubeflow           jupyter-web-6b44f59cfb-ppfh8                   1/1     Running            0          71m
kubeflow           pipelines-scheduledworkflow-operator-0         1/1     Running            0          70m
kubeflow           metadata-ui-d496d9c8c-tg4h9                    1/1     Running            0          70m
kubeflow           metadata-grpc-657b88cb95-6ck9t                 1/1     Running            0          70m
kubeflow           metadata-envoy-9d9679df8-7ds55                 1/1     Running            0          70m
kubeflow           metacontroller-operator-0                      1/1     Running            0          70m
kubeflow           pipelines-ui-operator-0                        1/1     Running            0          70m
kubeflow           pipelines-scheduledworkflow-76ffdfbd68-j8rt4   1/1     Running            0          70m
kubeflow           seldon-core-operator-0                         1/1     Running            0          69m
kubeflow           metacontroller-cc67694b7-grqbp                 1/1     Running            0          69m
kubeflow           tf-job-operator-operator-0                     1/1     Running            0          69m
kubeflow           kubeflow-dashboard-operator-0                  1/1     Running            0          69m
kubeflow           pipelines-ui-78dd896755-kznvn                  1/1     Running            0          69m
kubeflow           seldon-core-ddcbdf6c9-vr4dj                    1/1     Running            0          69m
kubeflow           tf-job-operator-66bd5b9997-44qtp               1/1     Running            0          69m
kubeflow           kubeflow-dashboard-689ddb8844-7j9h6            1/1     Running            0          69m
kubeflow           kubeflow-profiles-7d5f66cff8-ww647             2/2     Running            5          71m
kubeflow           oidc-gatekeeper-5c54d7b849-kqjhw               0/1     CrashLoopBackOff   18         70m
kubeflow           pipelines-api-9c77b9b65-qnt8r                  0/1     CrashLoopBackOff   15         71m
kubeflow           dex-auth-6d567f8cf7-6b4zn                      0/1     CrashLoopBackOff   18         70m
kubeflow           pipelines-persistence-76944457d4-wcnsv         1/1     Running            13         71m
ubuntu@rufflet:~$ microk8s.kubectl logs oidc-gatekeeper-5c54d7b849-kqjhw -nkubeflow
time="2020-10-27T17:14:07Z" level=info msg="Starting readiness probe at 8081"
time="2020-10-27T17:14:07Z" level=fatal msg="Env variable CLIENT_SECRET missing, exiting."
ubuntu@rufflet:~$ microk8s.kubectl logs pipelines-api-9c77b9b65-qnt8r -nkubeflow
I1027 17:14:16.709378       1 client_manager.go:135] Initializing client manager
I1027 17:14:16.709513       1 config.go:46] Config DBConfig.ExtraParams not specified, skipping
F1027 17:15:16.784068       1 client_manager.go:351] Failed to create Minio bucket. Error: Access Denied.
ubuntu@rufflet:~$ date
Tue Oct 27 17:16:58 UTC 2020
ubuntu@rufflet:~$ microk8s.kubectl logs dex-auth-6d567f8cf7-6b4zn -nkubeflow
time="2020-10-27T17:15:01Z" level=info msg="config using log level: debug"
time="2020-10-27T17:15:01Z" level=info msg="config issuer: /dex"
time="2020-10-27T17:15:01Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2020-10-27T17:15:01Z" level=info msg="creating custom Kubernetes resources"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource refreshtokens.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource passwords.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource passwords.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="checking if custom resource connectors.dex.coreos.com has been created already..."
time="2020-10-27T17:15:01Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create"
time="2020-10-27T17:15:01Z" level=info msg="config storage: kubernetes"
time="2020-10-27T17:15:01Z" level=info msg="config static client: Ambassador Auth OIDC"
time="2020-10-27T17:15:01Z" level=info msg="config skipping approval screen"
failed to initialize server: server: no connectors specified
jhobbs commented 4 years ago

http://paste.ubuntu.com/p/t5khN6bchC/ same/similar issue with the bundle from the charm store

DomFleischmann commented 3 years ago

This seemed to have been an issue of the specific bundle and juju version mentioned in the issue. With this not ocurring in the latest bundle and juju version we will procede to close this issue.