search

kubeflow / manifests

A repository for Kustomize manifests
Apache License 2.0
808 stars 869 forks source link

OIDC provider setup failed and authservice-0 not ready #2064

Closed TaibiaoGuo closed 2 years ago

TaibiaoGuo commented 2 years ago

Issue description

OIDC provider setup failed and authservice-0 not ready

2021-10-24T04:04:52.871399289+08:00 time="2021-10-23T20:04:52Z" level=error msg="OIDC provider setup failed, retrying in 10 seconds: Get http://dex.auth.svc.cluster.local:5556/dex/.well-known/openid-configuration: EOF"

Steps to reproduce the issue

  1. git clone https://github.com/kubeflow/manifests.git
  2. git checkout v1.4.0
  3. Complete pre-preparation: Kubernetes ( version 1.20) with a default StorageClass and kustomize version is 3.2.0
  4. install: while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

What's the expected result?

What's the actual result?

Additional details / screenshot

NAMESPACE                      NAME                                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                      AGE
auth                           dex                                                         NodePort    10.96.148.238   <none>        5556:32000/TCP                                                               44m
cert-manager                   cert-manager                                                ClusterIP   10.96.254.219   <none>        9402/TCP                                                                     44m
cert-manager                   cert-manager-webhook                                        ClusterIP   10.96.238.126   <none>        443/TCP                                                                      44m
default                        ingress-demo-app                                            ClusterIP   10.96.180.121   <none>        80/TCP                                                                       4d6h
default                        kubernetes                                                  ClusterIP   10.96.0.1       <none>        443/TCP                                                                      4d7h
ingress-nginx                  ingress-nginx-controller                                    NodePort    10.96.61.7      <none>        80:40495/TCP,443:49086/TCP                                                   4d6h
ingress-nginx                  ingress-nginx-controller-admission                          ClusterIP   10.96.241.14    <none>        443/TCP                                                                      4d6h
istio-system                   authservice                                                 ClusterIP   10.96.23.249    <none>        8080/TCP                                                                     44m
istio-system                   cluster-local-gateway                                       ClusterIP   10.96.183.251   <none>        15020/TCP,80/TCP                                                             44m
istio-system                   istio-ingressgateway                                        NodePort    10.96.134.107   <none>        15021:43624/TCP,80:48345/TCP,443:34807/TCP,31400:31775/TCP,15443:46122/TCP   44m
istio-system                   istiod                                                      ClusterIP   10.96.98.104    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                        44m
istio-system                   knative-local-gateway                                       ClusterIP   10.96.164.140   <none>        80/TCP                                                                       44m
knative-eventing               broker-filter                                               ClusterIP   10.96.70.244    <none>        80/TCP,9092/TCP                                                              44m
knative-eventing               broker-ingress                                              ClusterIP   10.96.154.168   <none>        80/TCP,9092/TCP                                                              44m
knative-eventing               eventing-webhook                                            ClusterIP   10.96.117.243   <none>        443/TCP                                                                      44m
knative-eventing               imc-dispatcher                                              ClusterIP   10.96.91.64     <none>        80/TCP                                                                       44m
knative-serving                activator-service                                           ClusterIP   10.96.40.226    <none>        9090/TCP,8008/TCP,80/TCP,81/TCP                                              44m
knative-serving                autoscaler                                                  ClusterIP   10.96.223.7     <none>        9090/TCP,8008/TCP,8080/TCP                                                   44m
knative-serving                autoscaler-bucket-00-of-01                                  ClusterIP   10.96.221.147   <none>        8080/TCP                                                                     38m
knative-serving                controller                                                  ClusterIP   10.96.44.5      <none>        9090/TCP,8008/TCP                                                            44m
knative-serving                istio-webhook                                               ClusterIP   10.96.227.41    <none>        9090/TCP,8008/TCP,443/TCP                                                    44m
knative-serving                webhook                                                     ClusterIP   10.96.152.80    <none>        9090/TCP,8008/TCP,443/TCP                                                    44m
kube-system                    default-http-backend                                        ClusterIP   10.96.212.137   <none>        80/TCP                                                                       4d6h
kube-system                    etcd                                                        ClusterIP   None            <none>        2379/TCP                                                                     4d1h
kube-system                    kube-controller-manager-svc                                 ClusterIP   None            <none>        10257/TCP                                                                    4d1h
kube-system                    kube-dns                                                    ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP                                                       4d7h
kube-system                    kube-scheduler-svc                                          ClusterIP   None            <none>        10259/TCP                                                                    4d1h
kube-system                    kubelet                                                     ClusterIP   None            <none>        10250/TCP,10255/TCP,4194/TCP                                                 4d1h
kube-system                    metrics-server                                              ClusterIP   10.96.136.22    <none>        443/TCP                                                                      4d6h
kubeflow-user-example-com      ml-pipeline-ui-artifact                                     ClusterIP   10.96.74.236    <none>        80/TCP                                                                       63m
kubeflow-user-example-com      ml-pipeline-visualizationserver                             ClusterIP   10.96.5.236     <none>        8888/TCP                                                                     63m
kubeflow                       admission-webhook-service                                   ClusterIP   10.96.188.197   <none>        443/TCP                                                                      44m
kubeflow                       cache-server                                                ClusterIP   10.96.248.28    <none>        443/TCP                                                                      44m
kubeflow                       centraldashboard                                            ClusterIP   10.96.78.32     <none>        80/TCP                                                                       44m
kubeflow                       jupyter-web-app-service                                     ClusterIP   10.96.238.38    <none>        80/TCP                                                                       44m
kubeflow                       katib-controller                                            ClusterIP   10.96.126.55    <none>        443/TCP,8080/TCP                                                             44m
kubeflow                       katib-db-manager                                            ClusterIP   10.96.76.127    <none>        6789/TCP                                                                     44m
kubeflow                       katib-mysql                                                 ClusterIP   10.96.166.99    <none>        3306/TCP                                                                     44m
kubeflow                       katib-ui                                                    ClusterIP   10.96.134.60    <none>        80/TCP                                                                       44m
kubeflow                       kfserving-controller-manager-metrics-service                ClusterIP   10.96.186.81    <none>        8443/TCP                                                                     44m
kubeflow                       kfserving-controller-manager-service                        ClusterIP   10.96.178.21    <none>        443/TCP                                                                      44m
kubeflow                       kfserving-models-web-app                                    ClusterIP   10.96.135.161   <none>        80/TCP                                                                       44m
kubeflow                       kfserving-webhook-server-service                            ClusterIP   10.96.141.200   <none>        443/TCP                                                                      44m
kubeflow                       kubeflow-pipelines-profile-controller                       ClusterIP   10.96.208.53    <none>        80/TCP                                                                       44m
kubeflow                       metadata-envoy-service                                      ClusterIP   10.96.25.241    <none>        9090/TCP                                                                     44m
kubeflow                       metadata-grpc-service                                       ClusterIP   10.96.204.4     <none>        8080/TCP                                                                     44m
kubeflow                       minio-service                                               ClusterIP   10.96.25.9      <none>        9000/TCP                                                                     44m
kubeflow                       ml-pipeline                                                 ClusterIP   10.96.238.80    <none>        8888/TCP,8887/TCP                                                            44m
kubeflow                       ml-pipeline-ui                                              ClusterIP   10.96.61.93     <none>        80/TCP                                                                       44m
kubeflow                       ml-pipeline-visualizationserver                             ClusterIP   10.96.171.245   <none>        8888/TCP                                                                     44m
kubeflow                       mysql                                                       ClusterIP   10.96.33.144    <none>        3306/TCP                                                                     44m
kubeflow                       notebook-controller-service                                 ClusterIP   10.96.186.138   <none>        443/TCP                                                                      44m
kubeflow                       profiles-kfam                                               ClusterIP   10.96.253.190   <none>        8081/TCP                                                                     44m
kubeflow                       tensorboard-controller-controller-manager-metrics-service   ClusterIP   10.96.5.46      <none>        8443/TCP                                                                     44m
kubeflow                       tensorboards-web-app-service                                ClusterIP   10.96.46.201    <none>        80/TCP                                                                       44m
kubeflow                       training-operator                                           ClusterIP   10.96.212.160   <none>        8080/TCP                                                                     44m
kubeflow                       volumes-web-app-service                                     ClusterIP   10.96.156.82    <none>        80/TCP                                                                       44m
kubeflow                       workflow-controller-metrics                                 ClusterIP   10.96.17.156    <none>        9090/TCP                                                                     44m
...
NAMESPACE                      NAME                                                              READY   STATUS      RESTARTS   AGE
auth                           dex-5ddf47d88d-j24kw                                              1/1     Running     0          45m
cert-manager                   cert-manager-7dd5854bb4-zwmrc                                     1/1     Running     0          45m
cert-manager                   cert-manager-cainjector-64c949654c-bsjtd                          1/1     Running     0          45m
cert-manager                   cert-manager-webhook-6bdffc7c9d-4tdp2                             1/1     Running     0          45m
default                        ingress-demo-app-694bf5d965-8j8f9                                 1/1     Running     0          4d6h
default                        ingress-demo-app-694bf5d965-htvqj                                 1/1     Running     0          4d6h
ingress-nginx                  ingress-nginx-admission-create-lqqlk                              0/1     Completed   0          4d6h
ingress-nginx                  ingress-nginx-admission-patch-w9sbl                               0/1     Completed   0          4d6h
ingress-nginx                  ingress-nginx-controller-686f6b6867-bzztx                         1/1     Running     0          3d23h
istio-system                   authservice-0                                                     0/1     Running     0          45m
istio-system                   cluster-local-gateway-7bf6b98855-pnhh6                            1/1     Running     0          45m
istio-system                   istio-ingressgateway-78bc678876-hlwh7                             1/1     Running     0          45m
istio-system                   istiod-755f4cc457-skk6s                                           1/1     Running     0          45m
knative-eventing               eventing-controller-64d97555b-6z6dt                               1/1     Running     0          45m
knative-eventing               eventing-webhook-5c5b8d5c6d-lnj6m                                 1/1     Running     0          45m
knative-eventing               imc-controller-688df5bdb4-sfgrx                                   1/1     Running     0          45m
knative-eventing               imc-dispatcher-5dbb47f555-g6n5c                                   1/1     Running     0          45m
knative-eventing               mt-broker-controller-856784c8ff-ldwxd                             1/1     Running     0          45m
knative-eventing               mt-broker-filter-68fcfcc6c8-blsxt                                 1/1     Running     0          45m
knative-eventing               mt-broker-ingress-bd54bc995-ztqxb                                 1/1     Running     0          45m
knative-serving                activator-546897c9dd-bpgpb                                        2/2     Running     1          42m
knative-serving                autoscaler-7ffbf6d595-7q8dp                                       2/2     Running     1          42m
knative-serving                controller-5894998d7b-t4hvr                                       2/2     Running     1          42m
knative-serving                istio-webhook-578b6b7654-6h4qh                                    2/2     Running     1          42m
knative-serving                networking-istio-6b88f745c-58hvc                                  2/2     Running     1          42m
knative-serving                webhook-c59cc5b48-9s2wk                                           2/2     Running     1          42m
kube-system                    coredns-8496bbfb78-52c27                                          1/1     Running     0          4d6h
kube-system                    coredns-8496bbfb78-ngp9h                                          1/1     Running     0          4d6h
kube-system                    default-http-backend-6946487d9b-9s5sp                             1/1     Running     0          4d6h
kube-system                    etcd-k8s-master-node1                                             1/1     Running     0          4d7h
kube-system                    etcd-snapshot-1634651059-86qzb                                    0/1     Completed   0          4d6h
kube-system                    etcd-snapshot-1634961600-drknl                                    0/1     Completed   0          16h
kube-system                    etcd-snapshot-1634983200-r48jf                                    0/1     Completed   0          10h
kube-system                    etcd-snapshot-1635004800-8chsx                                    0/1     Completed   0          4h21m
kube-system                    kube-apiserver-k8s-master-node1                                   1/1     Running     0          4d7h
kube-system                    kube-controller-manager-k8s-master-node1                          1/1     Running     0          4d7h
kube-system                    kube-flannel-ds-jp2w5                                             1/1     Running     0          4d7h
kube-system                    kube-flannel-ds-jtdxd                                             1/1     Running     0          4d7h
kube-system                    kube-flannel-ds-r5n9v                                             1/1     Running     0          4d7h
kube-system                    kube-flannel-ds-t29zs                                             1/1     Running     0          4d7h
kube-system                    kube-flannel-ds-xl42q                                             1/1     Running     0          4d7h
kube-system                    kube-proxy-jdvsj                                                  1/1     Running     0          3d23h
kube-system                    kube-proxy-jlzgp                                                  1/1     Running     0          3d23h
kube-system                    kube-proxy-nvj9n                                                  1/1     Running     0          3d23h
kube-system                    kube-proxy-qmg4d                                                  1/1     Running     0          3d23h
kube-system                    kube-proxy-tbvj9                                                  1/1     Running     0          3d23h
kube-system                    kube-scheduler-k8s-master-node1                                   1/1     Running     0          4d7h
kube-system                    metrics-server-57bcd9bccd-cd24c                                   1/1     Running     0          4d1h
kube-system                    snapshot-controller-0                                             1/1     Running     0          4d1h
kubeflow-user-example-com      ml-pipeline-ui-artifact-5dd95d555b-b94zm                          2/2     Running     0          41m
kubeflow-user-example-com      ml-pipeline-visualizationserver-6b44c6759f-8599x                  2/2     Running     0          41m
kubeflow                       admission-webhook-deployment-667bd68d94-wfz6r                     1/1     Running     0          42m
kubeflow                       cache-deployer-deployment-79fdf9c5c9-r594c                        2/2     Running     1          42m
kubeflow                       cache-server-5bdf4f4457-zgxkl                                     2/2     Running     0          42m
kubeflow                       centraldashboard-7d496c59bb-f65cp                                 1/1     Running     0          42m
kubeflow                       jupyter-web-app-deployment-6f744fbc54-dl765                       1/1     Running     0          42m
kubeflow                       katib-controller-68c47fbf8b-2rrr9                                 1/1     Running     0          41m
kubeflow                       katib-db-manager-6c76bdc855-7n6p6                                 1/1     Running     1          41m
kubeflow                       katib-mysql-6dcb447c6f-7ctnv                                      1/1     Running     0          41m
kubeflow                       katib-ui-64bb96d5bf-sxmsk                                         1/1     Running     0          41m
kubeflow                       kfserving-controller-manager-0                                    2/2     Running     0          42m
kubeflow                       kfserving-models-web-app-7884f597cf-vgb76                         2/2     Running     0          41m
kubeflow                       kubeflow-pipelines-profile-controller-7b947f4748-ntxkn            1/1     Running     0          41m
kubeflow                       metacontroller-0                                                  1/1     Running     0          42m
kubeflow                       metadata-envoy-deployment-5b4856dd5-x6cbb                         1/1     Running     0          41m
kubeflow                       metadata-grpc-deployment-748f868f64-hpkxr                         2/2     Running     1          41m
kubeflow                       metadata-writer-548bd879bb-cfms9                                  2/2     Running     0          41m
kubeflow                       minio-5b65df66c9-tzvrl                                            2/2     Running     0          41m
kubeflow                       ml-pipeline-5784f9d9cc-p82wc                                      2/2     Running     5          41m
kubeflow                       ml-pipeline-persistenceagent-d6bdc77bd-tvwh5                      2/2     Running     0          41m
kubeflow                       ml-pipeline-scheduledworkflow-5db54d75c5-2p9cr                    2/2     Running     0          41m
kubeflow                       ml-pipeline-ui-5447cb9556-g2lbv                                   2/2     Running     0          41m
kubeflow                       ml-pipeline-viewer-crd-68fb5f4d58-7cnjj                           2/2     Running     1          41m
kubeflow                       ml-pipeline-visualizationserver-cf88b98f7-dc8vd                   2/2     Running     0          41m
kubeflow                       mpi-operator-5c55d6cb8f-nlmr9                                     1/1     Running     0          41m
kubeflow                       mysql-f7b9b7dd4-7km4z                                             2/2     Running     0          41m
kubeflow                       notebook-controller-deployment-578fd4dc97-rcjtg                   1/1     Running     0          41m
kubeflow                       profiles-deployment-7cc7956dfd-vwv44                              2/2     Running     0          41m
kubeflow                       tensorboard-controller-controller-manager-954b7c544-jc9pb         3/3     Running     1          41m
kubeflow                       tensorboards-web-app-deployment-6ff79b7f44-zbbsw                  1/1     Running     0          41m
kubeflow                       training-operator-795c5cb864-zb9rd                                1/1     Running     0          41m
kubeflow                       volumes-web-app-deployment-8589d664cc-wbrlk                       1/1     Running     0          41m
kubeflow                       workflow-controller-76dd87cd85-62x4j                              2/2     Running     1          41m
...
rc-coderepo commented 2 years ago

Did you make any change to the manifests after pull the repo from github?

Check if dex in running in dex namespace and if running check the logs

check this issue It might be similar to your issue here

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been closed due to inactivity.

aishwaryaa021296 commented 2 years ago

how did we resolve the issue

chenbodeng719 commented 2 years ago

did you guys solve it?

Don12138 commented 1 year ago

hey, I resolve it. In my situation, the authservice pod call for dex.auth.svc.cluster.local, but I am using my school's dns setting withsearch shu.com , and the pod will add shu.com automatically, and it becomes dex.auth.svc.cluster.local.shu.com. I modify the statefulset.yaml in manifests/common/oidc-authservice/base and write

      dnsConfig:
        options:
        - name: ndots
          value: "4"

everything goes fine

ObserverWho commented 9 months ago

hey, I resolve it. In my situation, the authservice pod call for dex.auth.svc.cluster.local, but I am using my school's dns setting withsearch shu.com , and the pod will add shu.com automatically, and it becomes dex.auth.svc.cluster.local.shu.com. I modify the statefulset.yaml in manifests/common/oidc-authservice/base and write

      dnsConfig:
        options:
        - name: ndots
          value: "4"

everything goes fine

That works for me,Thank you!

xytsinghua commented 3 months ago
image image image

add env

image image