search

kenwoodjw / kubeflow-install

1 stars 1 forks source link

0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. #2

Open yanqiangmiffy opened 1 year ago

yanqiangmiffy commented 1 year ago

kubectl get pods --all-namespaces 其中有些pod状态为Pending

NAMESPACE            NAME                                                   READY   STATUS             RESTARTS   AGE
auth                 dex-559dbcd758-tk7kz                                   1/1     Running            0          3h47m
cert-manager         cert-manager-7cc4df4cd6-tc2rs                          1/1     Running            1          3h47m
cert-manager         cert-manager-cainjector-8d785bf4c-6nfz9                1/1     Running            1          3h47m
cert-manager         cert-manager-webhook-87648988f-rgb9g                   1/1     Running            0          3h47m
default              volume-test                                            1/1     Running            0          32m
istio-system         authservice-0                                          0/1     Pending            0          3h47m
istio-system         cluster-local-gateway-5449f87d9b-46g47                 1/1     Running            0          3h47m
istio-system         istio-ingressgateway-77b9d69b74-w4d4h                  1/1     Running            0          3h47m
istio-system         istiod-67fcb675b5-77rnh                                1/1     Running            0          3h47m
knative-eventing     eventing-controller-5c847567d9-lngqt                   1/1     Running            0          74m
knative-eventing     eventing-webhook-7f7fbb679f-9m97f                      1/1     Running            0          130m
knative-serving      activator-76f4874c56-xx2fw                             2/2     Running            15         130m
knative-serving      autoscaler-df6fb584d-g2kwt                             2/2     Running            0          63m
knative-serving      controller-9b64997bb-dl4wk                             2/2     Running            0          63m
knative-serving      domain-mapping-c475b7674-krbwh                         2/2     Running            0          63m
knative-serving      domainmapping-webhook-599b77575f-b92vl                 2/2     Running            0          130m
knative-serving      net-istio-controller-554557d85f-8ldqc                  2/2     Running            0          130m
knative-serving      net-istio-webhook-5dd85b4967-sz2cn                     2/2     Running            0          130m
knative-serving      webhook-6b494b646c-8w8pf                               2/2     Running            0          63m
kube-system          calico-kube-controllers-5bcd7db644-gdsj9               1/1     Running            0          4h5m
kube-system          calico-node-bmfsr                                      1/1     Running            3          4h5m
kube-system          coredns-59d64cd4d4-6z8pz                               1/1     Running            0          4h41m
kube-system          coredns-59d64cd4d4-ks8f2                               1/1     Running            0          4h41m
kube-system          etcd-server-szry1agd.novalocal                         1/1     Running            0          4h41m
kube-system          kube-apiserver-server-szry1agd.novalocal               1/1     Running            0          4h41m
kube-system          kube-controller-manager-server-szry1agd.novalocal      1/1     Running            11         4h41m
kube-system          kube-proxy-qgdwl                                       1/1     Running            0          4h41m
kube-system          kube-scheduler-server-szry1agd.novalocal               1/1     Running            10         4h41m
kubeflow             admission-webhook-deployment-c77d48bbb-2nbdt           1/1     Running            0          3h30m
kubeflow             cache-server-6647b76c64-m2rpv                          2/2     Running            3          43m
kubeflow             centraldashboard-5864f74d99-9w9hk                      2/2     Running            0          3h30m
kubeflow             jupyter-web-app-deployment-844cf8dd7c-mncll            1/1     Running            0          3h30m
kubeflow             katib-controller-6848d4dd9f-j9wbl                      1/1     Running            0          3h30m
kubeflow             katib-db-manager-665954948-5c9xd                       0/1     CrashLoopBackOff   43         3h30m
kubeflow             katib-mysql-5bf95ddfcc-5mw64                           0/1     Pending            0          3h30m
kubeflow             katib-ui-56ccff658f-6ljnh                              1/1     Running            0          3h30m
kubeflow             kserve-controller-manager-0                            2/2     Running            0          3h47m
kubeflow             kserve-models-web-app-544c6b55bb-znttv                 2/2     Running            1          3h30m
kubeflow             kubeflow-pipelines-profile-controller-94bf4544-m6ss8   1/1     Running            0          3h30m
kubeflow             metacontroller-0                                       1/1     Running            0          3h47m
kubeflow             metadata-envoy-deployment-868cf55d45-twrwm             1/1     Running            0          43m
kubeflow             metadata-grpc-deployment-f8d68f687-b2pvq               1/2     CrashLoopBackOff   34         3h30m
kubeflow             metadata-writer-d6498d6b4-9j7cr                        2/2     Running            27         3h30m
kubeflow             minio-5b65df66c9-gclvx                                 0/2     Pending            0          3h30m
kubeflow             ml-pipeline-67cf96c9bf-98vws                           1/2     CrashLoopBackOff   41         3h30m
kubeflow             ml-pipeline-persistenceagent-5854f86f8b-9b7rm          2/2     Running            0          3h30m
kubeflow             ml-pipeline-scheduledworkflow-5dddbf664f-gbtsc         2/2     Running            0          3h30m
kubeflow             ml-pipeline-ui-6bdfc6dbcd-5gxl6                        2/2     Running            2          3h30m
kubeflow             ml-pipeline-viewer-crd-84f7566c48-z6ntx                2/2     Running            1          43m
kubeflow             ml-pipeline-visualizationserver-54fccfcb5f-6hw7d       2/2     Running            1          52m
kubeflow             mysql-5c7f79f986-nhh2k                                 0/2     Pending            0          3h30m
kubeflow             notebook-controller-deployment-9954c4fd9-6hrg4         2/2     Running            1          3h30m
kubeflow             tensorboard-controller-deployment-844d9585fc-s4rjj     3/3     Running            1          3h30m
kubeflow             tensorboards-web-app-deployment-7578c885f7-jq7hz       1/1     Running            0          3h30m
kubeflow             training-operator-6c9f6fd894-8fdcx                     1/1     Running            0          3h30m
kubeflow             volumes-web-app-deployment-7bc5754bd4-47fft            1/1     Running            0          3h30m
kubeflow             workflow-controller-6b9b6c5b46-dthnc                   2/2     Running            5          3h30m
local-path-storage   local-path-provisioner-7b6c8448c7-g7w6g                1/1     Running            0          153m

然后查看其中描述 kubectl describe pod authservice-0 -n istio-system

Name:             authservice-0
Namespace:        istio-system
Priority:         0
Service Account:  default
Node:             <none>
Labels:           app=authservice
                  controller-revision-hash=authservice-6db8b4db64
                  statefulset.kubernetes.io/pod-name=authservice-0
Annotations:      sidecar.istio.io/inject: false
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    StatefulSet/authservice
Containers:
  authservice:
    Image:      gcr.io/arrikto/kubeflow/oidc-authservice:28c59ef
    Port:       8080/TCP
    Host Port:  0/TCP
    Readiness:  http-get http://:8081/ delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      oidc-authservice-client      Secret     Optional: false
      oidc-authservice-parameters  ConfigMap  Optional: false
    Environment:                   <none>
    Mounts:
      /var/lib/authservice from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hsh6m (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  authservice-pvc
    ReadOnly:   false
  kube-api-access-hsh6m:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  46m (x1 over 47m)   default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  31m                 default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  55s (x29 over 30m)  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
pvc-6b18608d-7141-4e9e-b294-c7780e86e747   128Mi      RWO            Delete           Bound    default/local-path-pvc   local-path              35m

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE local-path-pvc Bound pvc-6b18608d-7141-4e9e-b294-c7780e86e747 128Mi RWO local-path 36m


- kubectl get pod

NAME READY STATUS RESTARTS AGE volume-test 1/1 Running 0 36m

kenwoodjw commented 1 year ago

可以试下删除pvc 和pod，看重新创建有没有问题