kubesphere / ks-installer

Install KubeSphere on existing Kubernetes cluster
https://kubesphere.io
Apache License 2.0
532 stars 747 forks source link

Pod ks-apiserver报错:CrashLoopBackOff #488

Closed daiqingliang closed 4 years ago

daiqingliang commented 5 years ago

k8s版本:1.15.5

出现错误回顾

通过 最小化快速部署 安装完kubesphere,然后通过修改configMap来启用metric

$ kubectl edit cm ks-installer -n kubesphere-system

保存之后,发现ks-apiserver出现问题,web界面无法访问,pod一直就是CrashLoopBackOff。 然后我又通过configMap关闭了metric,发现ks-apiserver仍然报错,无法恢复。

[root@master ~]# kubectl get pod --all-namespaces -o wide
NAMESPACE                      NAME                                        READY   STATUS             RESTARTS   AGE     IP               NODE            NOMINATED NODE   READINESS GATES
default                        nfs-client-provisioner-687b8d957f-pwhdw     1/1     Running            0          22h     10.244.186.195   node03          <none>           <none>
kube-system                    calico-kube-controllers-845589bf5f-xklbg    1/1     Running            0          28h     10.244.225.131   apiserver.ioc   <none>           <none>
kube-system                    calico-node-5nf8r                           1/1     Running            0          27h     *.*.*.223      node01          <none>           <none>
kube-system                    calico-node-9lrsz                           0/1     Running            0          28h     *.*.*.6        apiserver.ioc   <none>           <none>
kube-system                    calico-node-pd4cq                           1/1     Running            0          27h     *.*.*.225      node03          <none>           <none>
kube-system                    calico-node-sk8sh                           1/1     Running            0          27h     *.*.*.224      node02          <none>           <none>
kube-system                    coredns-5c98db65d4-6vrxx                    1/1     Running            0          29h     10.244.225.129   apiserver.ioc   <none>           <none>
kube-system                    coredns-5c98db65d4-x8d2d                    1/1     Running            0          29h     10.244.225.130   apiserver.ioc   <none>           <none>
kube-system                    etcd-apiserver.ioc                          1/1     Running            0          29h     *.*.*.6        apiserver.ioc   <none>           <none>
kube-system                    kube-apiserver-apiserver.ioc                1/1     Running            0          29h     *.*.*.6        apiserver.ioc   <none>           <none>
kube-system                    kube-controller-manager-apiserver.ioc       1/1     Running            0          29h     *.*.*.6        apiserver.ioc   <none>           <none>
kube-system                    kube-proxy-4cqct                            1/1     Running            0          27h     *.*.*.223      node01          <none>           <none>
kube-system                    kube-proxy-czg6n                            1/1     Running            0          27h     *.*.*.225      node03          <none>           <none>
kube-system                    kube-proxy-dtv4z                            1/1     Running            0          29h     *.*.*.6        apiserver.ioc   <none>           <none>
kube-system                    kube-proxy-vm77f                            1/1     Running            0          27h     *.*.*.224      node02          <none>           <none>
kube-system                    kube-scheduler-apiserver.ioc                1/1     Running            0          29h     *.*.*.6        apiserver.ioc   <none>           <none>
kube-system                    metrics-server-dddf8574-d2c2z               1/1     Running            0          63m     10.244.196.137   node01          <none>           <none>
kube-system                    tiller-deploy-75f5747884-fv872              1/1     Running            0          25h     10.244.140.65    node02          <none>           <none>
kubernetes-dashboard           dashboard-metrics-scraper-fb986f88d-fzmlq   1/1     Running            0          23h     10.244.186.194   node03          <none>           <none>
kubernetes-dashboard           kubernetes-dashboard-6bb65fcc49-25kp8       1/1     Running            0          23h     10.244.140.66    node02          <none>           <none>
kubesphere-controls-system     default-http-backend-6555ff6898-gfzhs       1/1     Running            0          4h23m   10.244.186.197   node03          <none>           <none>
kubesphere-controls-system     kubectl-admin-74fdfc47c7-82r46              1/1     Running            0          4h19m   10.244.196.134   node01          <none>           <none>
kubesphere-monitoring-system   kube-state-metrics-67859c59fb-vb2vt         4/4     Running            0          4h20m   10.244.186.199   node03          <none>           <none>
kubesphere-monitoring-system   node-exporter-5nr6j                         2/2     Running            0          4h22m   *.*.*.223      node01          <none>           <none>
kubesphere-monitoring-system   node-exporter-9t8hb                         2/2     Running            0          4h22m   *.*.*.224      node02          <none>           <none>
kubesphere-monitoring-system   node-exporter-ckm8s                         2/2     Running            0          4h22m   *.*.*.225      node03          <none>           <none>
kubesphere-monitoring-system   node-exporter-tb7dr                         2/2     Running            0          4h22m   *.*.*.6        apiserver.ioc   <none>           <none>
kubesphere-monitoring-system   prometheus-k8s-0                            3/3     Running            1          4h22m   10.244.196.133   node01          <none>           <none>
kubesphere-monitoring-system   prometheus-k8s-system-0                     3/3     Running            1          4h22m   10.244.140.69    node02          <none>           <none>
kubesphere-monitoring-system   prometheus-operator-685bc484cb-hlnkh        1/1     Running            0          4h22m   10.244.186.198   node03          <none>           <none>
kubesphere-system              ks-account-d4c5cdf9d-5vjsk                  1/1     Running            0          39m     10.244.225.143   apiserver.ioc   <none>           <none>
kubesphere-system              ks-apigateway-65dd54f989-z2pn2              1/1     Running            0          39m     10.244.225.141   apiserver.ioc   <none>           <none>
kubesphere-system              ks-apiserver-6d7ddd7d-r7mll                 0/1     CrashLoopBackOff   8          17m     10.244.186.201   node03          <none>           <none>
kubesphere-system              ks-console-6f7f75bb48-zjsfd                 1/1     Running            0          39m     10.244.225.145   apiserver.ioc   <none>           <none>
kubesphere-system              ks-controller-manager-6dd9b76d75-xb47t      1/1     Running            0          39m     10.244.225.144   apiserver.ioc   <none>           <none>
kubesphere-system              ks-installer-556774c9fb-5bhbn               1/1     Running            0          4h24m   10.244.196.132   node01          <none>           <none>
kubesphere-system              openldap-0                                  1/1     Running            0          4h23m   10.244.225.133   apiserver.ioc   <none>           <none>
kubesphere-system              redis-5d4844b947-xzjf2                      1/1     Running            0          4h23m   10.244.225.132   apiserver.ioc   <none>           <none>

[root@master ~]# kubectl describe pod ks-apiserver-6d7ddd7d-r7mll -n kubesphere-system
Name:           ks-apiserver-6d7ddd7d-r7mll
Namespace:      kubesphere-system
Priority:       0
Node:           node03/*.*.*.225
Start Time:     Fri, 15 Nov 2019 14:24:31 +0800
Labels:         app=ks-apiserver
                pod-template-hash=6d7ddd7d
                tier=backend
                version=v2.1.0
Annotations:    cni.projectcalico.org/podIP: 10.244.186.201/32
Status:         Running
IP:             10.244.186.201
Controlled By:  ReplicaSet/ks-apiserver-6d7ddd7d
Containers:
  ks-apiserver:
    Container ID:  docker://15b1d5e34656fff7064772ea9a4de64dd0cf89b2ffc17ecb3ca38af8caad3fa4
    Image:         kubesphere/ks-apiserver:v2.1.0
    Image ID:      docker://sha256:e705063b9c6cb4e2693e7b43a1b580757543312a016ae3972b8ce1cd5122a151
    Port:          9090/TCP
    Host Port:     0/TCP
    Command:
      ks-apiserver
      --logtostderr=true
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 15 Nov 2019 14:40:42 +0800
      Finished:     Fri, 15 Nov 2019 14:40:42 +0800
    Ready:          False
    Restart Count:  8
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:        20m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /etc/kubesphere/ from kubesphere-config (rw)
      /etc/kubesphere/ingress-controller from ks-router-config (rw)
      /etc/kubesphere/rules from policy-rules (rw)
      /var/run/docker.sock from docker-sock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubesphere-token-5f7gf (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  docker-sock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  
  ks-router-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ks-router-config
    Optional:  false
  policy-rules:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      policy-rules
    Optional:  false
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  kubesphere-token-5f7gf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubesphere-token-5f7gf
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 60s
                 node.kubernetes.io/unreachable:NoExecute for 60s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  18m                   default-scheduler  Successfully assigned kubesphere-system/ks-apiserver-6d7ddd7d-r7mll to node03
  Normal   Pulled     17m (x5 over 18m)     kubelet, node03    Container image "kubesphere/ks-apiserver:v2.1.0" already present on machine
  Normal   Created    17m (x5 over 18m)     kubelet, node03    Created container ks-apiserver
  Normal   Started    16m (x5 over 18m)     kubelet, node03    Started container ks-apiserver
  Warning  BackOff    3m33s (x70 over 18m)  kubelet, node03    Back-off restarting failed container
pixiake commented 5 years ago

用kubectl logs xxx看下日志

daiqingliang commented 5 years ago

@pixiake

[root@master ~]# kubectl logs ks-apiserver-6d7ddd7d-r7mll -n kubesphere-system
W1115 08:12:41.210771       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1115 08:12:41.212742       1 server.go:179] Start cache objects
Error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Usage:
  ks-apiserver [flags]

Flags:
      --add-dir-header                                  If true, adds the file directory to the header
      --alsologtostderr                                 log to standard error as well as files
      --bind-address string                             server bind address (default "0.0.0.0")
      --elasticsearch-host string                       ElasticSearch logging service host. KubeSphere is using elastic as log store, if this filed left blank, KubeSphere will use kubernetes builtin log API instead, and the following elastic search options will be ignored.
      --elasticsearch-version string                    ElasticSearch major version, e.g. 5/6/7, if left blank, will detect automatically.Currently, minimum supported version is 5.x
  -h, --help                                            help for ks-apiserver
      --index-prefix string                             Index name prefix. KubeSphere will retrieve logs against indices matching the prefix. (default "fluentbit")
      --insecure-port int                               insecure port number (default 9090)
      --istio-pilot-host string                         istio pilot discovery service url
      --jaeger-query-host string                        jaeger query service url
      --jenkins-host string                             Jenkins service host address. If left blank, means Jenkins is unnecessary.
      --jenkins-max-connections int                     Maximum allowed connections to Jenkins.  (default 100)
      --jenkins-password string                         Password for access to Jenkins service, used pair with username.
      --jenkins-username string                         Username for access to Jenkins service. Leave it blank if there isn't any.
      --kubeconfig string                               Path for kubernetes kubeconfig file, if left blank, will use in cluster way.
      --log-backtrace-at traceLocation                  when logging hits line file:N, emit a stack trace (default :0)
      --log-dir string                                  If non-empty, write log files in this directory
      --log-file string                                 If non-empty, use this log file
      --log-file-max-size uint                          Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                                     log to standard error instead of files (default true)
      --master string                                   Used to generate kubeconfig for downloading, if not specified, will use host in kubeconfig.
      --mysql-host string                               MySQL service host address. If left blank, the following related mysql options will be ignored.
      --mysql-max-connection-life-time duration         Maximum connection life time allowed to connecto to mysql. (default 10s)
      --mysql-max-idle-connections int                  Maximum idle connections allowed to connect to mysql. (default 100)
      --mysql-max-open-connections int                  Maximum open connections allowed to connect to mysql. (default 100)
      --mysql-password string                           Password for access to mysql, should be used pair with password.
      --mysql-username string                           Username for access to mysql service.
      --openpitrix-app-manager-endpoint string          OpenPitrix app manager endpoint
      --openpitrix-attachment-manager-endpoint string   OpenPitrix attachment manager endpoint
      --openpitrix-category-manager-endpoint string     OpenPitrix category manager endpoint
      --openpitrix-cluster-manager-endpoint string      OpenPitrix cluster manager endpoint
      --openpitrix-repo-indexer-endpoint string         OpenPitrix repo indexer endpoint
      --openpitrix-repo-manager-endpoint string         OpenPitrix repo manager endpoint
      --openpitrix-runtime-manager-endpoint string      OpenPitrix runtime manager endpoint
      --prometheus-endpoint string                      Prometheus service endpoint which stores KubeSphere monitoring data, if left blank, will use builtin metrics-server as data source.
      --prometheus-secondary-endpoint string            Prometheus secondary service endpoint, if left empty and endpoint is set, will use endpoint instead.
      --s3-access-key-id string                         access key of s2i s3 (default "AKIAIOSFODNN7EXAMPLE")
      --s3-bucket string                                bucket name of s2i s3 (default "s2i-binaries")
      --s3-disable-SSL                                  disable ssl (default true)
      --s3-endpoint string                              Endpoint to access to s3 object storage service, if left blank, the following options will be ignored.
      --s3-force-path-style                             force path style (default true)
      --s3-region string                                Region of s3 that will access to, like us-east-1. (default "us-east-1")
      --s3-secret-access-key string                     secret access key of s2i s3 (default "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY")
      --s3-session-token string                         session token of s2i s3
      --secure-port int                                 secure port number
      --servicemesh-prometheus-host string              prometheus service for servicemesh
      --skip-headers                                    If true, avoid header prefixes in the log messages
      --skip-log-headers                                If true, avoid headers when opening log files
      --sonarqube-host string                           Sonarqube service address, if left empty, following sonarqube options will be ignored.
      --sonarqube-token string                          Sonarqube service access token.
      --stderrthreshold severity                        logs at or above this threshold go to stderr (default 2)
      --tls-cert-file string                            tls cert file
      --tls-private-key string                          tls private key
  -v, --v Level                                         number for the log level verbosity
      --vmodule moduleSpec                              comma-separated list of pattern=N settings for file-filtered logging

2019/11/15 08:12:41 unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
pixiake commented 5 years ago

@zryfish @wansir

zryfish commented 4 years ago

@daiqingliang 你的集群是不是没有安装 metrics-server ,你看下 kube-system 下是否有 metrics-server

然后执行 kubectl get apiservice | grep metrics 看下是否有结果