ks-apigateway Back-off restarting failed container

[gavin-gy]

1. multi-node install ok.
2.  run kubectl -n kubesphere-system  get pods :
NAME                                     READY   STATUS    RESTARTS   AGE
ks-account-d4c5cdf9d-4nth4               1/1     Running   0          65m
ks-apigateway-65dd54f989-rqjpl           0/1     Error     7          12m
ks-apiserver-6d7ddd7d-q9n5h              1/1     Running   1          77m
ks-console-6f7f75bb48-r545t              1/1     Running   1          77m
ks-controller-manager-6dd9b76d75-f64qj   1/1     Running   1          77m
ks-installer-7987c659d6-nhvhx            1/1     Running   0          78m
openldap-0                               1/1     Running   1          78m
redis-5d4844b947-jf75k                   1/1     Running   1          78m
[root@master01 ~]#

3. 
[root@master01 ~]# kubectl -n kubesphere-system  logs ks-apigateway-65dd54f989-rqjpl
[DEV NOTICE] Registered directive 'authenticate' before 'jwt'
[DEV NOTICE] Registered directive 'authentication' before 'jwt'
[DEV NOTICE] Registered directive 'swagger' before 'jwt'
Activating privacy features... done.
2019/12/10 10:40:36 [INFO][cache:0xc0008c4e10] Started certificate maintenance routine
E1210 10:40:41.415725       1 redis.go:51] unable to reach redis hostdial tcp: i/o timeout
2019/12/10 10:40:41 dial tcp: i/o timeout
[root@master01 ~]#

4.
[root@master01 ~]# kubectl -n kubesphere-system  describe pods ks-apigateway-65dd54f989-rqjpl
Name:           ks-apigateway-65dd54f989-rqjpl
Namespace:      kubesphere-system
Priority:       0
Node:           node02/192.168.3.12
Start Time:     Tue, 10 Dec 2019 18:28:34 +0800
Labels:         app=ks-apigateway
                pod-template-hash=65dd54f989
                tier=backend
                version=v2.1.0
Annotations:    <none>
Status:         Running
IP:             10.233.113.9
Controlled By:  ReplicaSet/ks-apigateway-65dd54f989
Containers:
  ks-apigateway:
    Container ID:  docker://c96788391e8dac70726d22e1a054dadf40c1cd83cb250f9ae07f2985fe376771
    Image:         kubesphere/ks-apigateway:v2.1.0
    Image ID:      docker-pullable://kubesphere/ks-apigateway@sha256:62b00bcde6253279365d48656ab9255f51769aa2b45aba4ade8ffb83ea48a6d9
    Port:          2018/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -c
      export KUBESPHERE_TOKEN=`cat /var/run/secrets/kubernetes.io/serviceaccount/token` && ks-apigateway --conf=/etc/caddy/Caddyfile --log=stderr
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 10 Dec 2019 18:40:36 +0800
      Finished:     Tue, 10 Dec 2019 18:40:41 +0800
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     1
      memory:  500Mi
    Requests:
      cpu:     20m
      memory:  100Mi
    Environment:
      JWT_SECRET:  <set to the key 'jwt-secret' in secret 'ks-account-secret'>  Optional: false
    Mounts:
      /etc/caddy from caddyfile (rw)
      /etc/kubesphere from kubesphere-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kubesphere-token-q77sf (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  caddyfile:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      caddyfile
    Optional:  false
  kubesphere-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kubesphere-config
    Optional:  false
  kubesphere-token-q77sf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubesphere-token-q77sf
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 60s
                 node.kubernetes.io/unreachable:NoExecute for 60s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned kubesphere-system/ks-apigateway-65dd54f989-rqjpl to node02
  Normal   Pulling    13m                   kubelet, node02    Pulling image "kubesphere/ks-apigateway:v2.1.0"
  Normal   Pulled     12m                   kubelet, node02    Successfully pulled image "kubesphere/ks-apigateway:v2.1.0"
  Normal   Pulled     10m (x4 over 12m)     kubelet, node02    Container image "kubesphere/ks-apigateway:v2.1.0" already present on machine
  Normal   Created    10m (x5 over 12m)     kubelet, node02    Created container ks-apigateway
  Normal   Started    10m (x5 over 12m)     kubelet, node02    Started container ks-apigateway
  Warning  BackOff    3m10s (x42 over 12m)  kubelet, node02    Back-off restarting failed container
[root@master01 ~]#

[wansir]

@gavin-gy
可以检查一下 kubesphere-sytem 下 redis svc 的连通性, 确认一下防火墙配置 https://kubesphere.com.cn/docs/v2.1/zh-CN/installation/port-firewall/

[johdandc]

try "iptables -F"