search

gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
875 stars 389 forks source link

deploy-heketi pod fails when firewalld is active #471

Open rwsu opened 6 years ago

rwsu commented 6 years ago

gk-deploy times out waiting for the deploy-heketi pod to start when I have firewalld running. If firewalld is disabled, gk-deploy completes successfully.

My environment is

[root@master ~]# kubectl get pods -n kube-system
NAME                                         READY     STATUS              RESTARTS   AGE
deploy-heketi-5c65fb849b-fppsd               0/1       ContainerCreating   0          11m
etcd-master.localdomain                      1/1       Running             0          8h
glusterfs-2v6zs                              1/1       Running             0          13m
glusterfs-clz2f                              1/1       Running             0          13m
glusterfs-p5bbp                              1/1       Running             0          13m
kube-apiserver-master.localdomain            1/1       Running             0          8h
kube-controller-manager-master.localdomain   1/1       Running             0          8h
kube-dns-86f4d74b45-wkcjx                    3/3       Running             0          8h
kube-proxy-58hds                             1/1       Running             0          8h
kube-proxy-hx97s                             1/1       Running             0          8h
kube-proxy-jd872                             1/1       Running             0          8h
kube-scheduler-master.localdomain            1/1       Running             0          8h
weave-net-55pkh                              2/2       Running             1          8h
weave-net-k8jzr                              2/2       Running             0          8h
weave-net-qt7fn                              2/2       Running             1          8h

[root@master ~]# kubectl describe pod deploy-heketi-5c65fb849b-fppsd -n kube-system
Name:           deploy-heketi-5c65fb849b-fppsd
Namespace:      kube-system
Node:           work2.localdomain/192.168.122.204
Start Time:     Fri, 04 May 2018 21:44:24 -0700
Labels:         deploy-heketi=pod
                glusterfs=heketi-pod
                pod-template-hash=1721964056
Annotations:    <none>
Status:         Pending
IP:             
Controlled By:  ReplicaSet/deploy-heketi-5c65fb849b
Containers:
  deploy-heketi:
    Container ID:   
    Image:          heketi/heketi:dev
    Image ID:       
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/hello delay=30s timeout=3s period=10s #success=1 #failure=3
    Readiness:      http-get http://:8080/hello delay=3s timeout=3s period=10s #success=1 #failure=3
    Environment:
      HEKETI_USER_KEY:                 
      HEKETI_ADMIN_KEY:                my-admin-key
      HEKETI_EXECUTOR:                 kubernetes
      HEKETI_FSTAB:                    /var/lib/heketi/fstab
      HEKETI_SNAPSHOT_LIMIT:           14
      HEKETI_KUBE_GLUSTER_DAEMONSET:   y
      HEKETI_IGNORE_STALE_OPERATIONS:  true
    Mounts:
      /etc/heketi from config (rw)
      /var/lib/heketi from db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from heketi-service-account-token-qjlhg (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  db:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  heketi-config-secret
    Optional:    false
  heketi-service-account-token-qjlhg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  heketi-service-account-token-qjlhg
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age               From                        Message
  ----     ------                  ----              ----                        -------
  Normal   Scheduled               12m               default-scheduler           Successfully assigned deploy-heketi-5c65fb849b-fppsd to work2.localdomain
  Normal   SuccessfulMountVolume   12m               kubelet, work2.localdomain  MountVolume.SetUp succeeded for volume "db"
  Normal   SuccessfulMountVolume   12m               kubelet, work2.localdomain  MountVolume.SetUp succeeded for volume "heketi-service-account-token-qjlhg"
  Normal   SuccessfulMountVolume   12m               kubelet, work2.localdomain  MountVolume.SetUp succeeded for volume "config"
  Warning  FailedCreatePodSandBox  41s (x3 over 8m)  kubelet, work2.localdomain  Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   SandboxChanged          41s (x3 over 8m)  kubelet, work2.localdomain  Pod sandbox changed, it will be killed and re-created.

[root@master ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:49152:49251 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24008 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:24007 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:EtherNet/IP-1 ctstate NEW
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10255 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:apollo-relay ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10251 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:10250 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpts:2379:2380 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:sun-sr-https ctstate NEW
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
INPUT_direct  all  --  anywhere             anywhere            
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere            
INPUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
WEAVE-NPC  all  --  anywhere             anywhere             /* NOTE: this must go before '-j KUBE-FORWARD' */
NFLOG      all  --  anywhere             anywhere             state NEW nflog-group 86
DROP       all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
DOCKER-ISOLATION  all  --  anywhere             anywhere            
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             192.168.124.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.124.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
FORWARD_direct  all  --  anywhere             anywhere            
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_IN_ZONES  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere            
FORWARD_OUT_ZONES  all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
OUTPUT_direct  all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination         
FWDI_public  all  --  anywhere             anywhere            [goto] 
FWDI_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination         
FWDO_public  all  --  anywhere             anywhere            [goto] 
FWDO_public  all  --  anywhere             anywhere            [goto] 

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain FORWARD_direct (1 references)
target     prot opt source               destination         

Chain FWDI_public (2 references)
target     prot opt source               destination         
FWDI_public_log  all  --  anywhere             anywhere            
FWDI_public_deny  all  --  anywhere             anywhere            
FWDI_public_allow  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            

Chain FWDI_public_allow (1 references)
target     prot opt source               destination         

Chain FWDI_public_deny (1 references)
target     prot opt source               destination         

Chain FWDI_public_log (1 references)
target     prot opt source               destination         

Chain FWDO_public (2 references)
target     prot opt source               destination         
FWDO_public_log  all  --  anywhere             anywhere            
FWDO_public_deny  all  --  anywhere             anywhere            
FWDO_public_allow  all  --  anywhere             anywhere            

Chain FWDO_public_allow (1 references)
target     prot opt source               destination         

Chain FWDO_public_deny (1 references)
target     prot opt source               destination         

Chain FWDO_public_log (1 references)
target     prot opt source               destination         

Chain INPUT_ZONES (1 references)
target     prot opt source               destination         
IN_public  all  --  anywhere             anywhere            [goto] 
IN_public  all  --  anywhere             anywhere            [goto] 

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination         

Chain INPUT_direct (1 references)
target     prot opt source               destination         

Chain IN_public (2 references)
target     prot opt source               destination         
IN_public_log  all  --  anywhere             anywhere            
IN_public_deny  all  --  anywhere             anywhere            
IN_public_allow  all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            

Chain IN_public_allow (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination         

Chain IN_public_log (1 references)
target     prot opt source               destination         

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  10.244.0.0/16        anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.244.0.0/16        /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (1 references)
target     prot opt source               destination         
REJECT     tcp  --  anywhere             10.98.225.94         /* kube-system/deploy-heketi:deploy-heketi has no endpoints */ tcp dpt:webcache reject-with icmp-port-unreachable

Chain OUTPUT_direct (1 references)
target     prot opt source               destination         

Chain WEAVE-NPC (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             base-address.mcast.net/4 
WEAVE-NPC-DEFAULT  all  --  anywhere             anywhere             state NEW
WEAVE-NPC-INGRESS  all  --  anywhere             anywhere             state NEW
ACCEPT     all  --  anywhere             anywhere             ! match-set weave-local-pods dst

Chain WEAVE-NPC-DEFAULT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             match-set weave-E.1.0W^NGSp]0_t5WwH/]gX@L dst /* DefaultAllow isolation for namespace: default */
ACCEPT     all  --  anywhere             anywhere             match-set weave-?b%zl9GIe0AET1(QI^7NWe*fO dst /* DefaultAllow isolation for namespace: kube-system */
ACCEPT     all  --  anywhere             anywhere             match-set weave-0EHD/vdN#O4]V?o4Tx7kS;APH dst /* DefaultAllow isolation for namespace: kube-public */

Chain WEAVE-NPC-INGRESS (1 references)
target     prot opt source               destination
jarrpa commented 6 years ago

@rwsu Hi! Sorry for the delay.

What did you use for your kube deployment and networking? I'm seeing a lot of issues with that when searching for your particular error message:

https://www.google.com/search?q=kube+Failed+create+pod+sandbox

rwsu commented 6 years ago

@jarrpa I used kubeadm to create the cluster and Weave Net as the CNI.

jarrpa commented 6 years ago

Did you check through any of the hits in that search? Anything you've determined is not a problem?

QinjieLin-NU commented 4 years ago

Hi, I also encounter the same problem in same condition. Could you tell me how to fix it?