search

coreos / kubernetes

Container Cluster Manager
http://kubernetes.io
Apache License 2.0
79 stars 21 forks source link

Unable to mount glusterfs volume #160

Open grebois opened 7 years ago

grebois commented 7 years ago

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

Uncomment only one, leave it on its own line:

/kind bug /kind feature

What happened:

1) Deployed kubernetes 1.7.5 2) Deployed GlusterFS + Heketi 3) Created a PVC 4) Fails to mount with:

Output: mount: unknown filesystem type 'glusterfs'

What you expected to happen:

Mount without problem

How to reproduce it (as minimally and precisely as possible):

1) Deployed kubernetes 1.7.5 2) Deployed GlusterFS + Heketi 3) Created a PVC

Anything else we need to know?:

Full output:

$ kubectl -n kube-system describe pod heketi-storage-copy-job-hrk8l
2017-09-06 19:21:23.457239 I | proto: duplicate proto type registered: google.protobuf.Any
2017-09-06 19:21:23.457326 I | proto: duplicate proto type registered: google.protobuf.Duration
2017-09-06 19:21:23.457353 I | proto: duplicate proto type registered: google.protobuf.Timestamp
Name:       heketi-storage-copy-job-hrk8l
Namespace:  kube-system
Node:       int00web11/10.105.144.69
Start Time: Wed, 06 Sep 2017 19:07:09 +0000
Labels:     controller-uid=9530a0af-9336-11e7-92f8-002590f2f522
        job-name=heketi-storage-copy-job
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"Job","namespace":"kube-system","name":"heketi-storage-copy-job","uid":"9530a0af-9336-11e7-92f8-002590f2f52...
Status:     Pending
IP:
Created By: Job/heketi-storage-copy-job
Controlled By:  Job/heketi-storage-copy-job
Containers:
  heketi:
    Container ID:
    Image:      heketi/heketi:dev
    Image ID:
    Port:       <none>
    Command:
      cp
      /db/heketi.db
      /heketi
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /db from heketi-storage-secret (rw)
      /heketi from heketi-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-r6kvs (ro)
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  heketi-storage:
    Type:       Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:  heketi-storage-endpoints
    Path:       heketidbstorage
    ReadOnly:       false
  heketi-storage-secret:
    Type:   Secret (a volume populated by a Secret)
    SecretName: heketi-storage-secret
    Optional:   false
  default-token-r6kvs:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-r6kvs
    Optional:   false
QoS Class:  BestEffort
Node-Selectors: <none>
Tolerations:    <none>
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  14m       14m     1   default-scheduler           Normal      Scheduled       Successfully assigned heketi-storage-copy-job-hrk8l to int00web11
  14m       14m     1   kubelet, int00web11         Normal      SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "default-token-r6kvs"
  14m       14m     1   kubelet, int00web11         Normal      SuccessfulMountVolume   MountVolume.SetUp succeeded for volume "heketi-storage-secret"
  14m       10m     4   kubelet, int00web11         Warning     FailedMount     MountVolume.SetUp failed for volume "heketi-storage" : glusterfs: mount failed: mount failed: exit status 32
Mounting command: mount
Mounting arguments: 10.105.144.79:heketidbstorage /var/lib/kubelet/pods/95349e36-9336-11e7-a9f3-0cc47a3079e8/volumes/kubernetes.io~glusterfs/heketi-storage glusterfs [log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hrk8l-glusterfs.log backup-volfile-servers=10.105.144.101:10.105.144.69:10.105.144.79 log-level=ERROR]
Output: mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: glusterfs: could not open log file for pod: heketi-storage-copy-job-hrk8l
  14m   8m  2   kubelet, int00web11     Warning FailedMount MountVolume.SetUp failed for volume "heketi-storage" : glusterfs: mount failed: mount failed: exit status 32
Mounting command: mount
Mounting arguments: 10.105.144.79:heketidbstorage /var/lib/kubelet/pods/95349e36-9336-11e7-a9f3-0cc47a3079e8/volumes/kubernetes.io~glusterfs/heketi-storage glusterfs [backup-volfile-servers=10.105.144.101:10.105.144.69:10.105.144.79 log-level=ERROR log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hrk8l-glusterfs.log]
Output: mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: glusterfs: could not open log file for pod: heketi-storage-copy-job-hrk8l
  14m   1m  8   kubelet, int00web11     Warning FailedMount MountVolume.SetUp failed for volume "heketi-storage" : glusterfs: mount failed: mount failed: exit status 32
Mounting command: mount
Mounting arguments: 10.105.144.79:heketidbstorage /var/lib/kubelet/pods/95349e36-9336-11e7-a9f3-0cc47a3079e8/volumes/kubernetes.io~glusterfs/heketi-storage glusterfs [log-level=ERROR log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-hrk8l-glusterfs.log backup-volfile-servers=10.105.144.101:10.105.144.69:10.105.144.79]
Output: mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: glusterfs: could not open log file for pod: heketi-storage-copy-job-hrk8l
  12m   48s 6   kubelet, int00web11     Warning FailedMount Unable to mount volumes for pod "heketi-storage-copy-job-hrk8l_kube-system(95349e36-9336-11e7-a9f3-0cc47a3079e8)": timeout expired waiting for volumes to attach/mount for pod "kube-system"/"heketi-storage-copy-job-hrk8l". list of unattached/unmounted volumes=[heketi-storage]
  12m   48s 6   kubelet, int00web11     Warning FailedSync  Error syncing pod

Environment:

grebois commented 7 years ago

This problems happens only when kubelet runs as a service in the host and not in a container:

https://github.com/kubernetes-incubator/kubespray/blob/master/inventory/group_vars/k8s-cluster.yml#L137

This has changed in kubespray not so long ago;

https://github.com/kubernetes-incubator/kubespray/commit/ca3050ec3d335647d9e703b6e0194b2345b99a55#diff-96edf7a6f008de9e928d04e1ae5e12a5R62

mattymo commented 7 years ago

We should install glusterfs client for the host