search

oomichi / try-kubernetes

12 stars 5 forks source link

[Fail] [sig-storage] GCP Volumes GlusterFS [It] should be mountable #57

Closed oomichi closed 5 years ago

oomichi commented 5 years ago 
Oct 10 19:41:53.723: INFO: Unexpected error occurred: timed out waiting for the condition
ESC[1mSTEPESC[0m: cleaning the environment after gluster
Oct 10 19:41:53.728: INFO: Waiting up to 5m0s for pod "gluster-client" in namespace "e2e-tests-gcp-volume-c4gsj" to be "terminated due to deadline exceeded"
Oct 10 19:41:53.744: INFO: Pod "gluster-client": Phase="Pending", Reason="", readiness=false. Elapsed: 15.792862ms
Oct 10 19:41:55.746: INFO: Pod "gluster-client": Phase="Pending", Reason="", readiness=false. Elapsed: 2.018585122s
Oct 10 19:41:57.750: INFO: Pod "gluster-client": Phase="Pending", Reason="", readiness=false. Elapsed: 4.02186064s
Oct 10 19:41:59.752: INFO: Pod "gluster-client" in namespace "e2e-tests-gcp-volume-c4gsj" not found. Error: pods "gluster-client" not found
ESC[1mSTEPESC[0m: sleeping a bit so kubelet can unmount and detach the volume
...
ESC[91mESC[1m~ Failure [352.583 seconds]ESC[0m
[sig-storage] GCP Volumes
ESC[90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:56ESC[0m
  GlusterFS
  ESC[90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:124ESC[0m
    ESC[91mESC[1mshould be mountable [It]ESC[0m
    ESC[90m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/volumes.go:125ESC[0m

    ESC[91mExpected error:
        <*errors.errorString | 0xc420087550>: {
            s: "timed out waiting for the condition",
        }
        timed out waiting for the condition
    not to have occurredESC[0m

    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/volume_util.go:457
oomichi commented 5 years ago

gluster-server の Pod 作成は成功したが、gluster-client の Pod 作成で失敗。 gluster-client Pod の状態は以下のとおり。

$ kubectl describe pod gluster-client  -n e2e-tests-gcp-volume-52tnq
Name:               gluster-client
Namespace:          e2e-tests-gcp-volume-52tnq
Priority:           0
PriorityClassName:  <none>
Node:               113-node01/192.168.1.103
Start Time:         Mon, 15 Oct 2018 23:36:58 +0000
Labels:             role=gluster-client
Annotations:        <none>
Status:             Pending
IP:
Containers:
  gluster-client:
    Container ID:
    Image:         busybox
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      while true ; do cat /opt/0/index.html ; sleep 2 ; ls -altrh /opt/  ; sleep 2 ; done
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /opt/0 from gluster-volume-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lr778 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  gluster-volume-0:
    Type:           Glusterfs (a Glusterfs mount on the host that shares a pod's lifetime)
    EndpointsName:  gluster-server
    Path:           test_vol
    ReadOnly:       true
  default-token-lr778:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-lr778
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age   From                 Message
  ----     ------       ----  ----                 -------
  Normal   Scheduled    53s   default-scheduler    Successfully assigned e2e-tests-gcp-volume-52tnq/gluster-client to 113-node01
  Warning  FailedMount  53s   kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log,backup-volfile-servers=10.244.1.31 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-rd84d3c8e4d64486f8518f1b87df61678.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
  Warning  FailedMount  53s  kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log,backup-volfile-servers=10.244.1.31 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-r6a0dcf0ae24e41ad9bd84d168b1baac0.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
  Warning  FailedMount  52s  kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o backup-volfile-servers=10.244.1.31,ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-r87128dccfbec48e3911607569a439a2e.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
  Warning  FailedMount  49s  kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log,backup-volfile-servers=10.244.1.31 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-rcf8433a6d53d4a3580789bcc3624c764.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
  Warning  FailedMount  45s  kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log,backup-volfile-servers=10.244.1.31 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-r8326ee934bbc47bf9115c13c35c71e3d.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
  Warning  FailedMount  37s  kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log,backup-volfile-servers=10.244.1.31 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-r8bd9b81824fb44bc82fc43c0f746c10d.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
  Warning  FailedMount  21s  kubelet, 113-node01  MountVolume.SetUp failed for volume "gluster-volume-0" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o backup-volfile-servers=10.244.1.31,ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-ra313db4c5a65460c8dc4230d043f8092.scope.
mount: unknown filesystem type 'glusterfs'

 the following error information was pulled from the glusterfs log to help diagnose this issue: could not open log file for pod gluster-client
oomichi commented 5 years ago

以下のエラーメッセージが原因っぽい。

Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0 --scope -- mount -t glusterfs -o ro,log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/gluster-volume-0/gluster-client-glusterfs.log,backup-volfile-servers=10.244.1.31 10.244.1.31:test_vol /var/lib/kubelet/pods/34efa644-d0d3-11e8-a9d2-fa163e4aa030/volumes/kubernetes.io~glusterfs/gluster-volume-0
Output: Running scope as unit run-rd84d3c8e4d64486f8518f1b87df61678.scope.
mount: unknown filesystem type 'glusterfs'

デフォルトでは glusterfs 用の mount が含まれていない。

$ mount.
mount.aufs        mount.fuse        mount.lowntfs-3g  mount.nfs         mount.nfs4        mount.ntfs        mount.ntfs-3g     mount.vmhgfs
oomichi commented 5 years ago

全ての実行ノード上で以下のコマンドを実行しmount.glusterfs をインストール、テストが通るようになった。

$ sudo add-apt-repository ppa:gluster/glusterfs-4.1
$ sudo apt-get update
$ sudo apt-get install glusterfs-client