search

gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
875 stars 389 forks source link

heketi-storage-copy-job does not start #546

Closed sluetze closed 5 years ago

sluetze commented 5 years ago

Hey Guys,

Environment: RHEL 7.6

Kubernetes 1.11 Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.2-docker-2", GitCommit:"01c76e4d3b4fa8a0d9f51c4d2c9bfdabb0db7462", GitTreeState:"clean", BuildDate:"2018-09-05T15:54:14Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Images: gluster/gluster-centos:latest (also tried gluster3u12_centos7) gluster/gluster-object:latest heketi/heketi:dev

im trying to setup a gluster in kubernetes... Important: since my environment is offline (no internet connection) I manually downloaded the images and renamed them for my local registry.

Warning  FailedMount  1m (x3 over 5m)  kubelet, node-1 (combined from similar events): MountVolume.SetUp failed for volume "heketi-storage" : mount failed: mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/d190eb97-0ea4-11e9-8b83-0242ac110009/volumes/kubernetes.io~glusterfs/heketi-storage --scope -- mount -t glusterfs -o log-level=ERROR,log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job-66r7l-glusterfs.log,backup-volfile-servers=10.0.0.1:10.0.0.2:10.0.0.3 10.0.0.1:heketidbstorage /var/lib/kubelet/pods/d190eb97-0ea4-11e9-8b83-0242ac110009/volumes/kubernetes.io~glusterfs/heketi-storage
Output: Running scope as unit run-r27b4f42fd0dd4d568a054b07b1070f60.scope.
mount: unknown filesystem type 'glusterfs'

I have seen the solutions around installing glusterfs-fuse... Which I already have:

rpm -qa | grep glusterfs
glusterfs-libs-3.12.2-18.el7.x86_64
glusterfs-3.12.2-18.el7.x86_64
glusterfs-client-xlators-3.12.2-18.el7.x86_64
glusterfs-fuse-3.12.2-18.el7.x86_64

kernel modules are also loaded

lsmod | cut -d' ' -f1 | grep -e dm_snapshot -e dm_mirror -e dm_thin_pool -e fuse
fuse
dm_thin_pool
dm_snapshot
dm_mirror

a manual mount from the host (yes it is the worker, which the pod is scheduled on) works!

# systemd-run --description="Kubernetes transient mount for /var/lib/kubelet/pods/d190eb97-0ea4-11e9-8b83-0242ac110009/volumes/kubernetes.io~glusterfs/heketi-storage" --scope -- mount -t glusterfs -o log-file=/var/lib/kubelet/plugins/kubernetes.io/glusterfs/heketi-storage/heketi-storage-copy-job2-66r7l-glusterfs.log,backup-volfile-servers=10.0.0.1:10.0.0.2:10.0.0.3,log-level=ERROR 10.0.0.1:heketidbstorage /mnt/gluster
Running scope as unit run-27671.scope.
mount | grep gluster
10.0.0.1:heketidbstorage on /mnt/gluster type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

I can reproduce this with different workers... Any Ideas?

Thanks!

sluetze commented 5 years ago

After some more hours of debugging I found no solution but the cause. The Kubernetes version I am using is included in DockerEE. They spawn a hyperkube-container per node. Thus the packages must be available in the hyperkube-container, which they are not.