I have a StorageClass with fsType: xfs. When I create a PVC and a pod to use it, the pod gets a FailedMount event with executable file not found in $PATH:
$ kubectl describe po busybox
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8s default-scheduler Successfully assigned default/busybox to ip-172-31-16-243.ec2.internal
Normal SuccessfulAttachVolume 7s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-43edb969ce3511e8"
Warning FailedMount 1s (x3 over 3s) kubelet, ip-172-31-16-243.ec2.internal MountVolume.SetUp failed for volume "pvc-43edb969ce3511e8" : rpc error: code = Unknown desc = executable file not found in $PATH
The error originates from the csi-rbdplugin container:
$ kubectl logs csi-rbdplugin-c2fg2 csi-rbdplugin
...
I1012 15:43:43.842687 1 nodeserver.go:74] rbd image: csi-rbd-43f35474-ce35-11e8-938c-0a7b6b3d3d9a/xfs-pool was successfully mapped at /dev/rbd1
I1012 15:43:43.842696 1 nodeserver.go:81] target /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount
fstype xfs
device /dev/rbd1
readonly false
attributes map[csiNodePublishSecretNamespace:default imageFormat:2 userid:admin imageFeatures:layering csiProvisionerSecretNamespace:default fsType:xfs monitors:172.31.30.178:6789 172.31.41.151:6789 172.31.84.1:6789 adminid:admin csiNodePublishSecretName:csi-rbd-secret pool:xfs-pool storage.kubernetes.io/csiProvisionerIdentity:1539291186916-8081-csi-rbdplugin csiProvisionerSecretName:csi-rbd-secret]
mountflags []
I1012 15:43:43.843576 1 mount_linux.go:209] Cannot run systemd-run, assuming non-systemd OS
I1012 15:43:43.843590 1 mount_linux.go:210] systemd-run failed with: exit status 1
I1012 15:43:43.843595 1 mount_linux.go:211] systemd-run output: Failed to create bus connection: No such file or directory
I1012 15:43:43.843604 1 mount_linux.go:471] Checking for issues with fsck on disk: /dev/rbd1
I1012 15:43:43.872778 1 mount_linux.go:484] `fsck` error fsck from util-linux 2.23.2
fsck.ext2: Bad magic number in super-block while trying to open /dev/rbd1
/dev/rbd1:
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>
I1012 15:43:43.872797 1 mount_linux.go:490] Attempting to mount disk: xfs /dev/rbd1 /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount
I1012 15:43:43.872809 1 mount_linux.go:146] Mounting cmd (mount) with arguments ([-t xfs -o defaults /dev/rbd1 /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount])
E1012 15:43:43.875501 1 mount_linux.go:151] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs -o defaults /dev/rbd1 /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount
Output: mount: wrong fs type, bad option, bad superblock on /dev/rbd1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so.
I1012 15:43:43.875518 1 mount_linux.go:545] Attempting to determine if disk "/dev/rbd1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd1])
I1012 15:43:43.900261 1 mount_linux.go:548] Output: "", err: exit status 2
I1012 15:43:43.900286 1 mount_linux.go:519] Disk "/dev/rbd1" appears to be unformatted, attempting to format as type: "xfs" with options: [/dev/rbd1]
E1012 15:43:43.900334 1 mount_linux.go:526] format of disk "/dev/rbd1" failed: type:("xfs") target:("/var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount") options:(["defaults"])error:(executable file not found in $PATH)
E1012 15:43:43.900351 1 utils.go:100] GRPC error: executable file not found in $PATH
Looks like it tries to mount it, fails because it is not formatted yet, tries to format it, then fails because a required executable is missing.
My best guess is that it tries to call mkfs.xfs which is not in the container:
I have a StorageClass with
fsType: xfs
. When I create a PVC and a pod to use it, the pod gets a FailedMount event withexecutable file not found in $PATH
:The error originates from the csi-rbdplugin container:
Looks like it tries to mount it, fails because it is not formatted yet, tries to format it, then fails because a required executable is missing.
My best guess is that it tries to call
mkfs.xfs
which is not in the container:When I use
fsType: ext4
, everything works fine - the initial mount fails, but then it successfully formats it, and the pod comes up.StorageClass, PersistentVolumeClaim, and Pod: https://gist.github.com/Cynerva/e54f3852f5d35a6482c711d5f22c874e