csi-rbdplugin FailedMount with fsType: xfs

I have a StorageClass with fsType: xfs. When I create a PVC and a pod to use it, the pod gets a FailedMount event with executable file not found in $PATH:

$ kubectl describe po busybox
...
Events:
  Type     Reason                  Age              From                                    Message
  ----     ------                  ----             ----                                    -------
  Normal   Scheduled               8s               default-scheduler                       Successfully assigned default/busybox to ip-172-31-16-243.ec2.internal
  Normal   SuccessfulAttachVolume  7s               attachdetach-controller                 AttachVolume.Attach succeeded for volume "pvc-43edb969ce3511e8"
  Warning  FailedMount             1s (x3 over 3s)  kubelet, ip-172-31-16-243.ec2.internal  MountVolume.SetUp failed for volume "pvc-43edb969ce3511e8" : rpc error: code = Unknown desc = executable file not found in $PATH

The error originates from the csi-rbdplugin container:

$ kubectl logs csi-rbdplugin-c2fg2 csi-rbdplugin
...
I1012 15:43:43.842687       1 nodeserver.go:74] rbd image: csi-rbd-43f35474-ce35-11e8-938c-0a7b6b3d3d9a/xfs-pool was successfully mapped at /dev/rbd1
I1012 15:43:43.842696       1 nodeserver.go:81] target /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount
fstype xfs
device /dev/rbd1
readonly false
attributes map[csiNodePublishSecretNamespace:default imageFormat:2 userid:admin imageFeatures:layering csiProvisionerSecretNamespace:default fsType:xfs monitors:172.31.30.178:6789 172.31.41.151:6789 172.31.84.1:6789 adminid:admin csiNodePublishSecretName:csi-rbd-secret pool:xfs-pool storage.kubernetes.io/csiProvisionerIdentity:1539291186916-8081-csi-rbdplugin csiProvisionerSecretName:csi-rbd-secret]
 mountflags []
I1012 15:43:43.843576       1 mount_linux.go:209] Cannot run systemd-run, assuming non-systemd OS
I1012 15:43:43.843590       1 mount_linux.go:210] systemd-run failed with: exit status 1
I1012 15:43:43.843595       1 mount_linux.go:211] systemd-run output: Failed to create bus connection: No such file or directory
I1012 15:43:43.843604       1 mount_linux.go:471] Checking for issues with fsck on disk: /dev/rbd1
I1012 15:43:43.872778       1 mount_linux.go:484] `fsck` error fsck from util-linux 2.23.2
fsck.ext2: Bad magic number in super-block while trying to open /dev/rbd1
/dev/rbd1: 
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I1012 15:43:43.872797       1 mount_linux.go:490] Attempting to mount disk: xfs /dev/rbd1 /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount
I1012 15:43:43.872809       1 mount_linux.go:146] Mounting cmd (mount) with arguments ([-t xfs -o defaults /dev/rbd1 /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount])
E1012 15:43:43.875501       1 mount_linux.go:151] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs -o defaults /dev/rbd1 /var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount
Output: mount: wrong fs type, bad option, bad superblock on /dev/rbd1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

I1012 15:43:43.875518       1 mount_linux.go:545] Attempting to determine if disk "/dev/rbd1" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd1])
I1012 15:43:43.900261       1 mount_linux.go:548] Output: "", err: exit status 2
I1012 15:43:43.900286       1 mount_linux.go:519] Disk "/dev/rbd1" appears to be unformatted, attempting to format as type: "xfs" with options: [/dev/rbd1]
E1012 15:43:43.900334       1 mount_linux.go:526] format of disk "/dev/rbd1" failed: type:("xfs") target:("/var/lib/kubelet/pods/4a3c555d-ce35-11e8-b182-02ccef257a4a/volumes/kubernetes.io~csi/pvc-43edb969ce3511e8/mount") options:(["defaults"])error:(executable file not found in $PATH)
E1012 15:43:43.900351       1 utils.go:100] GRPC error: executable file not found in $PATH

Looks like it tries to mount it, fails because it is not formatted yet, tries to format it, then fails because a required executable is missing.

My best guess is that it tries to call mkfs.xfs which is not in the container:

$ kubectl exec csi-rbdplugin-c2fg2 -c csi-rbdplugin ls /usr/sbin | grep mkfs
mkfs
mkfs.cramfs
mkfs.ext2
mkfs.ext3
mkfs.ext4
mkfs.minix

When I use fsType: ext4, everything works fine - the initial mount fails, but then it successfully formats it, and the pod comes up.

StorageClass, PersistentVolumeClaim, and Pod: https://gist.github.com/Cynerva/e54f3852f5d35a6482c711d5f22c874e

ceph / ceph-csi

csi-rbdplugin FailedMount with fsType: xfs #87