Cannot mount external gluster volume

MatteoManzoni commented 2 years ago

Hi guys, I was playing with my homelab kubernetes cluster and I was trying to mount an external gluster volume but without any success. The pod that supposedly should mount the kadalu provisioned pv returned the following error:

Warning  FailedMount             34s (x14 over 14m)   kubelet                  MountVolume.SetUp failed for volume "pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d" : rpc error: code = Unknown desc = Exception calling application: [32]  mount: /var/lib/kubelet/pods/297e6945-e738-4926-adb7-0bd27db24873/volumes/kubernetes.io~csi/pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d/mount: special device /mnt/k8s-kadalu/subvol/d2/09/pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d does not exist

Digging a little deeper with kubectl kadalu logs I saw something strange in the instance of kadalu-csi-nodeplugin residing in the same node of the pod mounting the pvc:

[2022-01-21 15:50:29,415] ERROR [_server - 454:_call_behavior] - Exception calling application: 'bricks'
Traceback (most recent call last):
  File "/kadalu/lib/python3.9/site-packages/grpc/_server.py", line 444, in _call_behavior
    response_or_iterator = behavior(argument, context)
  File "/kadalu/nodeserver.py", line 106, in NodePublishVolume
    mountpoint = mount_glusterfs(volume, mntdir, storage_options, True)
  File "/kadalu/volumeutils.py", line 1128, in mount_glusterfs
    generate_client_volfile(volname)
  File "/kadalu/volumeutils.py", line 925, in generate_client_volfile
    for i in range(0, int(len(data["bricks"]) / count)):
KeyError: 'bricks'
[2022-01-21 15:50:30,228] INFO [volumeutils - 1325:mount_glusterfs_with_host] - mount command failed     cmd=['/opt/sbin/glusterfs', '--process-name', 'fuse', '-l', '/var/log/gluster/gluster.log', '--volfile-id', 'k8s-kadalu', '--volfile-server', 'mttmnz-storage01', '--volfile-server', 'mttmnz-storage02', '/mnt/k8s-kadalu', '/mnt/k8s-kadalu'] error=[64]  Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
  or:  glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
Try `glusterfs --help' or `glusterfs --usage' for more information.
[2022-01-21 15:50:30,327] ERROR [_server - 454:_call_behavior] - Exception calling application: [32]  mount: /var/lib/kubelet/pods/297e6945-e738-4926-adb7-0bd27db24873/volumes/kubernetes.io~csi/pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d/mount: special device /mnt/k8s-kadalu/subvol/d2/09/pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d does not exist.
Traceback (most recent call last):
  File "/kadalu/lib/python3.9/site-packages/grpc/_server.py", line 444, in _call_behavior
    response_or_iterator = behavior(argument, context)
  File "/kadalu/nodeserver.py", line 121, in NodePublishVolume
    if mount_volume(pvpath_full, request.target_path, pvtype, fstype=None):
  File "/kadalu/volumeutils.py", line 852, in mount_volume
    execute(MOUNT_CMD, "--bind", pvpath, mountpoint)
  File "/kadalu/kadalulib.py", line 156, in execute
    raise CommandException(proc.returncode, out.strip(), err.strip())
kadalulib.CommandException: [32]  mount: /var/lib/kubelet/pods/297e6945-e738-4926-adb7-0bd27db24873/volumes/kubernetes.io~csi/pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d/mount: special device /mnt/k8s-kadalu/subvol/d2/09/pvc-3f7bf2c3-66b5-47d9-9116-7bdf050b239d does not exist.
[2022-01-21 15:50:31,449] INFO [volumeutils - 1325:mount_glusterfs_with_host] - mount command failed     cmd=['/opt/sbin/glusterfs', '--process-name', 'fuse', '-l', '/var/log/gluster/gluster.log', '--volfile-id', 'k8s-kadalu', '--volfile-server', 'mttmnz-storage01', '--volfile-server', 'mttmnz-storage02', '/mnt/k8s-kadalu', '/mnt/k8s-kadalu'] error=[64]  Usage: glusterfs [OPTION...] --volfile-server=SERVER [MOUNT-POINT]
  or:  glusterfs [OPTION...] --volfile=VOLFILE [MOUNT-POINT]
Try `glusterfs --help' or `glusterfs --usage' for more information.

KadaluStorage:

apiVersion: kadalu-operator.storage/v1alpha1
kind: KadaluStorage
metadata:
  name: k8s-kadalu
  namespace: kadalu
spec:
  details:
    gluster_hosts:
    - mttmnz-storage01
    - mttmnz-storage02
    gluster_volname: k8s-kadalu
  kadalu_format: native
  pvReclaimPolicy: delete
  type: External

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: traefik-acme-pvc
  namespace: traefik-ingresses
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: kadalu.k8s-kadalu

Kadalu versions:

kubectl-kadalu plugin: 0.8.9
kadalu pod(s) versions
pod/kadalu-csi-nodeplugin-998x8: 0.8.9
pod/kadalu-csi-nodeplugin-fp2hh: 0.8.9
pod/kadalu-csi-nodeplugin-fwszr: 0.8.9
pod/kadalu-csi-nodeplugin-hqmz5: 0.8.9
pod/kadalu-csi-nodeplugin-trkzm: 0.8.9
pod/kadalu-csi-nodeplugin-x68mp: 0.8.9
pod/kadalu-csi-provisioner-0: 0.8.8
pod/operator-5dc89c99f8-nfhch: 0.8.8

External server gluster version: glusterfs 9.3 K8s version: 1.23.1 CRI: CRI-O 1.23

Am I doing something wrong? Can I help troubleshoot the issue further?

leelavg commented 2 years ago

@MatteoManzoni pardon the inconvenience and thanks for troubleshooting the issue so far.

either #745 or #750 whichever gets merged will sove the issue
this is a regression for external gluster 😔

MatteoManzoni commented 2 years ago

Hi @leelavg thank you so much for your feedback, can you please point me to a release prior to the introduction of this regression?

leelavg commented 2 years ago

the fix itself is very easy however it involves editing atleast two files in different containers while they are running, still I encourage to apply #745 to your cluster (or)
you should probably go back before Oct release, so 0.8.6 is a good candidate
we'll try to add external gluster at the earliest possible to CI and this already regressed twice

kadalu / kadalu

Cannot mount external gluster volume #754