k8up backup doesn't support backing up openebs storage class PVs with RWO access

vistasunil commented 1 year ago

Description

Hello Team,

I was trying to use k8up to backup RWO PVs for openebs, iscsi and nfs storage classes each. It worked well for iscsi and nfs backed PVs but the backup pod failed with below error while mounting openebs backed PV for backup:

MountVolume.SetUp failed for volume "pvc-e67547c6-fd91-445b-8223-98694b15655e" : rpc error: code = Internal desc = verifyMount: device already mounted at [/var/lib/kubelet/pods/d0e858c5-8bce-4d1d-8d8e-07cf4abf8334/volumes/kubernetes.io~csi/pvc-e67547c6-fd91-445b-8223-98694b15655e/mount /var/lib/kubelet/pods/d0e858c5-8bce-4d1d-8d8e-07cf4abf8334/volume-subpaths/pvc-e67547c6-fd91-445b-8223-98694b15655e/yelb-db/0]

Kindly let us know, if k8up supports backing up openebs backed PVs. If yes, then please share the settings.

Thanks & Regards, Sunil Kumar

Additional Context

No response

Logs

No response

Expected Behavior

k8up should support backing up openebs backed PVs.

Steps To Reproduce

No response

Version of K8up

2.7.1

Version of Kubernetes

v1.24

Distribution of Kubernetes

Rancher

Kidswiss commented 1 year ago

Hi @vistasunil

In theory it should not make a difference for K8up what provider is behind the RWO PVCs. K8up tries to place the backup pods in such a way that, if they are mounted, it will run on the same node, thus allowing access to the data.

Openebs also seems to be using ISCSI under the hood, so it should behave the same as your other RWO PVCs. However we have not explicitly tested it with openebs. So there might be a chance of some implementation specific issue at hand.

One thing I could think of that lead to this issue: Is it possible that the pod mounting the openebs PVC has moved to another k8s node while the backup was running? If that happens, then the scheduled backup pod can't access the PVC anymore.

vistasunil commented 1 year ago

@Kidswiss I was also expecting same but backup did not work for openebs PVCs. Also, there was no pod movement during the backup run. So last statement do not stands valid case here. Can you please test the same at your end and let us know what is missing here?

vistasunil commented 1 year ago

@Kidswiss Any update on this?

rowa78 commented 11 months ago

if you configure your openebs-zfs-storageclass with parameter: shared: "yes" it works fine.

for example:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-zfs
parameters:
  recordsize: "128k"
  compression: "on"
  dedup: "off"
  fstype: "zfs"
  atime: "off"
  poolname: "zpool/pvcs"  
  shared: "yes"
provisioner: zfs.csi.openebs.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

see description here: https://github.com/openebs/zfs-localpv/blob/develop/docs/storageclasses.md#storageclass-for-sharing-the-persistence-volumes

k8up-io / k8up