ctdb-is-experimental: Failing to multi attach in a clustered setup

[spuiuk]

apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbShare
metadata:
  name: smbshare3
spec:
  scaling:
    availabilityMode: clustered
    minClusterSize: 2
  storage:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  readOnly: false
  shareName: "smbshare3"

[sprabhu@fedora samba-operator]$ kubectl get pods
NAME                               READY   STATUS     RESTARTS   AGE
samba-ad-server-86b7dd9856-zkptq   1/1     Running    0          5h22m
smbshare3-0                        3/3     Running    0          6m31s
smbshare3-1                        0/3     Init:0/4   0          6m10s

Checking with kubectl describe smbshare3-1, we see

  Type     Reason                  Age    From                     Message
  ----     ------                  ----   ----                     -------
  Normal   Scheduled               3m20s  default-scheduler        Successfully assigned default/smbshare3-1 to minikube-m02
  Warning  FailedAttachVolume      3m20s  attachdetach-controller  Multi-Attach error for volume "pvc-10d65634-15ab-4db1-ad35-243e9589d861" Volume is already used by pod(s) smbshare3-0
  Normal   SuccessfulAttachVolume  3m19s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-7d070fb6-76c0-48d7-a629-a99e3d4ee2a6"
  Warning  FailedMount             77s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[smbshare3-pvc-smb], unattached volumes=[smbshare3-state-ctdb ctdb-config ctdb-volatile samba-container-config ctdb-sockets samba-state-dir kube-api-access-8kcnn ctdb-persistent smbshare3-pvc-smb]: timed out waiting for the condition

[sprabhu@fedora samba-operator]$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
smbshare3-pvc     Bound    pvc-10d65634-15ab-4db1-ad35-243e9589d861   1Gi        RWO            rook-cephfs    7m41s
smbshare3-state   Bound    pvc-7d070fb6-76c0-48d7-a629-a99e3d4ee2a6   1Gi        RWX            rook-cephfs    7m41s

[spuiuk]

From a rook-ceph bug report I read,

RWO PVC can be attached in multiple pods on the same nodes, if you try to attach RWO PVC on multiple nodes you will face this error

smbshare3-pvc is created in RWO mode in my configuration and this seems to have caused this problem.

[spuiuk]

This is fixed by changing the PVC mode to ReadWriteMany

[sprabhu@fedora samba-operator]$ cat ~/smbshare3.yaml 
apiVersion: samba-operator.samba.org/v1alpha1
kind: SmbShare
metadata:
  name: smbshare3
spec:
  scaling:
    availabilityMode: clustered
    minClusterSize: 2
  storage:
    pvc:
      spec:
        accessModes:
          - ReadWriteMany
        resources:
          requests:
            storage: 1Gi
  readOnly: false
  shareName: "smbshare3"

[sprabhu@fedora samba-operator]$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
smbshare3-pvc     Bound    pvc-3eb64188-fc20-40f8-8c54-65660f10e91b   1Gi        RWX            rook-cephfs    3m32s
smbshare3-state   Bound    pvc-fe7ccd76-1701-401f-af3c-d95a1e0f9b87   1Gi        RWX            rook-cephfs    3m31s

[spuiuk]

NotABug

[phlogistonjohn]

From a UX point of view it is somewhat annoying to create the resource and then have it fail somewhat "mysteriously" later. We could look into validating the spec through an adminssion controller in the future.