ceph regional storage

[WladyX]

I have 3 proxmox nodes:

cube
tiny
tiny2

i have defined this sc:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: proxmox-sc-ceph
parameters:
  csi.storage.k8s.io/fstype: ext4
  storage: ceph-pool
provisioner: csi.proxmox.sinextra.dev
reclaimPolicy: Delete
volumeBindingMode: Immediate

when i create a PVC, the PV gets bound to a node, not regional (i think):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: foo-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: "proxmox-sc-ceph" # Empty string must be explicitly set otherwise default StorageClass will be set
  resources:
    requests:
      storage: 1Mi

❯ k describe pv pvc-5aea9ff4-6cb4-4431-b9db-057a98c5082f
Name:              pvc-5aea9ff4-6cb4-4431-b9db-057a98c5082f
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.proxmox.sinextra.dev
                   volume.kubernetes.io/provisioner-deletion-secret-name:
                   volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      proxmox-sc-ceph
Status:            Bound
Claim:             default/foo-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:
  Required Terms:
    Term 0:        topology.kubernetes.io/region in [prxcls]
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.proxmox.sinextra.dev
    FSType:            ext4
    VolumeHandle:      prxcls/tiny/ceph-pool/vm-9999-pvc-5aea9ff4-6cb4-4431-b9db-057a98c5082f
    ReadOnly:          false
    VolumeAttributes:      storage=ceph-pool
                           storage.kubernetes.io/csiProvisionerIdentity=1703323894432-7509-csi.proxmox.sinextra.dev
Events:                <none>

my guess is that VolumeHandle should not include tiny, because for other PVC i get erorrs like

0s          Warning   FailedAttachVolume       pod/test-pod                                               AttachVolume.Attach failed for volume "pvc-a5764d29-8de7-4d5f-b907-6c91375000b8" : rpc error: code = InvalidArgument desc = volume prxcls/cube/ceph-pool/vm-9999-pvc-a5764d29-8de7-4d5f-b907-6c91375000b8 does not exist on the node k8s-wn2

am i doing something wrong? or is this a bug? Thank you for this CSI and for your support!

[sergelogvinov]

Hello, i think you first who use this CSI with ceph. I've never tried ceph with proxmox... But lets understand what wrong here.

pv pvc-5aea9ff4-6cb4-4431-b9db-057a98c5082f has only regional Node Affinity. So it is good. (can migrate between nodes)

But csi-controller cannot find the block device prxcls/cube/ceph-pool/vm-9999-pvc-a5764d29-8de7-4d5f-b907-6c91375000b8 - probably this is the issue. It is trying to find the disk with wrong address.

Can you attach the block device vm-9999-pvc-a5764d29-8de7-4d5f-b907-6c91375000b8 to the VM manually, and show the result of VM-config? (/etc/pve/qemu-server/$id.conf)

Thanks.

[WladyX]

no ideea how to manually attach, if you can tell me i would be glad to do it. however i have put the nodeselector on a pod to force it to go to the cube node so i can paste the conf, i guess this is what you are looking for:

 cat /etc/pve/qemu-server/100.conf
#Built on 2023-12-22 18%3A18%3A53 UTC
agent: 1
balloon: 0
boot: order=scsi0;ide2;net0;ide3
cores: 4
cpu: cputype=x86-64-v2-AES
cpuunits: 1024
ide0: ceph-pool:vm-100-cloudinit,media=cdrom,size=4M
ide2: none,media=cdrom
ide3: none,media=cdrom
kvm: 1
memory: 16384
meta: creation-qemu=8.1.2,ctime=1703269172
name: k8s-wn1
net0: virtio=00:00:00:00:20:01,bridge=vmbr0,firewall=0
numa: 0
onboot: 1
ostype: l26
scsi0: ceph-pool:vm-100-disk-0,discard=on,iothread=1,size=50G
scsi1: ceph-pool:vm-9999-pvc-cb7859d1-9eb1-4cc7-a147-b1521524c08a,backup=0,iothread=1,size=1G,wwn=0x5056432d49443031
scsihw: virtio-scsi-single
smbios1: uuid=b7b7ae20-2ea0-45fb-a8d2-2fd09370e80e
sockets: 2
vmgenid: 1816cd12-29ec-4c09-bb77-8b6fd9d9702d

the pod with the problem is sitting on k8s-wn2 VM which is not on cube node, but the path is prxcls/cube/ceph-pool/vm-9999 when it should be prxcls/tiny/ceph-pool/vm-9999 (the node with k8s-wn2) or just prxcls/ceph-pool/vm-9999 i guess.

attaching a picture to help you understand the layout

[sergelogvinov]

I've found many price of buggy code with regional (shared) disk (i do not have any shared disks).

I think i can add small patch, but we need more changes in future.

[WladyX]

Initial tests look good, was able to move pod across nodes and PV was mounted accordingly. Thank you @sergelogvinov !