Migrate volume from csi-lvm to csi-driver-lvm

It may sooner or later be necessary to provide a way to migrate volumes from csi-lvm to csi-driver-lvm.

Idea: provide a tool, like others do, e.g.: tridenctl upgrade volume <name-of-trident-volume>

Following manually executed flow workes: (DO NOT USE THIS EXAMPLE. I might be full of bugs) These steps should better be done by a cli migration tool.

0.) make sure volume is not attached to a pod (scale sts to replicas=0 or delete a standalone pod)

1.) get the name of the node where the volume is mounted on:

k get pvc lvm-pvc-linear -o jsonpath='{.metadata.annotations.volume\.kubernetes\.io/selected-node}' && echo
shoot--ph2j95--mwen2-default-worker-579867dc45-w6r6c

2.) find pv for the given pvc

NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
lvm-pvc-linear   Bound    pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281   10Mi       RWO            csi-lvm        2m7s

3.) set reclaimPolicy to retain: k patch pv pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281 -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'

k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   REASON   AGE
pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281   10Mi       RWO            Retain           Bound    default/lvm-pvc-linear   csi-lvm                 4m25s

4.) delete old pvc and replace with new pvc

 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: lvm-pvc-linear
   namespace: default
-  annotations:
-    csi-lvm.metal-stack.io/type: "linear"
 spec:
   accessModes:
     - ReadWriteOnce
-  storageClassName: csi-lvm
+  storageClassName: csi-lvm-sc-linear
   resources:
     requests:
-      storage: 10Mi
+      storage: 1Mi  # small dummy value, will be resized later

5.) force the actual creation of the volume on the target node (i.e. create a pod on that node which claims the pvc), e.g.

apiVersion: batch/v1
kind: Job
metadata:
  name: mount
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchFields:
              - key: metadata.name
                operator: In
                values:
                - shoot--ph2j95--mwen2-default-worker-579867dc45-w6r6c
      serviceAccount: csi-lvmplugin
      serviceAccountName: csi-lvmplugin
      containers:
      - name: busybox
        image: busybox
        command: ["sleep",  "10"]
        volumeMounts:
        - name: dummy
          mountPath: /dummy
      restartPolicy: Never
      volumes:
      - name: dummy
        persistentVolumeClaim:
          claimName: lvm-pvc-linear

6.) get volume name of the newly created volume:

 k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                    STORAGECLASS        REASON   AGE
pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281   10Mi       RWO            Retain           Released   default/lvm-pvc-linear   csi-lvm                      41m
pvc-9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a   1Mi        RWO            Delete           Bound      default/lvm-pvc-linear   csi-lvm-sc-linear            16m

7.) rename the old lv to the newly created one, and change tags

  umount /tmp/pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281 && \
  lvremove -y csi-lvm/pvc-9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a && \
  lvrename csi-lvm/pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281 csi-lvm/pvc-9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a && \
  lvchange --deltag lv.metal-stack.io/csi-lvm csi-lvm/9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a && \
  lvchange --addtag vg.metal-stack.io/csi-lvm-driver csi-lvm/9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a

For demonstration purposes, this can be done by a job. Another way would be to exec into the corresponding csi-lvm-reviver pod.

apiVersion: batch/v1
kind: Job
metadata:
  name: rename-lvm-pv
spec:
  backoffLimit: 0
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchFields:
              - key: metadata.name
                operator: In
                values:
                - shoot--ph2j95--mwen2-default-worker-579867dc45-w6r6c
      containers:
      - name: rename-lvm-pv
        image: metalstack/lvmplugin:latest
        command: ["/bin/sh", "-c"]
        args: [ "umount /tmp/pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281 && \
                 lvremove -y csi-lvm/pvc-9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a && \
                 lvrename csi-lvm/pvc-47b80ab6-e98f-4ff1-bdc1-f4f739e2e281 csi-lvm/pvc-9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a && \
                 lvchange --deltag lv.metal-stack.io/csi-lvm csi-lvm/9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a && \
                 lvchange --addtag vg.metal-stack.io/csi-lvm-driver csi-lvm/9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a" ]
        securityContext:
          privileged: true
        terminationMessagePath: /termination.log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /dev
          mountPropagation: Bidirectional
          name: dev-dir
        - mountPath: /lib/modules
          name: mod-dir
        - mountPath: /etc/lvm/backup
          mountPropagation: Bidirectional
          name: lvmbackup
        - mountPath: /etc/lvm/cache
          mountPropagation: Bidirectional
          name: lvmcache
        - mountPath: /run/lock/lvm
          mountPropagation: Bidirectional
          name: lvmlock
        - mountPath: /tmp/csi-lvm
          mountPropagation: Bidirectional
          name: data
      restartPolicy: Never
      serviceAccount: csi-lvmplugin
      serviceAccountName: csi-lvmplugin
      volumes:
      - hostPath:
          path: /dev
          type: Directory
        name: dev-dir
      - hostPath:
          path: /lib/modules
          type: Directory
        name: mod-dir
      - hostPath:
          path: /etc/lvm/backup
          type: DirectoryOrCreate
        name: lvmbackup
      - hostPath:
          path: /etc/lvm/cache
          type: DirectoryOrCreate
        name: lvmcache
      - hostPath:
          path: /run/lock/lvm
          type: DirectoryOrCreate
        name: lvmlock
      - hostPath:
          path: /tmp/csi-lvm
          type: DirectoryOrCreate
        name: data

8.) "resize" the pvc entry back to its original size (doesn't actually change anything, since the volume on disk already has this size) k patch pvc lvm-pvc-linear -p '{"spec": {"resources": {"requests": { "storage": "10Mi"}}}}'

9.) start old pod/sts with updated pvc (now has the new storage class but the original content)

10.) delete the pv entry of the old, unused csi-lvm volume

k delete pv pvc-9cba4237-1e56-4f2a-9b49-ca7f94bfdb6a

metal-stack / csi-driver-lvm

provide a way/tool to migrate volumes from csi-lvm to csi-driver-lvm #21

Migrate volume from csi-lvm to csi-driver-lvm