SPC(disk replacement): steps to replace disk in SPC based pools

[mittachaitu]

How can we replace disk in cStor pool was created using SPC?

[mittachaitu]

Replace a Disk in cStor pool

0. Creating a cStor Pool

• Show the disks. kubectl get bd -n openebs --show-labels

• Create a claim.

  apiVersion: openebs.io/v1alpha1
  kind: StoragePoolClaim
  metadata:
  name: cstor-disk-pool
  spec:
  name: cstor-disk-pool
  type: disk
  poolSpec:
  poolType: mirrored
  blockDevices:
  blockDeviceList:

• Apply the claim. kubectl apply -f spc.yaml

• Verify the claim. kubectl get csp

• Verify pool creation on appropriate nodes. kubectl get pod -n openebs -o wide

• Create a StorageClass that uses the new claim.

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
  name: openebs-standard
  annotations:
  openebs.io/cas-type: cstor
  cas.openebs.io/config: |
    - name: StoragePoolClaim
      value: "cstor-disk-pool"

  provisioner: openebs.io/provisioner-iscsi

  1. Replacing a disk

• Exec into the pod associated with the disk that needs to be replace. kubectl exec -it cstor-disk-pool-nq1i-7c686f8cb9-zbc4n -n openebs -ccstor-pool-mgmt bash

• Use zpool command-line utility to check the disk status. zpool status

• Find the device node for the replacement disk. kubectl get bd -n openebs blockdevice-bd952ccaa0638d1a5f7ab334e65d7aad -o yaml

• e.g.

  ...
  devlinks:
  - kind: by-id
  links:
  - /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-10
  - /dev/disk/by-id/google-cstor-it-disk-10
  ...

• Create a BlockDeviceClaim for the new disk.

  apiVersion: openebs.io/v1alpha1
  kind: BlockDeviceClaim
  metadata:
  finalizers:
  - storagepoolclaim.openebs.io/finalizer
  Labels:
   # Put the name of your SPC
  openebs.io/storage-pool-claim: cstor-disk-pool
  # Put the name of bdc as bdc-<bd-uid>
  # This 'bd-uid' is the uid of new BD that we are going to use
  # The UID can be found via 'kubectl get bd <bd-name> -o yaml' 
  name: bdc-2a1e53a2-d06e-11e9-bfdd-42010a8000a5
  # If you have installed openebs in a different namespaces then put 
  # then put that namespace here.
  namespace: openebs
  ownerReferences:
  - apiVersion: openebs.io/v1alpha1
  blockOwnerDeletion: true
  controller: true
  kind: StoragePoolClaim
  # Put the name of your SPC
  name: cstor-disk-pool
  # Put the UID of your SPC
  uid: 70df195f-d08d-11e9-bfdd-42010a8000a5
  Spec:
  # Put the block device name that you are going to use as a replacement
  blockDeviceName: blockdevice-bd952ccaa0638d1a5f7ab334e65d7aad 
  # This is the host name of the new block device that we are going to
  # for replacement.
  hostName: gke-cstor-it-pool-1-bdf8ee93-tv69 
  resources:
  Requests:
     # Put the capacity of BD in GB here. Disks greater than 
     # or equal to this size will match.
    storage: 100G

• Apply the above bdc. kubectl apply -f bdc.yaml

• Verify whether status of claim(it should be in Bound) kubectl get bdc -n openebs

• Use zpool to replace the disk.

  zpool replace cstor-7108b3c7-d08d-11e9-bfdd-42010a8000a5  /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-9 /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-10

  where cstor-7108b3c7-d08d-11e9-bfdd-42010a8000a5 is the pool name, /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-9 is the old disk, and /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-10 is the replacement.

• Monitor progress. zpool status

Now we need to replace the blockdevice name in CSP and SPC resource to intact with replaced blockdevice

• Disable reconciliation on SPC By adding annotation reconcile.openebs.io/disable: true on SPC we can disable reconciliation of SPC kubectl edit spc <spc_name>

• Replace blockdevice on CSP Replace the blockdevice name and device link entry with new blockdevice name and device link using kubectl edit csp <csp_name> Note: csp_name will be the pool name where we performed replacement.

• Verify whether CSP has new blockdevice kubectl get csp <csp_name> -o yaml

• Remove device link entry and replace with new device. kubectl edit spc cstor-disk-pool

• Remove the block device claim for the replaced disk. kubectl delete bdc <old_blockdevice_claim_name> -n openebs after this remove finalizer entry on storagepoolclaim.openebs.io/finalizer the same bdc.

• Enable Reconciliation on SPC Enable the reconciliation on SPC by removing the annotation reconcile.openebs.io/disable: true kubectl edit spc <spc_name>