search

AmitKumarDas / Decisions

Apache License 2.0
10 stars 3 forks source link

K8s: Amazing Controllers ~ google csi driver #226

Open AmitKumarDas opened 4 years ago

AmitKumarDas commented 4 years ago

Motivation: This article shows how a volume (Google PD) can be attached, detached to a Kubernetes node. This can also be used as a reference to show how the same volume can be re-attached to a different Kubernetes node. VolumeAttachment API (csi attacher controller) is the one which makes this a reality with Kubernetes as the client side CSI implementation.

References

Kubernetes Cluster Details

  kubernetes > kubectl get nodes
NAME                                       STATUS   ROLES    AGE   VERSION
gke-amitd-ddp-default-pool-d5aa3f95-ht99   Ready    <none>   92m   v1.13.7-gke.24
gke-amitd-ddp-default-pool-d5aa3f95-t8p1   Ready    <none>   92m   v1.13.7-gke.24
gke-amitd-ddp-default-pool-d5aa3f95-wq0f   Ready    <none>   92m   v1.13.7-gke.24
  kubernetes >

Setup Details

CSI controller runs as a StatefulSet with single replica & uses following images:
- gcr.io/gke-release/csi-provisioner:v1.0.1-gke.0
- gcr.io/gke-release/csi-attacher:v1.0.1-gke.0
- gcr.io/gke-release/gcp-compute-persistent-disk-csi-driver:v0.4.0-gke.0
CSI node runs as a DaemonSet & uses following images:
- gcr.io/gke-release/csi-node-driver-registrar:v1.0.1-gke.0
- gcr.io/gke-release/gcp-compute-persistent-disk-csi-driver:v0.4.0-gke.0

Separate ServiceAccounts need to be created for following node as well as controller drivers

ClusterRole rules for provisioner need to be set for following resources:
- persistentvolumes
- persistentvolumeclaims
- storageclasses
- events
- nodes
- csinodes
ClusterRole rules for attacher need to be set for following resources:
- persistentvolumes
- nodes
- csinodes
- volumeattachments

PriorityClass needs to be set for both csi controller as well as csi node

Run Steps

  deploy > pwd
/home/amit/work/src/sigs.k8s.io/gcp-compute-persistent-disk-csi-driver/deploy

  deploy > ./setup-project.sh 
PROJECT is strong-xxx-112112
GCE_PD_SA_NAME is amitd-404
GCE_PD_SA_DIR is /home/amit/safe/gcp
Service account amitd-404@strong-xxx-112112.iam.gserviceaccount.com exists. Would you like to create a new one (y) or reuse the existing one (n)
(y/n)y
 deploy > ll /home/amit/mysafe/gcp/cloud-sa.json 
-rw------- 1 amit amit 2324 Sep 11 14:14 /home/amit/mysafe/gcp/cloud-sa.json

  deploy > export GCE_PD_SA_DIR=/home/amit/mysafe/gcp
  deploy > export GCE_PD_DRIVER_VERSION=stable

  ./kubernetes/deploy-driver.sh
  ddp > kubectl get sts
NAME                    READY   AGE
csi-gce-pd-controller   1/1     46s
  ddp > 
  ddp > kubectl get daemonset
NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
csi-gce-pd-node   3         3         3       3            3           <none>          51s
  ddp > 
  ddp > kubectl get po
NAME                      READY   STATUS    RESTARTS   AGE
csi-gce-pd-controller-0   3/3     Running   0          17s
csi-gce-pd-node-6f9rt     2/2     Running   0          17s
csi-gce-pd-node-cbb5v     2/2     Running   0          17s
csi-gce-pd-node-mwvwk     2/2     Running   0          17s
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: csi-gce-pd
provisioner: pd.csi.storage.gke.io
parameters:
  type: pd-standard
volumeBindingMode: Immediate
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: podpvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: csi-gce-pd
  resources:
    requests:
      storage: 4Gi
  kubernetes > kubectl get pvc
NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
podpvc   Bound    pvc-6e17e934-d473-11e9-9d63-42010a800067   4Gi        RWO            csi-gce-pd     8s
  kubernetes > 
  apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
    creationTimestamp: "2019-09-11T09:06:28Z"
    finalizers:
    - kubernetes.io/pv-protection
    name: pvc-6e17e934-d473-11e9-9d63-42010a800067
    resourceVersion: "12475"
    selfLink: /api/v1/persistentvolumes/pvc-6e17e934-d473-11e9-9d63-42010a800067
    uid: 70a718ec-d473-11e9-9d63-42010a800067
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 4Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: podpvc
      namespace: default
      resourceVersion: "12457"
      uid: 6e17e934-d473-11e9-9d63-42010a800067
    csi:
      driver: pd.csi.storage.gke.io
      fsType: ext4
      volumeAttributes:
        storage.kubernetes.io/csiProvisionerIdentity: 1568192432358-8081-
      volumeHandle: projects/strong-xxx-112112/zones/us-central1-a/disks/pvc-6e17e934-d473-11e9-9d63-42010a800067
    persistentVolumeReclaimPolicy: Delete
    storageClassName: csi-gce-pd
    volumeMode: Filesystem
  status:
    phase: Bound

Apply following VolumeAttachment resource

apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  name: my-attach
spec:
  attacher: pd.csi.storage.gke.io
  nodeName: gke-amitd-ddp-default-pool-d5aa3f95-ht99
  source:
    persistentVolumeName: pvc-6e17e934-d473-11e9-9d63-42010a800067

kubernetes > kubectl get volumeattachment my-attach -oyaml

apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  annotations:
    csi.alpha.kubernetes.io/node-id: projects/strong-xxx-112112/zones/us-central1-a/instances/gke-amitd-ddp-default-pool-d5aa3f95-ht99
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"VolumeAttachment","metadata":{"annotations":{},"name":"my-attach"},"spec":{"attacher":"pd.csi.storage.gke.io","nodeName":"gke-amitd-ddp-default-pool-d5aa3f95-ht99","source":{"persistentVolumeName":"pvc-6e17e934-d473-11e9-9d63-42010a800067"}}}
  creationTimestamp: "2019-09-11T09:28:08Z"
  finalizers:
  - external-attacher/pd-csi-storage-gke-io
  name: my-attach
  resourceVersion: "17008"
  selfLink: /apis/storage.k8s.io/v1/volumeattachments/my-attach
  uid: 7722574d-d476-11e9-9d63-42010a800067
spec:
  attacher: pd.csi.storage.gke.io
  nodeName: gke-amitd-ddp-default-pool-d5aa3f95-ht99
  source:
    persistentVolumeName: pvc-6e17e934-d473-11e9-9d63-42010a800067
status:
  attached: true
  kubernetes > kubectl get pv pvc-6e17e934-d473-11e9-9d63-42010a800067 -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
  creationTimestamp: "2019-09-11T09:06:28Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/pd-csi-storage-gke-io
  name: pvc-6e17e934-d473-11e9-9d63-42010a800067
  resourceVersion: "15021"
  selfLink: /api/v1/persistentvolumes/pvc-6e17e934-d473-11e9-9d63-42010a800067
  uid: 70a718ec-d473-11e9-9d63-42010a800067
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 4Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: podpvc
    namespace: default
    resourceVersion: "12457"
    uid: 6e17e934-d473-11e9-9d63-42010a800067
  csi:
    driver: pd.csi.storage.gke.io
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1568192432358-8081-
    volumeHandle: projects/strong-xxx-112112/zones/us-central1-a/disks/pvc-6e17e934-d473-11e9-9d63-42010a800067
  persistentVolumeReclaimPolicy: Delete
  storageClassName: csi-gce-pd
  volumeMode: Filesystem
status:
  phase: Bound
  kubernetes > kubectl delete volumeattachment --all
volumeattachment.storage.k8s.io "my-attach" deleted
  kubernetes >