[EKS] [request]: manually-provisioned PersistentVolumes lack failure-domain labels and nodeAffinity

Tell us about your request Dynamically-provisioned PersistentVolumes using the aws-ebs provisioner are automatically labeled with their zone and region and decorated with the corresponding nodeAffinity selectors:

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    failure-domain.beta.kubernetes.io/region: us-east-1
    failure-domain.beta.kubernetes.io/zone: us-east-1d
spec:
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: failure-domain.beta.kubernetes.io/zone
          operator: In
          values:
          - us-east-1d
        - key: failure-domain.beta.kubernetes.io/region
          operator: In
          values:
          - us-east-1

I believe that these labels and nodeAffinity selectors are required for Kubernetes's multi-zone awareness to work; i.e. they ensure a pod claiming such a volume is scheduled in the correct availability zone. I would like manually-provisioned PersistentVolumes to be automatically decorated with the same labels and selectors.

Reading the Kubernetes documentation suggests that this is already supported. In the article about Well-Known Labels, Annotations, and Taints, it says,

On the PersistentVolume: The PersistentVolumeLabel admission controller will automatically add zone labels to PersistentVolumes, on GCE and AWS.

However, in the Using Admission Controllers article, it says,

PersistentVolumeLabel is DEPRECATED and labeling persistent volumes has been taken over by cloud controller manager. Starting from 1.11, this admission controller is disabled by default.

Further reading about cloud controller manager suggests that it's part of the EKS control plane implementation. I enabled the EKS logs for "controller manager," but I did not see anything in those logs about failure-domain labeling, and my manually-created PVs are not labeled. It appears that neither the admission controller nor cloud controller manager is automatically labeling PersistentVolumes in EBS.

Which service(s) is this request for? EKS running Kubernetes 1.12

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I want to manually provision PersistentVolumes so that I can manage the lifecycle of the backing EBS volumes myself. In particular, I want to ensure that the data survives and can be re-attached in the event that I need to destroy and re-create the Kubernetes cluster.

Are you currently working around this issue? I am manually applying the labels and nodeAffinity selectors as shown in the YAML snippet above. This seems to have allowed my pods to be scheduled and bound to volumes in the correct availability zones. Note the docs also claim that node affinity is automatically populated for AWS EBS volumes, but manually adding the failure-domain labels was insufficient; I had to manually set node affinity, too.

aws / containers-roadmap

[EKS] [request]: manually-provisioned PersistentVolumes lack failure-domain labels and nodeAffinity #287