[EKS] [request]: EBS volume provisioner for StatefulSets

[heprotecbuthealsoattac]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Currently when using EBS for PVs the Stateful Set (or at least each of its pods) is bound to only one availability zone. Would it be possible to "migrate" the EBS volumes across availability zones in case a new pod is scheduled in a different az?

Which service(s) is this request for? EKS (PVC, PV, StorageClass)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? My application requires a persistent local filesystem. Right now I'm using Stateful Sets to achieve that, kubernetes wise it works well. However in case in which the node my pod runs on gets terminated it stops working as expected. Let's go through an example:

• ec2 instance 'i-abcd' runs in availability zone 'a' and belongs to an auto scaling group 'asg-foo'
• i-abcd is the only node in the cluster
• asg-foo's launch configuration is set to spin up instances in all the available availability zones that are available
• pet-0 pod (part of the stateful set 'pet') runs on instance i-abcd
• pet-0 has a PV associated which is an EBS volume 'vol-1234' which is located in availability zone 'a'
• i-abcd gets terminated
• asg-foo spins up a new instance 'i-wxyz' in availability zone 'b'
• vol-1234 cannot be attached to i-wxyz due to availability zone mismatch
• pet-0 is forever in pending state

Are you currently working around this issue? EFS storage class is AWS' recommendation, however NFS in this case is being used for single machine just to work around the az limitation. So in this case EFS' multi az capability is being used. To paraphrase Larry Wall's words: It's like trying to club someone to death with a loaded Uzi.

[wongma7]

Have you tried setting WaitForFirstConsumer on your StorageClass? https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode It should prevent your situation where a volume provisioned by kubernetes resides in one zone and scheduler schedules it in another.

[bhiravabhatla]

@wongma7 - This still wont the problem of existing pod moving to a worker node in another AZ ( this may happen for several reasons - upgrading, Crashing due to OOM etc).

[bhiravabhatla]

Any updates on this. Is this being looked into?