[EKS] [request]: Support IPv6 only subnets

[ajpauwels]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request In an ipv6 cluster with a public/private subnet setup (nodes in private subnets, NAT gateways and public LBs in public subnets), I would like my private node subnets to be IPv6-only, with the only IPv4 assignments happening in public subnets if dual-stack support is necessary.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? EKS' current IPv6 implementation requires subnets with nodes in them to have an IPv4 CIDR allocated, in order to assign nodes IPv4 addresses alongside their IPv6 addresses. From what I understand, this was to support outbound requests to IPv4 services when regions did not have DNS64/NAT64 support: DNS returns an A record, the pod is assigned a node-local IPv4 address, then that address is NAT'ed to the node's IPv4 before going out to the IGW.

As of March of this year, however, DNS64 and NAT64 capabilities have been expanded to all AWS commercial and govcloud regions (https://aws.amazon.com/about-aws/whats-new/2022/02/aws-expands-nat64-dns64-regions/). By enabling DNS64 in the node subnet and adding a route table entry forwarding 64:ff9b::/96 to a NAT gateway in a public subnet with an IPv4 address, it should be possible to support outbound IPv4 without requiring the private node subnets to have IPv4 CIDRs assigned.

I think this is important as it:

1. Eliminates the need for custom CNI logic for outbound IPv4 support in IPv6 clusters
2. Eliminates scaling restrictions caused by address exhaustion of small IPv4 subnets in clusters with large numbers of nodes

Are you currently working around this issue? I currently have everything setup as described above, with DNS64 and NAT64, and everything is working fine. I would just like to be able to flip the ipv6_native switch in the terraform definition of my VPC and completely eliminate any assignment of IPv4 CIDRs to my private subnets. If there's a way to disable the custom CNI logic to make sure it's not picking up something I'm missing, I'm happy to test.

If you made it this far, thank you very much :)

[xjiaqing]

I don't know why it is not supported yet, it is very needed for me.

[rkujawa]

I don't need IPv4 in the EKS subnet. It would be completely adequate to use DNS64 and NAT64 for outgoing IPv4 connectivity. It would make the whole setup somewhat less complex.

[ajpauwels]

Looks like it's in the "researching" phase as of early november so might be hearing good news soon! Hopefully it could apply to ECS as well as I don't believe it's possible to deploy ECS tasks in an IPv6-only subnet.

[Spacefish]

ECR does not support ipv6, so probably ECS is unable to pull the containers from the registry in an ipv6 only scenario. Maybe it works with external registries with ipv6 endpoints?

[ajpauwels]

ECR does not support ipv6, so probably ECS is unable to pull the containers from the registry in an ipv6 only scenario. Maybe it works with external registries with ipv6 endpoints?

@Spacefish With DNS64 and NAT64, EKS could pull containers from ECR even if ECR does not have a AAAA record.

[Spacefish]

With DNS64 and NAT64, EKS could pull containers from ECR even if ECR does not have a AAAA record.

Yes, possible, but NAT64 is a technology that shouldn´t have existed in the first place.. It makes next to no sense.. As you get all the drawbacks of NAT (still have to keep state tables somewhere) and gain almost nothing.. In this scenario you are better off asigning an private ipv4 to the clients and just using NATv4 so you don´t have the extra overhead of the ipv6 header + packet length changes in the NAT64 Equipment.

The benefit of ipv6 is that you don´t have to do NAT in the first place.

[galb-see]

Yes, possible, but NAT64 is a technology that shouldn´t have existed in the first place.. It makes next to no sense.. As you get all the drawbacks of NAT (still have to keep state tables somewhere) and gain almost nothing.. In this scenario you are better off asigning an private ipv4 to the clients and just using NATv4 so you don´t have the extra overhead of the ipv6 header + packet length changes in the NAT64 Equipment.

The benefit of ipv6 is that you don´t have to do NAT in the first place.

Another benfit of using IPv6 only subnet for EKS, with the NAT setup (depite it, obviously, not being ideal), is that you're not limited by the number of private IPv4 IPs available in the subnet.

[ajpauwels]

With DNS64 and NAT64, EKS could pull containers from ECR even if ECR does not have a AAAA record.

Yes, possible, but NAT64 is a technology that shouldn´t have existed in the first place.. It makes next to no sense.. As you get all the drawbacks of NAT (still have to keep state tables somewhere) and gain almost nothing.. In this scenario you are better off asigning an private ipv4 to the clients and just using NATv4 so you don´t have the extra overhead of the ipv6 header + packet length changes in the NAT64 Equipment.

The benefit of ipv6 is that you don´t have to do NAT in the first place.

Sure it's certainly not ideal but, regardless of ECR having a AAAA record, IPv6 support across the world is spotty enough that I imagine a large portion of people would need DNS64/NAT64 regardless in order to support making requests to services outside of AWS. It was always going to be a dirty half-measure, but a necessary one as the world's infrastructure catches up with IPv6.

Does it excuse ECR not having IPv6? No, but at this point it has taken SO long for ECR to have IPv6 support I imagine there has to be some specific problem as to why they haven't just flipped the switch. I naively hope it isn't simply out of a desire to keep the income streams from public IPv4 addresses/NAT gateways.

[xjiaqing]

My opinion is that supporting IPv6 only-subnet in eks clusters is a very real need, regardless of whether ECR supports ipv6 endpoints. Not everyone uses ECR as the image repository.

ECR's support for IPv6 is another thing, it is not an excuse for EKS not supporting IPv6-Only subnet.

[rmeloca]

Is this request really one of the last priorities in the backlog? Real shame if so

[Spacefish]

I naively hope it isn't simply out of a desire to keep the income streams from public IPv4 addresses/NAT gateways.

I don´t think so. Furthermore these income streams wouldn´t dry out as soon as they enable it, a lot of people will still opt for the ipv4 / the default setting anyway.

[victorvhpg]

Any update ?

[Spacefish]

IPv6 support across the world is spotty enough that I imagine a large portion of people would need DNS64/NAT64 regardless in order to support making requests to services outside of AWS.

If you need that / need to talk to random "outside" services, you would probably just use one ipv4 / NAT Gateway for that purpose.. But i would say ~80-90% of Containers don´t talk to the outside world at all and just run some kind of backend job which needs to be accessed from either other containers in the same VPC or from a Application Loadbalancer which has it´s own VPC for the "public" side of things.

[ajpauwels]

IPv6 support across the world is spotty enough that I imagine a large portion of people would need DNS64/NAT64 regardless in order to support making requests to services outside of AWS.

If you need that / need to talk to random "outside" services, you would probably just use one ipv4 / NAT Gateway for that purpose.. But i would say ~80-90% of Containers don´t talk to the outside world at all and just run some kind of backend job which needs to be accessed from either other containers in the same VPC or from a Application Loadbalancer which has it´s own VPC for the "public" side of things.

That's what I mean by DNS64/NAT64. DNS64 just means your A records get converted to AAAA that point to your NAT gateway, and the NAT gateway takes care of the translation. But it means you need to pay hourly for the NAT gateway and for the elastic IPv4 address assigned to it.