search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.22k stars 321 forks source link

[EKS]: EKS Support for IPv6 #835

Closed mikestef9 closed 2 years ago

mikestef9 commented 4 years ago

Hi all,

Looking to get input on your IPv6 requirements as we develop our strategy here. IPv6 for EKS is a broad topic and your feedback will help us prioritize the most requested IPv6 scenarios.

Some topics that would be especially useful to get clarity on:

We have identified various milestones and they are outlined separately in the initial comments below. Please upvote this issue if you are interested in IPv6 in general, but also add a +1 to any of the milestone comments below that matter most to you.

For anything you feel is not listed as a milestone below, please open a separate feature request issue on the roadmap.

Looking forward to hear your thoughts here!

mikestef9 commented 4 years ago

External IPv6 clients communicating with EKS API server Note: This is separate to API server access from pods within the cluster (via X-ENI), which depends on pod addressing.

mikestef9 commented 4 years ago

External IPv6 clients communicating with pods Services deployed on EKS are accessible from the IPv6 Internet. This includes Ingress via ALB and ALB Ingress Controller, and Services of type=LoadBalancer via NLB and the AWS cloud provider. Pods may run IPv4.

mikestef9 commented 4 years ago

Pod to external IPv6 clients / Pod to pod (dual-stack) Pods are able to connect to IPv6 addresses outside the cluster, for both ingress and egress (depending on security group policy). Serves as a good intermediate testing ground for IPv6-only, or for serving IPv6 external users. Every pod would still require an IPv4 address, which is the upstream Kubernetes “dual-stack” feature (https://kubernetes.io/docs/concepts/services-networking/dual-stack/).

mikestef9 commented 4 years ago

External IPv6 clients to node / Node to external IPv6 clients (dual-stack) Nodes (ie: EC2 instances) can connect to IPv6 addresses outside the cluster, depending on security group policy. Pods running in host network namespace can use this IPv6 connectivity even though the rest of Kubernetes is IPv4 only. Anything connecting to kube-proxy NodePorts (including NLB/ALB) can ingress over IPv6 and be proxied to an IPv4 pod. This requires IPv6 enabled at the VPC and within the host operating system.

mikestef9 commented 4 years ago

IPv6 only Pods Requirements we have in mind:

Note: Pods might still have access to IPv4 using an egress protocol-NAT device, or using a private (to the pod/node) IPv4 address and NATing to the node’s IPv4 address (using iptables).

With this mode, IPv6-only pods do not consume an IPv4 address and allow you to scale clusters far beyond any IPv4 addressing plan.

mikestef9 commented 4 years ago

IPv6 only nodes Worker nodes only get IPv6 addresses. We think this feature will only be useful after IPv6-only pods.

ffrediani commented 4 years ago

Hello mike Most of the things shouldn't be optional to have it or not. Anything that communicates must have IPv6 support now a days. If the intent it to choose priorities then fine.

I would say that any VPC resources should be accessible via IPv6. Dual-stack is and will be the most common way, but IPv6-only is becoming a reality in IPv6-only Datacenters like Facebook does. However being able to be IPv6-only feature is the type of thing that can surely be less priority than basic IPv6 support. Certainly pods must connect to IPv6 internet as well. They must be able to serve data directly or via Loadbalancers so there are not more locks and reasons for people to put their content available via IPv6 as well.

ffrediani commented 4 years ago

Hello folks Do you have any update on it so far ? I am getting often excuses for not having IPv6 on something mostly due to the of IPv6 in EKS. Not nice to have these type of arguments really.

Do you believe we an expect it for the beginning on next half of 2020 already ?

ffrediani commented 4 years ago

It is due to attitudes like this that we are having this battle going on for 20 years for IPv6 to be where it should already be. It seems still that developers (always them) can't get IPv6 into their minds. They truly believe "it's Ok" to release a product to market in 2020 without having IPv6 support, just because "nobody uses, so it it unimportant and why should I care?". This is normally the most absurd. Anyone developing a product in 2020 should be very ashamed to release it without IPv6 regardless the reasons.

What a pity we have to still go through these scenarios. And that keeps facilitating for people to have their chain of excuses for not having IPv6 on something.

zajdee commented 4 years ago

Please consider providing a managed NAT64 service so that the pods can run single-stack, IPv6-only (this helps administration and design) while maintaining access to the IPv4 Internet.

Also it would be extremely useful to provide a /64 routed subnet to a EC2 instance, to avoid the mess fiddling with multiple ENIs and adding/removing individual addresses (as is the current approach in AWS CNI plugin).

Another extremely feature would be a /48 assignment per VPC (or at least multiple /56s) to help with that /64 per EC2 instance.

Another feature I'd love to see is dual-stack support for those Kubernetes clusters that process both IPv4 and IPv6 ingress traffic from the Internet (that is: dual-stack the ingress, single-stack the rest of the communication within the cluster).

IPv6 support in the Network Load Balancer is obviously the next item on the IPv6 support agenda. :-)

Thank you for considering any of those ideas.

NoseyNick commented 4 years ago

Honestly I'd be delighted with 100% IPv6-only on the inside, with dual-stack LBs on the outside to support those poor people who are still stuck on the olde IPv4 internet of the early 2000s 😛

sftim commented 4 years ago

Fargate IPv6 support would be really handy for a bunch of different reasons.

heidemn commented 4 years ago

External IPv6 clients communicating with pods Services deployed on EKS are accessible from the IPv6 Internet. This includes Ingress via ALB and ALB Ingress Controller

Isn't this already possible today with ALB ingress controller? I saw e.g. this PR https://github.com/kubernetes-sigs/aws-alb-ingress-controller/pull/991 , merged 08/2019. (not sure, since we're currently using Nginx + ELBv1)

autarchprinceps commented 4 years ago

Since CloudFront and/or ALB Ingress or Service LB could handle any IPv4 only clients, it would be great to have a fully IPv6 backend network option, but this isn't even possible in just VPC & EC2 yet, nevermind EKS. Since the reverse is also possible (as in using CloudFront or ALB for IPv6 clients on IPv4 only internals), the main point inside the VPC would be to save on or not require IPv4 ranges in VPCs, especially in Transit Gateway, VPC Peering, etc. scenarios, where reusing the same IP range for different VPCs over and over again, would cause problems.

ffrediani commented 4 years ago

How long more are we going to wait to have IPv6 support ? This should have come form day zero and has been the reason people use as excuse to not have IPv6 for public facing on many product.

DanielDamito commented 4 years ago

Hello,

Do you have any update about the IPv6?

ghost commented 4 years ago

Any updates here? Roadmap?

I recently came across this as I wanted to create my first EKS cluster and - of course - was setting up dual-stack VPC/subnets as a basis. I was literally shocked to see the actual non-existence of IPv6 support of such a major infrastructure product in the cloud era. I took half a day trying to find a solution because my brain was resisting to believe this could be true.

ffrediani commented 4 years ago

It's unbelievable how people can still treat IPv6 support to new products now a days so badly as if it was something "optional, for later, less important or cosmetic". Meanwhile a lot of new platforms, among them some quiet large ones who generate a fair amount of traffic remain without IPv6 support because of this issue.

rkenney525 commented 4 years ago

If it helps get this stuff out the door faster, I'm more than happy to dedicate some after work time to review, pair, etc -- just point me in the right direction. I saw this PR a few days ago but havent seen much traction, so idk if its part of the main IPv6 effort or not. IPv6-only pods would be the biggest win for me but I'll help wherever I can

mikestef9 commented 4 years ago

Hey all,

IPv6 is a major priority for the EKS team, but it's also a major project that requires changes in nearly every component of EKS, along with hard dependencies on a few other AWS service feature launches.

We have come a long way in our design since originally opening this issue, and it will look similar to @zadjee comment from above.

At cluster creation time, you will have the option to choose IPv4 or IPv6 as the pod IP address family. If you choose IPv6, pods will only get IPv6 addresses. When new nodes are launched in the cluster, they will each be assigned a /80 IPv6 CIDR block to be used by pods. However, IPv4 connections will still be possible at the boundaries of the cluster. With dual stack ALB (available now) and NLB (coming soon), you can accept IPv4 traffic that will be routed to pods. For egress, we will use a NAT64 solution that will allow pods to communicate with IPv4 services outside the cluster.

This design solves all pod density and IP exhaustion challenges faced today, without requiring all IPv4 services in your environment to first be migrated to IPv6.

Again, this design requires some features to be first launched from other AWS service teams, so no timeline to share right now, but it is a project we are actively working on.

-Mike

TBBle commented 4 years ago

If I understand the last comment correctly, the nodes themselves, i.e. for NodePort Services, will be unaffected by this, as they will depend on the Node's IPv4/IPv6/Dual-Stack setup and can forward to the Service's ClusterIP in whatever the chosen Pod IP address family is? I'm assuming ClusterIP will match PodIP address family.

I ask mostly because our use-case has lots of UDP NodePort Services (Agones-managed), but partly because it seems that if I'm right, NLB instance-routing (i.e. without #981) talks to NodePort Services and so should be isolated from this change.

ffrediani commented 4 years ago

I wonder if this had been considered at the beginning of the project what would be the output. It's hard to take initiate a new project in the recent years and not have IPv6 as a mandatory feature since day zero. And it get even worst when it depends on other and other projects.

When asked people who develop platforms that use EKS why they don't have IPv6 support they response is because of EKS, then it seems there are yet other dependencies in a chain with prevents any timeline or commitment.

mikestef9 commented 4 years ago

Yes, Service ClusterIP will be IPv6 if pods are IPv6. Nodes will be dual stack and handle the translation if needed.

rkujawa commented 3 years ago

@mikestef9 I deeply appreciate the fact that this issue is now in "We're Working On It" state. Is there any update on timeline / projected availability of this feature?

My team is using EKS for a few development and test services now. As we are nearing production release for these services, it becomes more and more obvious that lack of native IPv6 in EKS will force us to create ugly workarounds. It would help greatly with our own planning, if some timeline of IPv6 availability in EKS was known. Thanks.

ffrediani commented 3 years ago

It is unbelievable the lack of commitment form developers in general with IPv6 these days. They don't feel guilty for no worrying about it. Probably they think "Ohh it's just a minor detail. The software works fine and that's what is important.". Then they keep refusing the fact they are one of the main responsible actor for delaying IPv6 deployment worldwide.

TBBle commented 3 years ago

It might be useful to have a sense of the other systems/products that need to evolve to support this.

Off hand, I assume https://github.com/aws/amazon-vpc-cni-k8s will need feature work (built on #398 I assume from older discussions there), and NLB will need to support both IPv6 backends and the https://github.com/kubernetes-sigs/aws-load-balancer-controller will need to be updated to support NLB-IP setup for IPv6 target IPs and NLB dual-stack listeners.

It'd be good particularly to make clear if there is any known work needed on non-AWS projects, e.g., the Kubernetes in-tree cloud provider. I see that for 1.21, that's now rehomed into https://github.com/kubernetes/cloud-provider-aws, so that's not a good example of a non-AWS project anymore. ^_^

On that topic, will the AWS Cloud Provider v2 (https://github.com/kubernetes/cloud-provider-aws/issues/125) be required? I haven't looked closely at the cloud providers for a little while; I'm hoping that the IPv6-complexity-related pieces of work are going to be in the AWS Load Balancer Controller v2 and the AWS VPC CNI driver, but if there are AWS Cloud Provider interactions needed, they're either trivial and can be applied to v1, or will I assume need to go with v2.

jbottum commented 3 years ago

@mikestef9 any update on ipv6 for eks that you can share? thanks in advance.

gauravmittal80 commented 3 years ago

Hi,

I understand currently the support for IPv6 is not available for EKS, but is there a workaround that can be used to get the dual stack support (both IPv4 and IPv6).

Any pointers will be highly appreciated.

Regards, Gaurav

zajdee commented 3 years ago

To everyone watching this thread: one of the features mentioned by Mike @mikestef9 in https://github.com/aws/containers-roadmap/issues/835#issuecomment-705330015 was the /80 assigned to a node. This is now working with the t3.* instances. (You can assign an IPv4 /28 to a node too, but that's a pretty small subnet, compared to the IPv6 /80.)

The /80 is fully routed to the EC2 instance, so you may want to start playing around with dual-stacked or IPv6-only containers with no need for IPv6 NDP. Some links:

Together with the Dual-Stack NLB (available since the end of last year, but the target groups still don't support IPv6 addresses) the routed prefixes clearly pave the way towards the IPv6-only containers (pods). The NAT64 gateway is yet to be announced, as it's required too (although you can run Jool in an EC2 instance to translate from IPv6 to IPv4, it's not what one would prefer).

ffrediani commented 3 years ago

Hello @zajdee Thanks for the update.

If I understand it correctly from the provided URLs de IP addresses that can be assigned to instances are all private - including the IPv6 ones. When you say that the /80 is fully router does it mean only inside customer environment or globally routed ?

TBBle commented 3 years ago

From my reading of those links, it's only the IPv4 addresses which are private, the IPv6 addresses are publicly routable.

Assuming you're seeing this note:

Prefixes for network interfaces are limited to private IPv4 and IPv6 addresses.

I think that sentence is missing a comma, and it should be:

Prefixes for network interfaces are limited to private IPv4, and IPv6 addresses.

or reordered to remove ambiguity as:

Prefixes for network interfaces are limited to IPv6 and private IPv4 addresses.

The examples use 2600:1f13:fc2:a700:1768::/80 which is a publicly-routable IPv6 subnet owned by Amazon.com, Inc, and the term private only appears in the APIs that deal with IPv4, e.g., assign-private-ip-addresses, not the APIs that deal with IPv6, e.g., assign-ipv6-addresses

"Private IPv6 addresses" would be fc00::/7 or fec0::/10. The former have a specific use-case that doesn't apply here, and latter (which were the equivalent of the private IPv4 address ranges) were deprecated in 2004 and almost 17 years later, should no longer concern anyone in any way.

zajdee commented 3 years ago

I've actually performed the IPv6 routing test yesterday, before writing the comment above, and these indeed are globally routable IPv6 addresses. A screenshot of the test is available on https://twitter.com/zajdee/status/1420101326554402825

olemarkus commented 3 years ago

There has been a number of fixes for the opensource components, and the 1.22 release of https://github.com/kubernetes/cloud-provider-aws will have the necessary fixes for running IPv6.

kOps already supports IPv6 with non-global ipv6 addresses.

There is also a s a PR for using the new ipv6 prefix functionality natively with any CNI that can use the kubernetes IPAM (which most can). This will give all pods global ipv6 addresses. (I do have a screenshot of this too: https://twitter.com/olemarkus/status/1424426211393122311)

So at least EKS-D should be able to use this functionality.

autarchprinceps commented 2 years ago

AWS released support for IPv6 only VPCs, subnets and NLB/ALBs. This would be very interesting to see in EKS as well. IPv4 only clients, if they still exist, can still be served by CloudFront Edge Locations. It would be a big improvement for us to no longer have to assign IPv4 to VPCs, since for TGW purposes they then need to be unique again. Especially, since this will likely take some time to propagate into support in various elements, having it be possible in core components should be planned early on.

hakman commented 2 years ago

@autarchprinceps I think there is still a need for a NAT64 appliance and/or maybe some STS dual-stack endpoints. Pretty annoying to not have IRSA working on IPv6.

zajdee commented 2 years ago

AWS has also deployed the DNS64 support on the VPC DNS infrastructure, although it wasn't communicated publicly (yet).

marcuz commented 2 years ago

@autarchprinceps I think there is still a need for a NAT64 appliance and/or maybe some STS dual-stack endpoints. Pretty annoying to not have IRSA working on IPv6.

I agree 100%. IPv6-only is a step further, but the reality is most of us need dual-stack (particularly for egress).

hakman commented 2 years ago

@zajdee I am aware of the DNS64 support, though that can be done from CoreDNS too. You still need the NAT64 part to be of any use. I could add that IPv6 rollout in AWS is a little slow in general. Even the the recent EC2 dual-stack API is only available in very few regions. Hopefully things will speed-up a little.

tvi commented 2 years ago

NAT64 was announced in a few regions: https://aws.amazon.com/about-aws/whats-new/2021/11/aws-nat64-dns64-communication-ipv6-ipv4-services/

mikestef9 commented 2 years ago

EKS support for IPv6 is now available!

A point I want to emphasize is that to solve the IPv4 exhaustion problem, we have gone with an approach of single-stack IPv6 K8s, but with dual-stack pods. We give pods an 'egress-only' node-internal private IPv4 address that gets SNATed to the node's primary VPC IPv4 address on the way out if necessary. Kubernetes doesn't know anything about this non-unique v4 address.

Note: EKS IPv6 support is not available in Beijing or Ningxia regions, as VPC does not support prefix assignment in those regions.

Resources associated with launch:

sftim commented 2 years ago

I'm guessing that any workload that's aware of RFC6052 NAT64 can also send traffic to the IPv4 internet provided the VPC is set up for that.

sheetaljoshi commented 2 years ago

Yes, IPv4 endpoints are supported. EKS takes a different approach of supporting IPv4 endpoints via egress-only IPv4 solution for pods https://github.com/aws/amazon-vpc-cni-k8s/blob/master/misc/10-aws.conflist#L14. Details to be published soon.

anguslees commented 2 years ago

(and yes, any workload that's aware of RFC6052 NAT64, can just send those IPv6 packets out to their nat64 gateway directly - it will be treated the same as any other v6 traffic to/from the pod.)

GouravSingh2580 commented 8 months ago

I've set up a dual-stack VPC with IPv6 subnets and enabled DNS64. However, I'm facing connectivity issues with the Stripe API, which I believe is still IPv4.

image

Given that DNS64 is enabled with 64:ff9b::/96 prefix in the subnet, it should ideally allow access to any IPv4 service, correct?

Can someone assist me in identifying what might be missing here?

Thank you!