search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.17k stars 313 forks source link

[ECS] : ECS Support for IPv6 #1340

Open vibhav-ag opened 3 years ago

vibhav-ag commented 3 years ago

Hi all,

Looking to get input on your IPv6 requirements as we develop our strategy here. IPv6 for ECS is a broad area and your feedback will help us prioritize our feature delivery to support the most important use cases.

Some topics that would be especially useful to get clarity on:

Anything else that is important to you!

Community Note

Which service(s) is this request for? ECS, Fargate

jfrlb commented 3 years ago
  1. Bridge mode would be extremely interesting, as we are trying to run many application instances on "own" (and separate) IPv6 addresses within the same EC2 instance. We have rather small containers that need something like 0.25 vCPU and 200MB RAM, so we would like to run several of those on small t3-instances, but that does not work with awsvpc mode due to the ENI limits (and not trunking on t-instances). Currently we are working around it with docker host mode, but I'd rather say that's really just a work around with associated disadvantages (e.g. container has to run as root).
  2. We are absolutely lacking the ability to "fix" the IPv6 addresses used by the containers. Our (client) application needs to contact our backend behind static IPv6 addresses but currently ECS only allows to specify the /64 prefix; but we'd rather need something like an "elastic IP" for our backend service. It would be a great starting point already, if a "secondary" IPv6 could be configured to the EC2's ENI. Through docker host mode we could use that as well in our containers. It is already possible to configure this on the linux machine, but it looks like the additional address is not considered in the VPC routing and the VPC thus does not forward any packets to such addresses today.
  3. Network load balancer does not support UDP + IPv6 combination!! And no IPv6 for internal load balancers at all.
  4. We would need IPv6 multicast support within VPC networking (we are receiving traffic into the VPC through a VPN endpoint and would like to forward received multicast packets to a number of ECS containers who are listening for these)
jfrlb commented 3 years ago
  1. Especially for development and test stages it would be beneficial if AWS/VPC would also support ULA addressing (RFC4193) for VPC traffic. This would allow to create setups with specified, static and independent IPv6 addresses that you can rely on.
zoltanmajo commented 2 years ago

For our setup currently running in production we'd need IPv6 support for tasks launched by ECS services. That way we could access services outside of the VPC using IPv6.

So from the points mentioned in the description the following matches our requirements the best: "Are you planning to use IPv6 only within your VPC(s), or are you also planning to connect your Tasks to IPv6 internet?"

The answer is yes. Many thanks in advance.

leezu commented 2 years ago

Do you need IPv6-only (IPv4 disallowed) access?

Definitely. That's because AWS has bad support for IPv4 addresses on multi-network interface instance types such as p4d. Multi-NIC instances do not support association of public IPv4 addresses at startup (they only support association of private IPv4 addresses and IPv6 addresses). Thus multi-NIC EC2 instances currently can't join the ECS clusters until an EIP is manually assigned or a NAT Gateway is configured in the subnet. NAT Gateway adds extra costs to customers and automatic EIP assignment is hard to configure for customers (requires implementation of Lambda functions based on SNS events of p4 instance startup).

Once ECS supports IPv6-only (IPv4 disallowed) access, ECS will work out-of-the box for multi-NIC instances.

mogadanez commented 2 years ago

ECS already supports dual stack (IPv4+IPv6) for Host and AWSVPC Networking modes.

but there is a lack of documentation on how to properly connect it to ALB.

ECS works with instance type of target group, but in loadbalancer setting I cant create instance type with ipv6 selection

tlaverdure commented 2 years ago

Do you require image pulls from ECR over IPv6?

Yes, this would great for IPv6 egress only workloads.

martin-pronestor commented 1 year ago

I was experimenting setting up a VPC using dualstack and having two types of subnets (two public and one private) for a simple ECS hosted web application with an elastic load balancer in front. The images for the web application were hosted on ECR and I wanted to see if I could avoid having a NAT gateway (and elastic IPv4) associated to the private subnet and only use IPv6 and an egress-only gateway.

I was a little disappointed to find out that images couldn't be pulled from ECR to a running ECS with IPv6. It didn't even work if I used a different image repository.

Do you require image pulls from ECR over IPv6?

Yes, please. Hopefully this could fix our issue and we could save the cost associated with using a NAT gateway and Elastic IP.

CGarces commented 1 year ago

It didn't even work if I used a different image repository.

I don't get it. If you use and external repo like Docker Hub, egress-only gateway with IP6 not works?

we could save the cost associated with using a NAT gateway and Elastic IP.

Why you need and Elastic IP?. Based on your requirements maybe VPC endpoint is cheaper than NAT gateway

martin-pronestor commented 1 year ago

I don't get it. If you use and external repo like Docker Hub, egress-only gateway with IP6 not works?

From the error message that I received, when the ECS task was provisioning, it said it got a TCP timeout using the IPv4 address of the repository, so I assume the agent responsible for pulling images doesn't know how to use the IPv6 version of the repository endpoint. In my tests I tried pulling from ECR and from GitHub private container registry (https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry).

Why you need and Elastic IP?. Based on your requirements maybe VPC endpoint is cheaper than NAT gateway

From what I've been told you have to have an elastic IP in order to use a managed NAT gateway. (https://stackoverflow.com/questions/43094786/why-does-a-aws-nat-gateway-require-an-elasticip) I tried setting up VPC endpoint but ultimately found that it ends up being almost as expensive as the NAT gateway because you need to add S3, ECR-DKR, ECR-API, and possibly CloudWatch (https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html). It is also way harder to set up and debug :(

gregwebs commented 1 year ago

Can we create separate issues to support an IPv6 egress only gateway? A lot of the comments here are in support of this and I think there are too many other things being asked about on this issue.

Personally I would really like IPv6 egress only because it is a more secure architecture to be able to block ingress at the gateway level.

relgames commented 1 year ago

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78
julienp-b2wise commented 1 year ago

Hi @vibhav-ag, as of today it's possible to launch fargate tasks in a dual stack subnet (ipv4 + ipv6), are there any plans to support ipv6-only subnets in the future?

mbtran1 commented 1 year ago

Hi @vibhav-ag, as of today it's possible to launch fargate tasks in a dual stack subnet (ipv4 + ipv6), are there any plans to support ipv6-only subnets in the future?

hi @julienp-b2wise and team, good timing of your inquiry. We (ECS) have plans for expanded IPv6 support, where ECS will deliver IPv6 support for its tasks and services on all compute platforms in dual stack (IPv4+IPv6) and IPv6-only subnets, with load balancer integration, image pulls, Domain Name System (DNS), and Task Metadata Service (TDMS) over IPv6.

Couple questions -- Please share your requirements in terms of ECR public IPv6 endpoint support and ECR PrivateLink IPv6 endpoint support for production deployments. Is there a hard requirement for ECR public IPv6 endpoint support for production deployments in 2023? Thanks.

ngoyal16 commented 1 year ago

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

We are also looking solution for same. @relgames have you found any alternative.

julienp-b2wise commented 1 year ago

Hi @mbtran1, thx for the feedback. As of our requirements we would just need to have ECS/Fargate tasks ipv6-only with the ability to pull from ECR of course, but we wouldn't need ecr ipv6 public endpoint support nor privatelink support at least for now.

grahamschuckman commented 12 months ago

Adding in the federal perspective here. Our agency client is responding to the White House OMB Mandate requiring that federal agencies begin operating in IPv6-only modes. See Page 3, Section 4a: https://www.whitehouse.gov/wp-content/uploads/2020/11/M-21-07.pdf

As such, we're looking to migrate our ECS workloads into an IPv6-only mode of operation.

CGarces commented 11 months ago

If we can't connect to ECR with IP6, there will be some impact con bills in 2024. Run big Fargate clusters (each one with a public IPv4) will be more expensive. https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/

IllarionovDimitri commented 11 months ago

Ran today onto the same issue. "IPv6 only" configuration is not yet supported by AWS, so the public IPv4 must be assigned to the tasks running in public subnets regardless of IPv6. If there is no solution until Feb. 2024, even attaching IPv6 will not prevent the costs increase for public IPv4

tophercullen commented 11 months ago

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

We are also looking solution for same. @relgames have you found any alternative.

@ngoyal16 same issue here with running Fargate tasks. It had the added benefit of causing all sorts of internal breakage with AWS. Tasks and services stuck in various defunct states for hours despite deleting.

ngoyal16 commented 10 months ago

@tophercullen what we have did is we have added a event bridge listener for task creation and invoke lammda and adding the ipv6 as target to nlb... That what we are doing as of now..but if AWS team can this own our behalf would be much better solution

majkel-hyggio commented 9 months ago

Guys, isn't that the anwser https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html ? If I get it right, that switch should enable EC2 ipv6 only instances to connect to ECR.

tophercullen commented 9 months ago

@majkel-hyggio You will incur additional costs in both NAT gateway hours and data transfer. NAT data transfer costs are not cheap, and if you don't setup a NAT per subnet (multiply NAT gateway hour charges), you'll also incur regional data transfer costs. Depending on the various elements (task durations, counts, image sizes etc.) it may actually be cheaper to just to incur the costs of public ipv4 addresses.

In either case, you're simply paying more for AWS ECR just not having an IPv6 endpoint available.

jnawk commented 7 months ago

Given there's a charge for public IPv4 addresses coming in February, it's pretty important that there's a status-quo (ie free) alternative available, lest the change be viewed as purely an opportunistic money grab under the guise of IP address exhaustion.

leonardlin commented 7 months ago

I was caught off-guard by this too.

In preparation to ipv6 migration push (charge on public ipv4), I tried to move as much as possible to ipv6.

Turns out ec2 instances without public ipv4 can't join ecs-clusters.

ECS do not support IPv6 EC2 instances, even if they have a private IPv4

See also here: https://stackoverflow.com/questions/38599701/register-ec2-instance-to-ecs-cluster-without-public-ip

ajpauwels commented 6 months ago

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

@relgames Were you able to resolve this? Hitting it right now, and I can't tell if it was just never implemented, or there's a switch I need to flip.

relgames commented 6 months ago

No, seems like it's still not supported.

timzuiddam commented 4 months ago

And now it's February and we are hit with this, where's the customer-centric approach here? Still no update?

gregwebs commented 4 months ago

There are many AWS services lacking in IPV6 support: https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html AWS is making an estimated billion dollars a year (revenue, but there's certainly some profit there) charging for IPV4 addresses. That money could be used to fund a roadmap with commitments for full IPV6 support for various services.

maxgruebneraeroqual commented 4 months ago

We're also being reamed by AWS switching to charge for IPv4 addresses, but not providing IPv6 functionality required to move away from IPv4.

Pretty transparent cash grab from AWS, considering they'll make back the cost of the IPv4 addresses in under two years.

It's lovely running into situations like:

I need public addresses for ECS containers, because AWS Secrets Manager requires public IP addresses (for some reason?). I can't make my ECS containers IPv6 only (for some reason?), so all my containers now cost me an extra $50 a year, or $150 if I have a load balancer in front of them (which are also only dualstack and must have at least two IPv4 addresses themselves).

Allegedly this can be worked around by using PrivateLink, so we get to waste a bunch of time trying to get that working (and hoping that there's no extortionate charges involved in that too).

jnawk commented 4 months ago

I need public addresses for ECS containers, because AWS Secrets Manager requires public IP addresses (for some reason?). I can't make my ECS containers IPv6 only (for some reason?), so all my containers now cost me an extra $50 a year, or $150 if I have a load balancer in front of them (which are also only dualstack and must have at least two IPv4 addresses themselves).

Allegedly this can be worked around by using PrivateLink, so we get to waste a bunch of time trying to get that working (and hoping that there's no extortionate charges involved in that too).

While this is a pretty shit move by AWS, it does sound like you can save a bit of money. If you've got a load balancer, then you don't need your containers to have public IP addresses - chuck them in private subnets. You'll need a NAT gateway or NAT instance to provide them with outbound internet access to access the secrets manager endpoint, OR, you can create an interface endpoint for secrets manager in a private subnet, which will give your containers access to secrets manager without needing outbound internet access - there's a per hour charge associated, naturally - both with any NAT solution you use, and for the interface endpoint, but run the numbers, it might come to less than what you're otherwise paying.

Of note, by running your containers in private subnets, you'll need to make sure they can access your container registry - if that's ECR, then you'll need an ECR interface endpoint, and a S3 gateway endpoint. Same story with any other things your containers access.

Yeah, it might cost more to sort it out than you'd otherwise pay to AWS for IP addresses, so you might be better just paying that racket.

Spacefish commented 4 months ago

A first step would be to add Ipv6 support to ECR / dkr.ecr.. If i look at my ECR Registry, it´s CNAME pointing to a public network load balancer: nlb3-21115c3760ec6ec9.elb.ap-east-1.amazonaws.com but this hostname only has an A record but no AAAA record.

Same thing for the layer download endpoint as well, prod-ap-east-1-starport-layer-bucket.s3.ap-east-1.amazonaws.com is a CNAME to s3-r-w.ap-east-1.amazonaws.com which has no AAAA record.

klevo commented 4 months ago

C'mon AWS this is embarrassing. Let's get that ECR IPv6 support in at least.

maxgruebneraeroqual commented 4 months ago

Yeah, it might cost more to sort it out than you'd otherwise pay to AWS for IP addresses, so you might be better just paying that racket.

With such transparent pricing claims as:

"It is important to note that although the NAT gateway does not charge for data transfer in, there will still be a per-GB data processing charge on data that flows through the NAT gateway, regardless of direction. The data transfer out charges are in addition to this charge. NAT gateway pricing can be found on the Amazon VPC pricing page."

What could possibly go wrong?

But yes, it is more expensive to add all these workarounds and use 20 additional AWS services to compensate, so we just have to take it.

mocknen commented 4 months ago

I also tried, but still failed to specify ECS configured with IPv6 and Fargate as the target group for ALB, with the following error message.

service failed to register targets in target-group with (error The IP address '' is not a valid IPv6 address)

I am certain that the ENI configured for the task has an valid IPv6 address. I have also tried configuring my account for dualStackIPv6 as described in the documentation, but nothing has changed.

TeoTN commented 4 months ago

The whole issue could have been easily solved by AWS for low-cost tinkerers by offering up to 2-3 public ipv4 addresses in Free Tier, assignable to ALB, just like they offer one for an EC2 instance.

Given that barely anything at AWS at the moment supports ipv6-only mode, it's a ludicrous move. It's no customer centricity just purely squeezing as much money from people as possible, for no value added.

gauravgoyal591 commented 4 months ago

Due to the AWS team charging for IPv4 and not being ready for IPv6 connectivity, please refrain from charging for IPv4 until your architecture is ready.

TJNII commented 4 months ago

Is this expected?

$ dig AAAA ecs.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.5 <<>> AAAA ecs.us-east-2.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4379
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ecs.us-east-2.amazonaws.com.   IN      AAAA

;; AUTHORITY SECTION:
ecs.us-east-2.amazonaws.com. 59 IN      SOA     ns-1999.awsdns-57.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

;; Query time: 0 msec
;; SERVER: 10.192.0.2#53(10.192.0.2)
;; WHEN: Thu Feb 29 02:11:06 UTC 2024
;; MSG SIZE  rcvd: 170

IPv6 in my VPC is working, I was able to install dig into the ec2 instance running amazon/amzn2-ami-ecs-hvm-2.0.20240109-x86_64-ebs over IPv6. When I search for "ECS IPv6" the top hit is a press release proudly proclaiming "Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode" so I'd expect this to work.

gauravgoyal591 commented 4 months ago

No it's not working from the IPV6 servers

Connect timeout on endpoint URL: "https://api.ecr.us-east-1.amazonaws.com/" [image: birdeye-icon] https://birdeye.com/?utm_source=product_email&utm_medium=customers&utm_campaign=default_signature&utm_term=birdeye_icon

Gaurav Kumar Goyal

Software DevOps Engineer (797) 620-2418 birdeye.com https://birdeye.com/?utm_source=product_email&utm_medium=customers&utm_campaign=default_signature&utm_term=birdeye_dot_com

On Thu, Feb 29, 2024 at 7:48 AM Tom Noonan II @.***> wrote:

Is this expected?

$ dig AAAA ecs.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.5 <<>> AAAA ecs.us-east-2.amazonaws.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4379 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;ecs.us-east-2.amazonaws.com. IN AAAA

;; AUTHORITY SECTION:ecs.us-east-2.amazonaws.com. 59 IN SOA ns-1999.awsdns-57.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

;; Query time: 0 msec ;; SERVER: 10.192.0.2#53(10.192.0.2) ;; WHEN: Thu Feb 29 02:11:06 UTC 2024 ;; MSG SIZE rcvd: 170

IPv6 in my VPC is working, I was able to install dig into the ec2 instance running amazon/amzn2-ami-ecs-hvm-2.0.20240109-x86_64-ebs over IPv6. When I search for "ECS IPv6" the top hit is a press release proudly proclaiming "Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode" so I'd expect this to work.

— Reply to this email directly, view it on GitHub https://github.com/aws/containers-roadmap/issues/1340#issuecomment-1970276203, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATNDOEMIVVHJM24WW3R373DYV7QQVAVCNFSM424AH2SKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJXGAZDONRSGAZQ . You are receiving this because you commented.Message ID: @.***>

CGarces-Apser commented 4 months ago

Is this expected?

$ dig AAAA ecs.us-east-2.amazonaws.com

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.amzn2.13.5 <<>> AAAA ecs.us-east-2.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4379
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ecs.us-east-2.amazonaws.com.   IN      AAAA

;; AUTHORITY SECTION:
ecs.us-east-2.amazonaws.com. 59 IN      SOA     ns-1999.awsdns-57.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 60

;; Query time: 0 msec
;; SERVER: 10.192.0.2#53(10.192.0.2)
;; WHEN: Thu Feb 29 02:11:06 UTC 2024
;; MSG SIZE  rcvd: 170

IPv6 in my VPC is working, I was able to install dig into the ec2 instance running amazon/amzn2-ami-ecs-hvm-2.0.20240109-x86_64-ebs over IPv6. When I search for "ECS IPv6" the top hit is a press release proudly proclaiming "Amazon ECS now supports Internet Protocol Version 6 (IPv6) in awsvpc networking mode" so I'd expect this to work.

ECS partially support IPv6 but ECR not, ECR is the big issue https://docs.aws.amazon.com/vpc/latest/userguide/aws-ipv6-support.html

pawelfraczyk commented 3 months ago

Hi, it would be nice to support IPv6 addresses to register ECS tasks in ALB target group (awsvpc mode). Right now, when I tried to enable IPv6 in the target group, ECS is trying to use ipv4 to register in the ipv6 target group:

service test-EcsService-fYJDJS3mLw5J failed to register targets in target-group test-tg  with (error The IP address '172.17.2.196' is not a valid IPv6 address)

The task itself has IPv6:

Private IP
172.17.2.196

IPv6 address
2a05:d018:1cee:b902:bd1e:854e:44a4:1f78

End of March getting the same error.

VPC enabled ipv6, all subnets with routing tables. ALB in dual-stack mode ECS enabled in region for whole account to use dual-stack mode Target group with ipv6

ECS Fargate Task has both ipv4 & ipv6 and can't register as a target in TG because it tries ipv4....

Any solutions?

Laurenfrost commented 3 months ago

Hello! Any udpates here?

I got exactly the same error when I tried to register a ecs fargate task to an ipv6 target group.

service `test-service` failed to register targets in target-group `test-tg-tbt-v6` with (error The IP address '192.168.162.68' is not a valid IPv6 address)
ajpauwels commented 3 months ago

@pawelfraczyk @Laurenfrost

There is unfortunately no solution here, it requires AWS changing things on the back-end to detect if a target group is IPv6-only and then registering IPv6 targets rather than IPv4 targets.

victorvhpg commented 2 months ago

Any update ? @tabern