[ECS,Fargate] Multiple target groups for a service

[zaitsman]

Summary

I want to be able to route traffic from two different Path rules in my ALB to the same ECS instance and have ECS automatically register targets in both groups.

Description

Pretty much what the summary says. I can manually add targets to the second group, but if the service is restarted, then the second group loses the targets

Expected Behavior

Unable to have 2 targets for the same ECS service

Observed Behavior

N/A

Environment Details

Supporting Log Snippets

[dastbe]

hey @zaitsman, ECS service team member here.

We don't currently support multiple target groups per service today. However, if you're able to use the same port on the same ALB you can re-use the same target group for multiple rules on a single loadbalancer.

[alexzaytsev-newsroomly]

@dastbe I do use that, actually, but my use case is that i want the same container to respond to two different TG's for different ALB's. I understand that is a feature request. One way to solve it would be if we could get a Cloudwatch alarm each time ECS registers something, we could then lambda those additional registrations.

[adnxn]

One way to solve it would be if we could get a Cloudwatch alarm each time ECS registers something

@alexzaytsev-newsroomly, have you looked at the ECS event stream? There are container instance registration/deregistration that are surfaced through CloudWatch events. Let us know if this captures your specific use case.

[hridyeshpant]

@dastbe does we have Multiple target groups for ECS service support in Fargate mode?

[bataras]

Is multiple target groups per ECS service feature going to happen? On the roadmap?

[DenisBY]

I need this feature too. Our application inside a container listens on 3 different ports for different purposes but I'm able to (automatically) register container only in one target group

[Raizex]

We could also really use this as well. We've reached the maximum number of certificates per load balancer and need a way to add the service to additional load balancers without creating a duplicate service.

[sergedomk]

I could use this as well. We would like to put up a single Fargate ECS service with several containers each providing a different branch of the code running on a different port for our product and QA teams to use.

[mcmar]

This is a blocking issue for me. I have one primary service and one secondary service. The primary receives traffic for /management route. The primary and secondary both receive traffic for all other routes *. This pattern is impossible to implement without the ability to register 2 target groups for the primary instance.

This is the pattern currently used by Prisma as described here: https://github.com/prismagraphql/prisma/issues/2850#issuecomment-409868271

[zymtx5g79k]

I solved an issue by pointing current instances in stack, but it will not works with auto scaling. The feature will be useful.

[mcmar]

@zymtx5g79k Can you clarify what you mean by pointing current instances in stack? I'd love to get a solution to this and I'm not currently using autoscaling. Does your solution work with ECS on Fargate?

[zymtx5g79k]

@mcmar, hello:

EC2 -> Target groups -> Create target group -> Target type (instance) -> Create.
Select just created target group -> Actions -> Register and deregister targets -> Check all Instances that work in your stack -> Add to registered -> Save.

After some time the target group will detect which an instance has your service and will mark it as healthy. Load balancer will redirect traffic on the healthy instance.

In my case: it slanger service. Port 4567 was assigned in ECS service and port 8080 (websocket) is listen from all instances.

Tip: Target group need some time to correct healthy status after ECS service migration across EC2 instances. You can get connections refused until new healthy instances was found.

In my case: websocket connections will be repeated. It is regular case for frontend. I use EC2 stack "because I can"! (:

[theladyjaye]

I could use this too.

In my use case, we have one legacy application running behind an Application load balancer but rather than rebuild the whole application from scratch, we can go down the route of rebuilding pieces and just forwarding from the first Application Load Balancer to a new target group (listener rules). Unfortunately when using Fargate/ECS it owns the registrations for the Target Group and a target group can only be associated with 1 Load Balancer (which Fargate/ECS owns). So we are left with trying to figure out a way to mirror that Target Group when we get new ECS Registrations.

[m0n5t3r]

another use case would be something that exposes multiple ports (like, say, rabbitmq's 5672 (AMQP) + 1567 (management UI)); since network load balancers seem to force traffic to the target group port instead of keeping the listener port, one needs to have a target group for each port, but only one of them ever gets updated by ECS

[hajdukd]

This is a blocker - we require to have an option to register more than single port for our services.

[theladyjaye]

@hajdukd I hate to see you blocked by this. Per one of the comments above:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html

Using ECS Event Streams via CloudWatch events and triggering a Lambda definitely is a solution that works. It’s not as convenient if they offered it automatically, but you can add and remove from as many target groups as you want. So for a small bit of extra work (1 Lambda function and some event wiring) you will be unblocked.

[renuez]

Nathan Peck from ECS product team has recommended a few very useful options: https://gist.github.com/nathanpeck/a8537e21627b3a0f3e735eecbe6f6384

[coultn]

Thanks everyone for the feedback on this issue. I wanted to let you know that we on the ECS team are aware of it, and are actively considering solutions. +1's and additional details on use cases are always appreciated and will help inform our work moving forward.

[cnelson]

+1

[davidfti]

+1

[outofcoffee]

@coultn we are trying to model multiple ingress routes to a shared service. In particular, some traffic comes into a load balancer in a private subnet with one set of path routing rules, whilst other traffic comes via a load balancer in a public subnet with a different set of path rules.

Using separate security groups to logically represent the rulesets for the different ingress routes can be somewhat helpful here, but with the default max of 5 SGs per load balancer this can be a squeeze. It also doesn't provide the routing benefits of separate load balancers in public/private subnets.

Isolation at the task level is not required, and it would be inefficient to run multiple service instances, each with multiple task instances for redundancy, simply for the purposes of attaching multiple load balancers.

Essentially the pattern we're looking to emulate is how Pods can have multiple Services pointing to them in Kubernetes. This is one of the main drivers pushing us away from ECS.

[dodwmd]

+1

[senthilkumar-s]

+1

[sampsasaarela]

+1

[broglep-work]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb (it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

[tkgregory]

+1 My use case is a Gitlab task that needs to expose ports 443 and 22. I'm currently using an ALB for the HTTPS, and would need another load balancer and target group for the SSH.

[sandeepboyapati]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb (it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

This works perfectly.....Thanks @broglep-koubachi

[kokuryuha]

+1 as a feature request as well. Currently looking at deployments and realized this was going to be an issue immediately.

[joemoore13]

+1 the case of a single application listening on multiple ports (web and tcp) has multiple cases for many technologies and this makes working with them in ECS more difficult.

[abby-fuller]

moving this over to the containers roadmap since this is an ECS feature request, rather than an agent issue.

[ghost]

+1

Is there an estimate when this might be available?

[polarizeme]

Just wanted to chime in with a +1 on this request. HItting this as a roadblock at the moment, which is a huge bummer/frustration.

Usecase is the same as @outofcoffee :

we are trying to model multiple ingress routes to a shared service. In particular, some traffic comes into a load balancer in a private subnet with one set of path routing rules, whilst other traffic comes via a load balancer in a public subnet with a different set of path rules.

[v-g-ustinov]

+1

[jbouse]

I have a use case where I have an ECS TaskDefinition that is launching 2 containers. The first container has 2 ports exposed, one is TCP passthrough and the other can either be HTTPS (ELB) or TLS (NLB) but they point to the same container. The second container exposes a different port that is TCP passthrough as well. The second container is new and only compounds the issue the first one exposed. With just the first container I used a Classic ELB with the two listeners and it works but I have to use static HostPort values for the container PortMappings. This causes an issue when updating the service as the ECS instances can only have 1 container listening on the static host port. If I could use a TargetGroup and NLB then I could use dynamic HostPort mapping and not run into this issue but I can't do this if I can only add 1 entry under LoadBalancers for the ECS Service. It would be extremely helpful if I could provide the ECS Service configuration something along the lines of:

      LoadBalancers:
        - TargetGroupArn: !Ref Service1TG
          ContainerPort: !Ref Container1Port1
          ContainerName: !Ref Container1Name
        - TargetGroupArn: !Ref Service2TG
          ContainerPort: !Ref Container1Port2
          ContainerName: !Ref Container1Name
        - TargetGroupArn: !Ref Service3TG
          ContainerPort: !Ref Container2Port1
          ContainerName: !Ref Container2Name

[jasonnerothin]

Just trying to flip on SSL/TLS: +1

[sanglt]

We did a PoC with Cloud Watch Events & Lambda function. You can find the CF stack at https://gist.github.com/broglep-koubachi/0ceed52c509f75a6c26afc43b807a0eb (it uses an ecs service and its load balancer config as basis to create targets for the twin target group)

Thanks, It's work perfect for us.

[bbonnefon]

+1 We have micro services, some public and others private, but they should talk together. It would be very helpfull that public micro services are available on 2 different ALB (public access and private access).

[rtjarvis]

I've encounted this problem today too. Come on AWS - the CFT even supports a list!

[sstelfox]

I'd also like to see this, though for the NLB not the ALB. Our use case is that we are running conditional client certificate authentication on our system, so TLS listeners won't work. This requires handling both HTTP and HTTPS at our service which isn't currently possible without the use of an adapted form of the lambda mentioned earlier in this thread.

[markusrydh]

Yes, we are also needing this. Will try the workarounds.

[dnsymphony]

This is a common requirement for a Spring Boot-based microservice using Actuator, where you want the Actuator endpoints running on a different port available only internally with elevated security and the other endpoints available publicly.

[matandavid]

👍

[talolard]

+1

[angusfz]

Any update after 1 year ?

[polarizeme]

@angusfz added to their kanban board in Nov, no one's shown as assigned yet =/

[derekbedwards]

Just hit this as well trying to host a service that exposes multiple ports. Didn't expect there to be such a fundamental limitation in ECS :(

[apoorvkapil]

I was working on Kong APIM deployment on ECS [Fargate] (https://github.com/Kong/docker-kong/issues/242) and stuck on this. This should be taken as bug rather than a feature as it's blocking many AWS clients moving forward with fargate. The assumption of exposing a single port from container is really unconvincing.

[krisread]

This is a blocker to us as we have docker containers exposing 2 ports and cannot split this into 2 containers. There seems to be no way for us to get this deployed with terraform.

[coultn]

This is the same issue as https://github.com/aws/containers-roadmap/issues/12

[bkali]

+1 this is a critical roadblock for us, preventing us from using Fargate/ECS. We need service processes (like https://nats.io/ ) that use different ports within the same server process. @coultn , any thoughts on when this feature may be released ? Thank you!