search

aws / amazon-ecs-cli

The Amazon ECS CLI enables users to run their applications on ECS/Fargate using the Docker Compose file format, quickly provision resources, push/pull images in ECR, and monitor running applications on ECS/Fargate.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html
Other
1.78k stars 302 forks source link

Deployment doesn't completes in 5 minutes when using Fargate with ALB #902

Open laur1s opened 5 years ago

laur1s commented 5 years ago

Summary

Deployment doesn't completes in 5 minutes when using Fargate with ALB

Description

Observed Behavior

time="2019-08-08T13:32:21Z" level=info msg="Using ECS task definition" TaskDefinition="tutorial:15"
time="2019-08-08T13:32:21Z" level=warning msg="You cannot update the load balancer configuration on an existing service." serviceName=tutorial
time="2019-08-08T13:32:21Z" level=info msg="Updated the ECS service with a new task definition. Old containers will be stopped automatically, and replaced with new ones" desiredCount=1 force-deployment=false service=tutorial
time="2019-08-08T13:32:21Z" level=info msg="Service status" desiredCount=1 runningCount=1 serviceName=tutorial
time="2019-08-08T13:32:51Z" level=info msg="(service tutorial) has started 1 tasks: (task 58c1a587-ec79-469c-8046-1b8b84231a4d)." timestamp="2019-08-08 13:32:45 +0000 UTC"
time="2019-08-08T13:33:06Z" level=info msg="Service status" desiredCount=1 runningCount=2 serviceName=tutorial
time="2019-08-08T13:33:21Z" level=info msg="(service tutorial) registered 1 targets in (target-group arn:aws:elasticloadbalancing:eu-west-1:xxxx:targetgroup/laurynas-alb-target/xxx)" timestamp="2019-08-08 13:33:07 +0000 UTC"
time="2019-08-08T13:33:51Z" level=info msg="(service tutorial) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:eu-west-1:xxx:targetgroup/laurynas-alb-target/xxx)" timestamp="2019-08-08 13:33:46 +0000 UTC"
time="2019-08-08T13:33:51Z" level=info msg="(service tutorial) has begun draining connections on 1 tasks." timestamp="2019-08-08 13:33:46 +0000 UTC"
time="2019-08-08T13:38:06Z" level=fatal msg="Deployment has not completed: Running count has not changed for 5.00 minutes"
hencrice commented 5 years ago

Hey @laur1s ! Could you be more specific about the issue?

  1. Do you wish to have a configurable instead of the hardcoded 5-minute timeout?
  2. Or would you like us to provide assistance on your specific Fargate service?
laur1s commented 5 years ago

Hi @hencrice The service works fine. It actually finishes deploying in just over 5 minutes due to the fact that I use Load Balancer with it. Configurable timeout would work too, or ability to do actual Blue- Green Deployments using ECS CLI

mkrn commented 4 years ago

The same happens to me. Is anyone working on it or can I help with the pull request? Is there a workaround in a way ECS service or ALB may be configured @laur1s ?

collinglass commented 4 years ago

It should be already configurable using the timeout flag.

https://github.com/aws/amazon-ecs-cli/blob/61f83b2e945571e10f14688d19237998dd178a57/ecs-cli/modules/cli/compose/entity/service/service_helper.go#L116-L123