Service update taking a long time

[buddhike]

Hey, I have a very basic app that I deploy as a Backend Service using the manifest below.

name: idgen
type: Backend Service

image:
  build: Dockerfile
  port: 8080
  healthcheck:
    command: [ "CMD-SHELL", "curl -f http://localhost:8080/healthcheck || exit 1" ]
    interval: 10s
    retries: 2
    timeout: 5s
    start_period: 0s

cpu: 1024
memory: 2048
count: 2
exec: true
entrypoint: [ gopi, idgen, --port, 8080 ]

network:
  vpc:
    placement: private

As you can see in the log snippet below, when I run copilot svc deploy to update the service, it takes a long time (218.8s).

✔ Proposing infrastructure changes for stack gopi-team-demo-idgen
- Updating the infrastructure for stack gopi-team-demo-idgen                  [update complete]  [218.8s]
  - An ECS service to run and maintain your tasks in the environment cluster  [update complete]  [195.3s]
    Deployments
               Revision  Rollout      Desired  Running  Failed  Pending
      PRIMARY  6         [completed]  2        2        0       0
  - An ECS task definition to group your containers and run them on ECS       [delete complete]  [0.0s]
✔ Deployed idgen, its service discovery endpoint is idgen.gopi.local:8080.

Is there anything I should do to make it quicker?

[efekarakus]

Hi @buddyspike !

We recently merged #2576 which will allow you to speed up the deployment by configuring:

http:
  deregistration_delay: 5s

Another big speed up usually occurs by minimizing your container image as much as possible using multi-stage builds: https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds if not done! 🏃 💨

[buddhike]

Hey @efekarakus, Thanks a lot for the suggestions 🙏🏾.

It's already built using multi-stage builds as you suggested and current size is 27 MB.

I will try latest bits and see if it improves. Does deregistration_delay apply to Backend Service as well? I thought it's a setting applied at the ALB level 😉

[efekarakus]

Oh whoops! I didn't realize it was a backend service you are totally right, then the field won't help. Welp in this situation I believe this is as fast as we can go!

[buddhike]

😭 I think this is more of a problem for ECS folks. I tried aws ecs update-service --force-new-deployment and it take about the same time. I will keep you posted if I get to a proper RCA. Thanks again.

[mvn-bachhuynh-dn]

@buddyspike It seems this is the nature of ECS, not related to AWS Copilot. my Pipeline always take 10 minutes for the duration ( For 1 service, 1 task :| )

[buddhike]

Hey @mvn-bachhuynh-dn, Thanks for sharing stats from your end 🙏🏾. It's good to know that others are also experiencing this problem.

[rubiagatra]

yes, I just tried the ECS workshop and deploy the backend site it takes so long