caltechads / deployfish

Manage your whole application lifecycle in Amazon AWS ECS.
http://deployfish.readthedocs.io
Other
90 stars 20 forks source link

Tasks stuck in loop when service discovery is enabled #39

Open ookblah opened 5 years ago

ookblah commented 5 years ago

When I enable service discovery my tasks go into "activating" status and then stop, get recreated again, etc.

Not entirely sure how to fix, but I was doing some digging and this might be of help. Adding some extra config around healthchecks:

https://forums.aws.amazon.com/thread.jspa?threadID=283572

cmalek commented 5 years ago

Thanks for the bug report! Could you attach your deployfish.yml so we can try to replicate your issue here?

ookblah commented 5 years ago

Sure thing, here's the relevant service from my deployfish.yml and output from the CLI. Don't mind the stuff about traefik. I'm basically spinning up a php service.

#PHP Service
  - name: app
    cluster: cv-staging
    launch_type: FARGATE
    execution_role: <service role arn>
    network_mode: awsvpc
    service_discovery:
      namespace: cv-staging
      name: app
      dns_records:
        type: A
        ttl: 10
    vpc_configuration:
      subnets:
        - <subnet 1>
        - <subnet 2>
      security_groups:
        - <security group 1>
      public_ip: ENABLED
    cpu: 256
    memory: 512
    count: 2
    family: app
    containers:
      - name: app
        image: <php image>
        labels:
          - "traefik.enable=false"
        logging:
          driver: awslogs
          options:
            awslogs-group: cv-staging
            awslogs-region: us-east-1
            awslogs-stream-prefix: app

I'm able to initiate creating the service but then it repeatedly goes into this in my cli before failing deployment. Tasks in the ECS dashboard go into an endless cycle of activating, provisioning, etc. At first glance everything seems to run fine and the service appears to be up, but if you switch to the stopped tasks you see all the dead ones. If I remove the block about service discovery it works fine.

Deployment Desired Pending Running
PRIMARY 2 2 0

Service:
(service app) has started 2 tasks: (task 6b2b6a2d-a0ca-4904-a971-76895c65f3ae) (task f416f133-6e8e-4872-852f-64d0668619ae).

Deployment unready

Deployment Desired Pending Running
PRIMARY 2 2 0

Service:
(service app) has started 2 tasks: (task 6b2b6a2d-a0ca-4904-a971-76895c65f3ae) (task f416f133-6e8e-4872-852f-64d0668619ae).

Deployment unready