search

bluehalo / yadda

Yet Another Docker Deployment Application
MIT License
4 stars 1 forks source link

Issues with TargetGroup Cloudformation updates #2

Open andrewmarcus opened 7 years ago

andrewmarcus commented 7 years ago

When updating the target group, Cloudformation blows away the old target group and creates a new one. That's a known issue and is annoying because the app becomes unavailable until you redeploy it.

However, a bug in Yadda prevents it from redeploying. Even when the Manifest is updated to point to the new TargetGroupARN, the old one is still used resulting in an error like the following:

error:   Error updating serviceauth. Check manifest file for missing parameters. (The target group arn:aws:elasticloadbalancing:us-west-1:538167825603:targetgroup/Mobil-Mobil-H9VCVRZZBNEG/7f19cf7eb5b55963 does not exist.)

Steps to replicate:

  1. Deploy cloudformation template
  2. Deploy application with Yadda
  3. Change cloudformation template (for instance, update health check timeout) and redeploy template.
  4. Verify that the old TargetGroup was deleted and a new TargetGroup was created.
  5. Copy the new TargetGroupARN from the Cloudformation outputs.
  6. Paste the new TargetGroupARN into the appropriate Yadda environment Manifest file.
  7. Rerun the Yadda deployment. ** It fails, indicating that the previous TargetGroupARN does not exist
  8. Delete the service in ECS (after first setting the number of instances to 0)
  9. Wait for a moment for everything to drain, then rerun the Yadda deployment ** It succeeds
bodnarbm commented 7 years ago

yadda can be updated to handle that case.

Essentially the issue is that once a service is created, its load balancer attributes cannot be changed (including changing the target group ARN). This is a limitation in the AWS SDK. What yadda can do is delete the existing service and then create a new service attached to the new target group arn.

This will still result in a service outage, but should be faster to resolve than forcing the user to themselves go to the ECS console, delete the service, and then redeploy.

As this will result in a service outage, I'll also add a guard step to ask for the users permission to delete the service.

bodnarbm commented 7 years ago

Sorry. Removing myself from this assignment due to limited capability to test and prototype this enhancement on AWS.