search

ThoughtWorksStudios / eb_deployer

AWS Elastic Beanstalk blue-green deployment automation from ThoughtWorks Mingle Team
MIT License
400 stars 94 forks source link

blue-green hangs on 'errors' in update env operation? #69

Open mepatterson opened 8 years ago

mepatterson commented 8 years ago

So I'm using eb_deployer to push an app with a Dockerfile up to EB. EB then handles all the spin-up of Docker containers and all that. Works great. But when I started using blue-green for my production deploys, I ran into this issue: once I have a green, running environment and a red, inactive environment (red because the instance has been killed so health check is failing, but that's okay), deploys with eb_deployer will hang at:

[2016-02-23 03:34:56 UTC][environment:prod-b-9cf234d] Successfully deployed new configuration to environment.
[2016-02-23 03:34:57 UTC][environment:prod-b-9cf234d] Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

The 'errors', as far as I can tell, are actually because it's just taking a while to spin up a whole new instance and rebuild the docker container from scratch. But that's okay with me. If I wait for a bit longer, on the EB side, the deploy finishes fine and it switches from red to green. But eb_deployer never seems to catch on that this has (finally) happened, and it just hangs there forever, so it never does the rest of the magic to tell EB to switch cnames and set inactive and all that.

Any ideas on how I could resolve this? Is there a setting in eb_deployer, or a timeout or something I can change?

wpc commented 8 years ago

hi, @mepatterson Can you add a --debug option and past the full deploy log here. The hanging behavior is not acceptable, I need figure out where exactly it happens.

-- wpc

mepatterson commented 8 years ago

Sorry, I ended up switching to phoenix mode for my docker-based deploys and I haven't been able to reproduce this issue since. It might have been something unique to my configuration of eb_deployer. I'll add more details if I see this happen on any future deploys I setup.

On Thu, Feb 25, 2016 at 11:52 AM WPC notifications@github.com wrote:

hi, @mepatterson https://github.com/mepatterson Can you add a --debug option and past the full deploy log here. The hanging behavior is not acceptable, I need figure out where exactly it happens.

-- wpc

— Reply to this email directly or view it on GitHub https://github.com/ThoughtWorksStudios/eb_deployer/issues/69#issuecomment-188902854 .

harryherbig commented 8 years ago

Same problem here. Deployment with eb_deployer fails, if inactive has 0 instances:

2016-04-27 15:31:58 UTC+0200 INFO Environment health has transitioned from RED to GREEN

2016-04-27 15:28:56 UTC+0200 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

2016-04-27 15:28:56 UTC+0200 INFO Successfully deployed new configuration to environment.

2016-04-27 15:28:56 UTC+0200 INFO New application version was deployed to running EC2 instances.

2016-04-27 15:28:22 UTC+0200 INFO Command execution completed on all instances successfully.

2016-04-27 15:28:21 UTC+0200 INFO Deploying new version to instance(s).

2016-04-27 15:26:53 UTC+0200 INFO Adding instance 'i-XXXXXX' to your environment.

2016-04-27 15:26:52 UTC+0200 INFO Added EC2 instance 'i-XXXXX' to Auto Scaling Group 'XXXXX'.

2016-04-27 15:26:28 UTC+0200 INFO Updating environment production-a-XXXX configuration settings.

2016-04-27 15:26:20 UTC+0200 INFO Environment update is starting.

wpc commented 8 years ago

@harryurban thanks for reporting this. I will take a look

harryherbig commented 8 years ago

Even increasing the Command Timeout at elasticbeanstalk doesnt help: - namespace: aws:elasticbeanstalk:command option_name: Timeout value: "1000"

tried it with our CI Server and from my machine with --debug option. -> no difference I guess spinning up the first instance takes to long?

Jwata commented 8 years ago

Same issue here. Any update for this? If not, I'll try to have some waiting time for it.

I'm thanksfully using eb deployer for our projects, but the issue happens quit often. When the number of instance is zero, I think it always happens. Then I retry deployment again and it successes.

wpc commented 8 years ago

I found the issue. Will release a fix in 0.6.7 soon

dgolombek commented 8 years ago

Any further updates on this? I see that #79 was merged, but that simply handles the error and aborts -- did you find a root cause here?

ghost commented 8 years ago

I'm seeing this issue all the time. Any update on the fix?

fkelly commented 8 years ago

We're also very keen to get this fixed. I'm sure you're busy ... any chance you want to describe how you'd do the fix and one of us can implement it and do a pull request? Thanks!

ghost commented 8 years ago

So looks like elasticbeanstalk doen't support the configuration of asg MinMax 0. So when we do a deployment on a blue-green stack and blue has 0 instances and you deploy code. It'll update the asg to MinMax that you set but doesn't wait till the ec2 instances is up and ready before it says that its updated the code for ec2 instances. So you get that error. So we'll never get that success message the eb_deployer is looking for. The fix seems to be listening for another common message.

If you listen for

"Successfully deployed new configuration to environment"

this seems to fix the problem. The smoke test is initiated and we just keep polling till the ec2 instance is up and running

ghost commented 8 years ago

here's the pull request #86

dcrockwell commented 7 years ago

Can confirm this still seems to be an issue. Waiting for #86 to be merged in.

New application version was deployed to running EC2 instances.
Successfully deployed new configuration to environment.
Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

Then, after a few more minutes:

Environment health has transitioned from RED to GREEN