search

cloudtools / stacker

An AWS CloudFormation Stack orchestrator/manager.
http://stacker.readthedocs.io/en/stable/
BSD 2-Clause "Simplified" License
709 stars 167 forks source link

Disabling Rollback and Custom S3 Bucket/Key #631

Open Spareo opened 6 years ago

Spareo commented 6 years ago

I just have a couple questions for some things I haven't been able to find. I have dug around the source a bit but just wanted to verify here.

  1. Is it possible to disable the rollback feature if the stack fails to deploy? We have a lot of stuff that goes on in UserData and if that fails, we like to get on the EC2 instance and troubleshoot. With rollback, it kills the instances so we can't do this anymore.

  2. I am aware that you can set a custom bucket for where the template is uploaded, but is it possible to also set the Key so it goes into specific folders in the bucket?

xiaket commented 6 years ago

To answer your question:

  1. I think the current codebase does not support this feature. To disable rollback, we must specify OnFailure='DO_NOTHING' during stack creation. When we build the arguments for Cloudformation in generate_cloudformation_args in ~/providers/aws/default.py file, we have not included this flag at all. Correct me if I'm wrong, but I think you cannot disable rollback for an existing stack.

I think it is possible to add support for this feature, reading environment variables would be a quick fix, but I don't know whether that's a good idea. I think it's more about design than the implementation.

  1. I think the path is hardcoded in stack_template_key_name in actions/base.py file. If you are familiar with Python, I guess you could somehow override this function with a wrapper around stacker command. Again, this could be customisable, but I don't think it's a good idea. Mixing other things with cloudformation templates in a single bucket does not make sense to me, but that's just my personal opinion.
Spareo commented 6 years ago

You're right you cant disable rollback for an existing stack, it has to be passed in when creating the stack. I was thinking it could be a property on the stack objects in the configuration file like the enabled property. True by default but set false to disable rollback.

xiaket commented 6 years ago

@Spareo Excuse me but why was this issue closed? I guess we don't have a solution yet?

Spareo commented 6 years ago

I assumed it was dead since I never got anymore replies and just accepted that I can’t set the bucket key.

phobologic commented 6 years ago

I've never actually worked with stacks that don't auto rollback in the case of a failure - how do you recover when things get in that state?

As far as changing the key - I'd be fine with seeing that config option added, but I don't think it'd be a huge priority.

Spareo commented 6 years ago

Ive since just accepted the fact that we can't provide a key and worked with it. So its not a huge priority for us either.

Disabling the rollback would be a bigger deal for my team. We work with a legacy platform that is still pegged to windows. The biggest use case for us here is when we have EC2 instances in the stack and the stack fails, we will remote into the instance and look over logs, in particular the ec2 config log since the failure usually happens while the Userdata script is executing. We will also adjust the script on the instance and retry it so we don't have to redeploy each time to test minor changes.

The other situation where not rolling back has been handy is troubleshooting the deployment of Windows Server 2016 with docker images and getting the ec2 configuration and deployment scripts correct.

Normally we just log into the AWS console and kill the stack after we are done reviewing it. By the way, all these scenarios are only played out in our dev environment. The other way we can kill stacks is through our gitlab runners which allow you to stand up and bring down environments. You can execute whatever scripts you want when you bring down the environment, including killing the stacks.