stacker import capability to create blueprints from existing cloudformation

[GarisonLotus]

So, I wanted to open this up to start a discussion about something I kind of want to work on to help with the massive existing JSON CFN problem we have at Sturdy. Recently @troyready and I had a quick conversation in passing about the problem we will eventually have: getting the thousands of exisiting deployed CFM stacks from our older clients into stacker blueprints so we could potentially use them elsewhere/redeploy. If we had a way to import a deployed stack into a blueprint, that would help... even if it's a blueprint without any defined variables. At least we would have a starting point.

Troy mentioned this: https://github.com/cloudtools/troposphere/blob/master/scripts/cfn2py

I'm wondering what you guys think about the idea. I'm imagining something like stacker import <fq-stack-name> <output> would build a blueprint and populate the troposphere portion of the blueprint by using the native cfn2py functionality. Then I would have a real blueprint that could be a part of my stacker yaml config.

What does everyone think?

[GarisonLotus]

@phobologic, Any feedback on this idea? I'm probably going to try to take a stab at it this weekend (if I can find the time.) A feature like this would allow us to easily migrate the sturdy "hand rolled" cloudformation customers into stacker. Curious if you guys had something like this on your roadmap.

[phobologic]

I'm all for it - though from everything I've seen in maintaining troposphere it's a very difficult feature to implement in a consistent way. The code for cfn2py was always a bit of a nightmare, and really painful to maintain. If you can crack it though, I'd be all for it!

We don't have any plans ourselves, since we start with stacker from scratch.

[ejholmes]

I've been thinking a bit about this. I think you could just subclass stacker.blueprints.base.Blueprint and override the render_template method to just return some raw CloudFormation json. Atleast as a quick way to get existing stacks into stacker.

I guess the only problem then becomes namespacing; if existing stacks don't follow the same naming convention, there's no way to tell stacker to not namespace a stack.

[phobologic]

@ejholmes We really need to just create that plain cloudformation template blueprint class - it's been brought up many times, and it can't be all that hard to implement. The namespacing shouldn't be a problem, since namespaces, for the most part, are only a concern inside a single stack, and the blueprint doesn't control what the stack is named.

[ejholmes]

I meant that if you have stacker configured with a namespace, but you have existing stacks in CloudFormation that don't match that namespace, you can't really import those stacks into stacker.

[GarisonLotus]

I agree Eric re: your namespace comment and have had that thought in the past when thinking about this. We would need some sort of "xref" like ability at the stack name level since any imported stack would not necessarily follow the standard stacker namespace.

Although I went the path of trying to cfn2py, I have a somewhat functional cfn2blueprint script in my fork and (if I can find the time) I should get the last few issues resolved to submit here for feedback. I haven't touched it in a few weeks since I didn't have an immediate need for it. There always can be a better way.

[ekenstam]

@GarisonLotus I'm interested in your cfn2blueprint script. I have a bunch of raw CFN that I want to use as Stacker blueprints, until I have a chance to convert them to Troposphere. Would your script do the trick? Can give a pointer to where that code is in your fork? Or did you come up with a better way to solve this problem?

[GarisonLotus]

@ekenstam - My script was never finished as I ended up not needing the ability in the job I was working at the time, but you are welcome to see if it helps. Here is the script itself in my own fork: https://github.com/GarisonLotus/stacker/blob/cfn2blueprint_script/scripts/cfn2blueprint

Make sure to read over the docstring before you use it. I have used it previously, but the output still required heavy modification before it worked. You will still need to know troposphere to get your blueprint into a working state. Basically, as is... it can give you a starting point. Also, since stacker doesn't really allow you to specify fully qualified stack names in the config, you can't use it to import existing stacks that doesn't already have a naming scheme that can be used with the stacker - schema. If you do choose to use it on something actually running, make sure you run stacker in interactive mode (-i) to ensure a changeset gets created if something has changed.

I'll eventually get around to finishing this... I swear.

[ralph-tice]

Along these same lines, I have some WIP for cfn2py for YAML support -- complicated by the !Ref shorthand functions... https://github.com/ralph-tice/troposphere/commit/49a7a04fb2de01e5bd908c9939989bd78a44a564