Customizing the Docker4AWS Template

[hierozbosch]

I want to discuss/learn about how to adapt the Docker4AWS Template to more complex apps.

For example, I am already modifying the stack by adding an extra worker node for my database after the Docker4AWS build completes. I need a node with a larger instance size, different from all other workers, so I add it manually and then target it for the db service via placement constraint.

Now I'm looking at moving up to a db cluster that wants 2 types of worker nodes configured in auto-scaling groups--in addition to the nodes running webservers. Hacking an extra node is one thing, but adding 2 additional ASG's is WAY off the update path.

So are there options? Nested stacks? Parallel stacks tied with exported output values? Either way, I'd be looking for some kind of hooks into the D4AWS template.

Thanks, Matt Hodges

[hierozbosch]

AWS::Include Transform

I add an extra worker to my swarm to get a bigger database engine. I was doing this manually after the D4AWS template finished; I used the EC2 console to create, and then joined it to the swarm by hand.

Now I've got my instance defined as a Cloudformation resource, so it's fully integrated as a swarm worker. I have it stored on S3 and can pull it in using the AWS::Include transform in a changeset.

I'm not saying this is worth doing, but if there was an "Custom Resources" parameter in the D4AWS template, I might be able to slide in my resources without changing the template at all.

[FrenchBen]

I believe that nested stacks may be a good solution - There are a couple of examples out there, but currently not something we support directly. https://github.com/widdix/learn-cloudformation/tree/master/lab7-nested-stacks

With that being said, I'd very much like to see what your final template looks like, if you go the nested route way; additionally it would be great to document how you went about it, so that it can be added to our official docs.

In regards to AWS::Include Transform, that would also be a good solution, but I'm unsure how the upgrade process would handle it. Can you share the JSON for the changeset, so we can get an idea of what it would entail if we did want to integrate it?

[hierozbosch]

Thanks @FrenchBen, I think my changes got too complex--not an easy plug-n-play solution. Turns out I needed some additional policies--S3 access for the db instances, and an ECR-pull across the swarm. That kicked off a set of dominoes and I finally needed to change the base template. So an easy "include" didn't work out.

At the same time, what I'm doing becomes irrelevant as soon as D4AWS switches over to Infrakit. Do you have a timetable on that? I'm gearing up to make the switch.

So I'm still interested in extensibility for D4AWS, and would be happy to break down the changes offline. At this point we're looking at a full CFN template with significant additions and mods for our use case, so I'm not sure it would be useful to post the whole thing here.

PS. One of the problems I was working on was Issue #5, trying to get IAM Role credentials to work with ECR.

[jpaas]

I also have a need to extend the template and I had a look at the cloudformation template from AWSLabs https://github.com/awslabs/ecs-refarch-cloudformation. See how they've modularized it? Something like that would be perfect.

[jpaas]

I've tried nesting the D4AWS stack in another stack so that I can create the RDS database my app needs, but it doesn't output the subnets for me to do so :(

[hierozbosch]

@jpaas, thanks for the awslabs pointer. Re. the database, I've had luck the other way around, inserting my resources into the D4AWS stack--first a single instance for the db. The instance profile pointed to WorkerRole--this role is already tied to the policies it needs to launch into the swarm. I first tried as an include::transform, but went back to a simple changeset on the D4AWS stack until I figured out how to do it.

Doing this as a nested stack will probably need some exported stack output values from D4AWS in order to locate the subnets and policies to integrate with the swarm.

[jpaas]

@hierozbosch I was really trying to avoid modifying the D4AWS template to make it easy to pick up newer versions. I've made some reasonable progress by nesting the D4AWS stack without the VPC. Just as in that sample from AWSLabs, I create the VPC first. I just needed to add a 3rd AZ.

The D4AWS template takes PubSubnetAz1, PubSubnetAz2, PubSubnetAz3, Vpc, VpcCidr as parameters.

I still have some issues however like:

• The D4AWS template creates but does not output security groups which makes it hard to use them for the SecurityGroupIngress on my databases. Would be nicer if it took these as Parameters rather than creating.
• The D4AWS template creates but does not output the load balancer which makes it hard to assign my SSL cert and hard to force it to only accept HTTPS.

So in the end, I think I'll be stuck making a copy of the template :(

[hierozbosch]

@jpaas, I also had to modify the base template. Looking for a way D4AWS could be extended/adapted without doing that in the future.

D4AWS is moving to Infrakit: this is more modular (along w/ other advantages). I'm looking at the example here: https://dzone.com/articles/infrakit-and-docker-swarm-mode-a-fault-tolerant-and-self-healing-cluster.