Deployment using CloudFormation templates

numtel commented 7 years ago

On the same thread as #3 but in a simpler direction, this PR is hoping to provide Cloudformation stack templates and a bit of documentation for deploying this boilerplate app to AWS ECS.

[x] loadbalancer.yml Internet-facing load balancer CloudFormation stack template
[x] application.yml Task Definition/ECS Service stack template
[ ] Update readme with how-to use these templates

ruslantalpa commented 7 years ago

Good direction. An idea, not sure if it's good. The application will also need some details about the cluster, should the LB stack export the info about the cluster or should the app stack have it's own lambda. I am leaning towards Lb stack exporting that info.

numtel commented 7 years ago

For application.yml to use the StackResourcesLambda, it can just export the lambda ARN. It would be no problem to invoke it again in another stack template. This would be similar to how those cloudformation-helpers get installed.

That's a good point about the target group. I'll change it to export the listener ARNs (HTTP and possibly HTTPS) then application.yml will contain a target group and up to 2 listener rules

numtel commented 7 years ago

Cool, this works for a development setup with the sandbox Postgres container. I haven't tried to instantiate a production stack yet or use HTTPS.

ruslantalpa commented 7 years ago

how about removing the "development" concept altogether. This kit makes it possible to do the development on your computer, i don't see why one would wnat to have the development stack in aws ...? Maybe what would make sense is to say "db-location" "container" or "rds" but i would not complicate this template, this is supposed to be for production so let's just go with RDS (after we implement all the other stuff, for now keep it in docker so that we can iterate) I'll review in detail later today and see about feedback.

ruslantalpa commented 7 years ago

haven't forgotten about this, have been busy with this https://github.com/subzerocloud/pg-amqp-bridge past few days. Will try to review this weekend. Great work and sorry for the delay

numtel commented 7 years ago

Hey no worries, I'm away from my computer for a few days anyways. No rush.

On Jun 23, 2017 1:41 PM, "Ruslan Talpa" notifications@github.com wrote:

haven't forgotten about this, have been busy with this https://github.com/ subzerocloud/pg-amqp-bridge past few days. Will try to review this weekend. Great work and sorry for the delay

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/subzerocloud/postgrest-starter-kit/pull/5#issuecomment-310768025, or mute the thread https://github.com/notifications/unsubscribe-auth/AAfqKvdY3z8MFiiTlA7enXUKGbsWCHfdks5sHCL6gaJpZM4N_F0- .

ruslantalpa commented 7 years ago

The big picture i would like to get to (feel free to comment on it): First deployment process should be like:

create the cluster in aws
start Lb stack
start application stack
- on init the desired service count should be 0
- the stack also creates the openresty repository and uses the the ListenerHostNamePattern in the name of repository, like mydomain.com/openresty
- the task definition should use this repo by default for openresty but a custom one should also be allowed to be specified like you have it now
- the stack should also have the option to create an RDS instance
build and push your the docker image to openresty repo
if the db is created by the stack (RDS), initialise it (methodology will be developed later)
update the stack with only the DesiredCount param changed to 1 (or more), at this point the app comes online

to push a new version one would do:

push changes to the db
push new openresty image
update the stack and change only one param (version, see my sample templates) which triggers ECS to pull new images and do a rolling update

With this plan in mind, here are the things i see missing from template

[ ] remove the "development" notion, the template is for production
[ ] create the openresty repository (only if the parameter for image is the default one)
[ ] bring up an RDS instance (this is the last thing, for now keep the docker version but use the official image, no need for custom repo)

I'll also comment on individual lines if i see issues there (If you feel this is a lot of work for which you have no time, let me know, i'll merge to a separate branch and continue the work there)

numtel commented 7 years ago

I agree with most of that. I do have these points though.

If using a standard PostgreSQL image, that container wouldn't be part of this task definition because you wouldn't want a SQL server for each application instance. Including it is more for a staging environment with ephemeral storage. Because it is reset in sync with the application, it would not be convenient to have to run the set-up SQL commands manually.

If we're going to include RDS instantiation in the template, it seems like we might as well have kept in the EC2 cluster. Why not rely on the wizard that AWS provides in this case as well?

The "development" switch bugged me after I committed it. What's the effect of that env var? Does it cause more verbose logging? Maybe we just have a parameter and have it only connected to the env var with a separate parameter for creating a postgres server container. It would still be useful to be able to start an application in that state. As for the Serial value to change to update the container image, specifying the image tag has been sufficient in my experience. Pushing every version to a unique tag makes it easy to revert If needed, among other benefits. I've used a serial like that for nested stacks before but I'm not sure of a good way to use nested stacks in the project since it requires an S3 bucket to keep the templates. Creating the repository with cloudformation does seem like a good idea though as well as starting with 0 desired.

If you want to work on a branch go ahead. I'm not in an extreme hurry getting this completed and it seems like you have many ideas. I do appreciate the feedback though, it gives me good perspective on other ways to make the templates.

ruslantalpa commented 7 years ago

postgresql image: i was commenting on that only as a temporary thing, until we get the stack working and then switch to dedicated RDS.

RDS in application stack, i am not 100% it should be in stack but i think it makes sense especially in PostgREST projects. Your application code is split between db and openresty so db is part of the application, not an external system. Another advantage i see is that you have one command to "scale" your application (stack update). I think cloudformation is able to scale rds instances by cloning the existing one and spinning a new one and since it's part of the stack, the containers should automatically switch to the new instance.

please make this pr agains https://github.com/subzerocloud/postgrest-starter-kit/tree/cloudformation branch. I'll leave it open in case you get to it before i do but when i have time to work on it, i'll merge then create my own PR and ask for your feedback there.

Thank you again for this PR, really moved things forward a lot!

ruslantalpa commented 7 years ago

Actually i think i'll merge this since it's in a separate folder and does not interfere with the rest of the project.

If you want to continue and and add changes, you are very welcomed to open a new PR. Thank you such for the contribution 👍

subzerocloud / postgrest-starter-kit

Deployment using CloudFormation templates #5