api: set up deployment - Githubissues

jrheard commented 3 years ago

[x] set up personal aws account if i don't already have one
[x] set up aws cli environment
[x] learn more about ecr
[x] push docker image to ecr instead of dockerhub in github actions
[x] learn more about ecs/fargate
[x] deploy app to ecs/fargate manually
[x] deploy app to ecs/fargate in github actions (see https://docs.github.com/en/actions/guides/deploying-to-amazon-elastic-container-service )

jrheard commented 3 years ago

more stuff to look into:

jrheard commented 3 years ago

this one was a doozy, i spent a whole week on it. may have lost some hair. here are some of the things that i learned:

ECR is basically amazon's dockerhub; if you want to deploy run docker images on aws, you should push 'em to ECR (edit: actually maybe dockerhub was fine the whole time, whoops, https://github.com/jrheard/rask/issues/23)
public ECR is a recent addition (december 2020). it's cheaper than private ecr because you get a bunch of storage for free. it has a few peculiarities due to being slightly different from regular ECR and also being new (so it's a bit underdocumented). it has different permissions that need to be set up for its use in the context of github actions, and you kind of need to cast a magic spell in order to push to it successfully from gha. https://github.com/jrheard/rask/blob/559953e9630c5da704010f98b98b41044761cafc/.github/workflows/main.yml is what i came up with, it took me a lot of time and research to sort that out, the tag format in particular wasn't documented anywhere that i could find. also, you need to interact with it from us-east-1 no matter what region the rest of your aws stuff is in. r/o public ecr appears to be available/shared in all regions, but needs to be written to from us-east-1.
i now know what these things area:
- vpc
- rds
- alb
- nacl
- security group
- target group
- ecs (systems, task definitions, tasks)
- secret manager (and how to pipe secrets into my running ecs container as env vars)
- aws vpn client
- free tier
- cost explorer

cost explorer in particular was really important:

i had a hard time figuring out how to set up free-tier rds and ended up setting up a paid instance on accident, i noticed my mistake $5 later. the underlying issue was that there isn't free tier support for postgres 13, but if you select postgres 12 you can find free-tier instances just fine.
i used aws vpn client (god this was complicated to set up) while i was investigating a db connectivity issue (see below); it turns out that having a vpn endpoint set up costs you money, regardless of whether or not you're using it. this cost me about $5 too before i turned it off. it turns out that you can leave the vpn endpoint intact, you just need to remove its subnet associations because those are what costs money.
you can set up aws cost explorer (or whatever its parent tool is called, not sure) to email you when you hit certain budget thresholds; i've got it set up with a bunch of manual alerts that email me every $5 or so, can you tell?

these are some of the main time sinks that i ran into:

figuring out how to push to public ecr via github actions (mentioned above) - this took a couple of days and a lot of embarrassing tentative commits (that have since been rebased out, nobody will ever know); i got most of the way there by researching and trying stuff and guessing, but the last bit (the exact correct format for the image tag) i found by doing a code search in github to find other repos that were doing public ecr stuff. it turns out that i was about 90% of the way to the right image tag format, but that was just by guesswork, and i don't know if i would have gotten to the actual correct format before giving up.
figuring out why my web API container (running on ecs) couldn't talk to my postgres db (running on rds). this took me 2-3 days and about 40 tabs and 5 long aws networking youtube videos. the problem ended up being that while my rds security group was configured to be pretty permissive about allowing incoming traffic, that was in the context of a rule that applied for traffic coming from its own security group, and the web api was intentionally on a different security group. i didn't notice this until about the twenty-fifth time that i checked the rds security group, because the incoming rules list was wide enough that there was an H-scroll bar, and the "applies only to " column of that rule was way off of the right-hand side of my laptop's screen.
how to set up an ALB that takes incoming traffic on port 80 and routes it to an ecs container's port 8000. the issue turned out to be that i had foolishly opted into the new web UI for ecs, and i'm now 99.9% certain that the new UI doesn't actually expose this feature. i found an incredible youtube video (posted just a few months ago) that walks you through this process from start to finish and saw that it used the old UI and that it let you do what i was trying to do. i genuinely don't know what i would have done without that youtube video.

these videos were totally indispensable, i don't think i could have done this without them:

it's saturday night and i didn't know what any of those words meant when i woke up this past monday morning. what a week!

jrheard commented 3 years ago

and cd's done. v thankful that the instructions on https://docs.github.com/en/actions/guides/deploying-to-amazon-elastic-container-service were good. ticket done!!!!!!!!!!!!!!

jrheard / rask

api: set up deployment #14