Terraform GH action automation & bootstraping

elreydetoda commented 2 years ago

Alright...so, there's a bit of manual work for enabling this but when it comes to secrets that's probably a good thing :grin:

So, I've created accompanying README.md files in the different terraform directories. They cover, in general (with a YouTube video), what terraform is, how to setup the local development environment so it's similar to the CI, and the current required CI environment variables needing to be setup in the GH repo's settings.

Bootstrap

So, to handle terraform's remote state (as mentioned in #9 for a to-do) I've setup a folder called boostrap, this folder has all the necessary terraform files to deploy a Linode s3 bucket to host terraform's remote state. Like the picture below is hosting the remote tfstate for my testing:

The bootstrap section should contain everything else you need to know to use the terraform backend in other folders.

CI Setup

The CI setup (under the main terraform folder's readme) links to my blog where I talk about how to setup GitHub (GH) environments with secrets in them: https://blog.elreydetoda.site/github-action-security/

For the currently defined GH action it requires the environment to be called production.

CI workflow

Open PR (tf plan)

So, whenever a person does a PR against the infra repo it'll do a terraform fmt to do some very light "linting" (currently failing, but it'll reformat a lot of things so I'll plan on doing that after our PRs are merged to main.

After that, the next step will require manual intervention from the team/individual(s) you define based on the environment settings defined (if not using the approval, there is a potential risk (from my understanding) of exposing secrets (i.e. Linode s3 access/secret keys)).

After approving the environment, it'll run a validate and plan. The validate and plan gets commented back on the PR and this would be what it'd look like after a PR on any branch that's passed all tests (and approval for accessing environment): https://github.com/elreydetoda/jb_infra/pull/4

The GH action will update the comment after new commits are added to the PR, so if the person modifies their PR it'll automatically pickup and try to do the plan again (which'll require manual approval again).

Merged PR (tf apply)

If everything looks good from the PR, you'll end up merging the PR to master. After merged to master it'll run the the GH action again, but that also means it'll require manual approval again.

Once it's approved, it'll automatically run a terraform apply after the plan (like this action run): https://github.com/elreydetoda/jb_infra/runs/8173700896?check_suite_focus=true#step:9:10

Improvements

Currently I'm hard-coding in the working directory (i.e. the cloudflare DNS) for the GH action, but I'm planning on writing a script which'll do the following:

check what files have change
from those file, see if any of them end in .tf
get the folder paths for those
run plans for them all (only on changed folders)
(if on main) run applies for them all (only on changed folders)

Quirks

So, apparently Linode's s3 bucket versioning is still a WIP...I've posted a blog here on the commands I used to delete all objects from the bucket and all the versioned items as well: https://blog.elreydetoda.site/deleting-versions-from-a-linode-s3-bucket/

(currently I still can't delete my bucket, but I've got an open support ticket with linode right now)

Other notes

Code modifications

Instead of looking the zone id up in a YAML file which is defined locally, you can just query cloudflare's API (via terraform) based on the domain's name. (trying to eliminate using local files)
defined a priority, because after each apply it was do an in place modification (because page rules where swapping for 1st priority)
Used @ instead of jb.com, because it's essentially the same thing but @ is more portable across domains

Personal

I'm going to delete my domain from the commits, but I wanted to demonstrate what a successful run looks like (and I obviously don't have access to JB's cloudflare :sweat_smile: ). So, I used my personal domain for the examples and am planning on scrubbing that from commits in my fork's master branch (doesn't affect this PR at all, just a heads up)

I know this is a lot, so please let me know if you have any questions :grin:

gerbrent commented 2 years ago

woooo!

elreydetoda commented 2 years ago

alright, updated the original comment with lots of info :grin:

please let me know if you have any questions, but after this is merged is should "just work" :sweat_smile:

gerbrent commented 2 years ago

This is damn impressive.

JupiterBroadcasting / infra