colekettler
commented
4 years ago @hectcastro I took a pretty deep dive into this. Setting up container builds and publishing is simple enough. Terraform is a lot more complicated.
Unfortunately there's no easy, fire-and-forget solution. It's possible, but all of the potential approaches involve some changes to how we handle deployment, either at the scripting or infrastructural level. Here are the options I've considered, grouped by rough categories:
Repository / Actions Settings
Make build logs private
We evaluated this approach in azavea/operations#425 in the context of CircleCI.
This is not an option for Actions at this time. We can't make only the build logs private while keeping the repository public. As such, this isn't useful as a general approach and I recommend against it.
Terraform Settings
Indicate all sensitive values
We evlauated this approach in azavea/operations#425. There have not been significant developments in this area since.
At this time, this has to happen at the provider level and is not always consistent. There is an open issue to give consumers a way to indicate that variables are sensitive, but that feature is not going to be available within a definite timeframe. We should revisit this later on.
Masking Values
Mask values in output with tfmask
Source: https://github.com/cloudposse/tfmask
Pros:
- Would only require a small change to our Terraform commands.
Cons:
- Would require installing an additional dependency into our Terraform image.
- Relying on a regex for masking introduces high risk for error and unintentionally leaked secrets, especially when introducing new secrets.
Recommendation: No. The potential for human error is too high and we wouldn't have a good way to detect if we've leaked something.
Mask values in output with terrahelp
Source: https://github.com/opencredo/terrahelp/tree/master/examples/mask
Pros:
- Would only require a small change to our Terraform commands.
Cons:
- Would require installing an additional dependency into our Terraform image.
- Shotgun approach. Would remove a large amount of potentially useful debugging information from our log output.
Recommendation: Probably no. This is better than the regex option, but losing that much non-sensitive information from our build logs could make it much more time-consuming and awkward to investigate failed deploys.
Mask values in output with GitHub Actions masking
Source: https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#masking-a-value-in-log Pros:
- All config is attached to the GitHub repository. No general changes needed in STRTA or Terraform.
Cons:
- There are several gotchas around specifying masks in workflows like this.
- Would also require us to specify all masked values ahead of time, likely as GitHub Secrets to avoid storing them in cleartext. This is potentially a lot of duplication with respect to the
terraform.tfvarsfile, and we'd need to remember to update both values if we ever roll credentials or else we'll leak the new ones.
Recommendation: No. Creates too much duplication and potential for either human or workflow errors.
Secrets Management
Manage sensitive values with AWS Secrets Manager instead of Terraform
Pros:
- This is one of the canonical "right" ways to handle secret management. If implemented correctly, it would have an overall positive effect on the secure handling of credentials throughout our infrastructure.
- Doesn't rely on Terraform for secrets management, because Terraform wasn't built to be a secrets manager.
- Preserves our existing methods of providing Terraform variables and the structure of our
infrascript.
Cons:
- Very large architectural change. Would require full buy-in to using AWS Secrets Manager for loading all secrets in at runtime. Would require application-level and infrastructure-level integration. May require an application-level caching layer to preserve performance.
- Incurs additional cost of $0.40 per secret stored and $0.05 per 10,000 API calls. Secrets can be grouped together as JSON, but we'd need to consider these groupings carefully.
- Probably security overkill for many of our projects and likely too much implementation overhead to be relied on as a general approach for all projects.
Recommendation: A begrudging no. I love this idea conceptually, it's a very correct and complete approach, but I don't think it's practical for us in the slightest.
Deployment Architecture
Split deployment and build jobs
Pros:
- We could use Actions for the test / build / publish process, then invoke a Jenkins or a private CI platform job for deployment.
Cons:
- Now we maintain two deployment systems for this project: Actions and likely Jenkins.
- Troubleshooting failed builds may now require a context switch to another CI platform.
- Split deployment infrastructure, another network hop, another point of failure.
Recommendation: No. I don't think GitHub Actions offers enough distinguishing value to justify using it for the easy parts but still leaning on private CI for the more sensitive parts.
Redirect Terraform log output to S3
Pros:
- Most changes will be confined to the
infrascript. Our deployment workflow and architecture does not need to change significantly. - AWS credentials are the only credentials we will need to store with GitHub Secrets.
- Preserves a deploy log in S3.
Cons:
- Makes workflow output logs considerably less useful when troubleshooting a failed deploy.
- Requires setting up a new subdirectory within the Terraform config bucket, or a new bucket outright.
Recommendation: Maybe. It's a slight inconvenience, but I think we can make it less annoying by outputting S3 links like we do with ECS tasks in django-ecsmanage. There's room to smooth over the disconnect between workflow logs and Terraform logs.
All told, if I had to suggest one of these, I'd suggest redirecting plan and apply output to S3. This gives us a layer of access control that we're used to working with and allows us to script around some of its inconvenience if we want.
Barring that approach, I'd recommend we stick with private CI until there are significant developments in either Terraform or GitHub Actions to cover this use case.
hectcastro
Can we safely use GitHub Actions to build container images, publish them to ECR, and run Terraform from within GitHub Actions?