gruntwork-io / terragrunt-infrastructure-live-example

A repo used to show examples file/folder structures you can use with Terragrunt and Terraform
https://www.gruntwork.io/
Apache License 2.0
788 stars 474 forks source link

How does CI\CD fit into all of this? #59

Open nabladev opened 3 years ago

nabladev commented 3 years ago

I think this is still an open problem today.

Given that modules are stored in another repo. Integrating all of this with Jenkins would be like:

  1. new branch on modules repo
  2. commit and submit PR
  3. Jenkins triggers and comments the output of terraform plan on the PR
  4. PR gets merged

We can suppose that on 3. terragrunt plan is run under /non-prod/us-east-1/stage because that's where we deploy stuff under development. So we know the environment but not the module, is it mysql or webserver-cluster?

Furthermore, if we want to deploy stuff in /non-prod/us-east-1/qa how are we going to do that via Jenkins? Should I have like a separate Job just to run the plan/apply command? The same issue arises, how doeas Jenkins know what module to plan/apply?

yorinasub17 commented 3 years ago

Indeed, infrastructure CI/CD is an area of active development to identify the right approach. Our latest thinking on this is summarized in https://gruntwork.io/guides/automations/how-to-configure-a-production-grade-ci-cd-setup-for-apps-and-infrastructure-code/, which has some answers to your questions (but not all).

The key idea here is that module releases should be completely independent of actual infrastructure releases, and you should be able to test and validate your infrastructure modules without deploying to live environments in your infrastructure-live repo. Specifically, all the module infrastructure CI should be handled by "unit tests" using a framework like https://terratest.gruntwork.io/.

In that world, the flow would be more like:

You can then configure automation pipelines for releasing this code depending on your tolerance. For example, a common workflow would be:

alxy commented 3 years ago

@yorinasub17 Thanks for your answer, however, we have made the experience that very often our modules have a lot of (expensive) dependencies. For example we use Azure user managed identities to handle passwordless authentication, but they need a special role assignment scoped to our kubernetes cluster (AKS) to do their job and are allowed to talk with our applications. So if we wanted to test this (otherwise very simple) module in isolation, we would need to spawn and configure an entire AKS just for the test and tear it down after that. Also talking about pipeline agents and the fact that multiple teams actively work on all the modules, it seems very hard to follow the setup you described.

yorinasub17 commented 3 years ago

For example we use Azure user managed identities to handle passwordless authentication, but they need a special role assignment scoped to our kubernetes cluster (AKS) to do their job and are allowed to talk with our applications. So if we wanted to test this (otherwise very simple) module in isolation, we would need to spawn and configure an entire AKS just for the test and tear it down after that.

Yes as mentioned above, infrastructure CI/CD is an area of active development and research. You are not going to find a single solution that works for all use cases. It all comes down to what tradeoffs you are willing to make based on your level of tolerance for risk.

Given that, for the situation you described, you have a few options:

yorinasub17 commented 3 years ago

Also talking about pipeline agents and the fact that multiple teams actively work on all the modules, it seems very hard to follow the setup you described.

This setup actually works much better in larger scale teams, since the module testing can happen in isolation (because they don't conflict). The more you share resources, the more contention you are going to have across the teams, so the advantage of always deploying new resources (even if expensive) is that you can have the confidence that teams can work in isolation (in "unit") without worrying about test contention and conflicts that pollute their environments when developing the modules.