Add Option to Sort Projects Based on Module Dependency Graph

[austinsherron]

This is an awesome tool that we've just incorporated into our Terragrunt + Atlantis stack. Thanks for developing and maintaining this!

This tool is almost perfect, but something that would put it over the top would be the ability to sort generated projects in atlantis.yaml based on the module dependency graph. Since atlantis runs plans/applies based on the order of projects in atlantis.yaml (assuming non-parallel excution), this would allow me and my team to confidently run atlantis apply and trust atlantis to apply dependencies before their dependent modules. As it stands, we address this issue by maintaining a separate project/workflow for running terragrunt run-all apply.

For example, a project with this structure:

module-1/
  dependency on module-2
module-2/
  dependency on module-3
module-3/
  no dependencies
module-4/
  dependency on module-1

Would result in an atlantis.yaml with a project ordering of:

• module-3
• module-2
• module-1
• module-4

Functionally, the generated projects would be ordered by a topological sort of the dependency graph. For my use case, the topological sort wouldn't necessarily have to have repeatable ordering (exact same order on subsequent runs), but I can imagine that'd be useful as well.

I recognize that there are complexities and nuances to this feature, but I'd be happy to lend a hand implementing it if others think it'd be useful.

[dmattia]

Oh heck yeah, I'd love the output top sorted. If you want to try your hand go for it, and let me know if you have any questions or would like help

[austinsherron]

As it turns out, Atlantis doesn't actually plan/apply projects based on project order in atlantis.yaml. After https://github.com/runatlantis/atlantis/pull/2178 is added to an Atlantis release, however, it should be relatively straightforward to use this issue to track adding correct execution_order_groups based on a topological sort of project dependencies.

[okgolove]

https://github.com/runatlantis/atlantis/pull/2178 has been merged and released in 0.19.5. @tufitko thank you for the contribution! Any chances you add ordering support in this repo?

[tufitko]

@okgolove Yeah! I already have a poor commit and now I'm remaking it. I'm going to create PR this week.

[okgolove]

@tufitko hello! I've just tested execution_order_group and can't get one thing. I have:

terraform {
  source = "tfr://registry.terraform.io/terraform-google-modules/cloud-storage/google//?version=3.3.0"
}

include "root" {
  path   = find_in_parent_folders()
  expose = true
}

dependency "service_account" {
  config_path = "../workload-identity"

  mock_outputs = {
    gcp_service_account = {
      email = "test@mock.com"
    }
  }
}

inputs = {
  project_id = "test"
  region     = "europe-west-3"
  names      = ["test-dumps"]
  prefix     = ""
  bucket_storage_admins = {
    test-dumps = "serviceAccount:${dependency.service_account.gcp_service_account["email"]}"
  }
  set_storage_admin_roles = true
}

In plan I see a value from the mock ("test@mock.com"). workload-identity hasn't applied yet and it has order 0, this module has 1. atlantis plan shows test-dumps = "test@mock.com"

When I run atlantis apply in a PR it still tries to use test@mock.com as an output. What should I do? Will mock_outputs_merge_strategy_with_state = "shallow" help me here?

[tufitko]

@okgolove Hello! When you run atlatnis apply is workload-identity applied first? If yes, you can check outputs in the workload-identity state. I see that you don't use outputs in path ${dependency.service_account.gcp_service_account["email"]}. Is it working? 🤔

Also, I recommend using mock_outputs_allowed_terraform_commands option to ensure that mocks aren't used when triggered apply

[tufitko]

Will mock_outputs_merge_strategy_with_state = "shallow" help me here?

It is an option just merging outputs from state to mocks. It helps for example in cases when you already have state and want to add a new key in this state

[okgolove]

Ah, sorry, accidentally deleted outputs from the string during obsufcation the example. Of course this is:

 test-dumps = "serviceAccount:${dependency.service_account.outputs.gcp_service_account["email"]}"

This is plan result: 1st is workload-identity, 2nd is bucket. It seems it applies them respecting their order. When I run apply 1st is applied successfully, but second throws an error:

Error: Error applying IAM policy for storage bucket "b/test-dumps": Error setting IAM policy for storage bucket "b/test-dumps": googleapi: Error 400: Service account test@mock.com does not exist., invalid

So it still tries to use mocks. May be I should set mock_outputs_allowed_terraform_commands = ["validate", "plan"].

[tufitko]

Oh! Do you use planfile generated from plan step? Because it's outdated after first apply you need executing apply step like this: terragrunt apply -no-color -auto-approve

[okgolove]

Yeah, it seems this is the problem. I use default workflow from Atlantis docs, something like:

repos:
- id: "/.*/"
  workflow: terragrunt
workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -input=false -out=$PLANFILE
      - run: terragrunt show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply -input=false $PLANFILE

[tufitko]

I think this workflow works only in cases when terragrunt modules haven't dependencies. here is my workflows:

workflows:
  terragrunt:
    apply:
      steps:
        - run: terragrunt apply -no-color -auto-approve
    plan:
      steps:
        - run: terragrunt hclfmt --terragrunt-check
        - run: terraform fmt -check -recursive
        - run: terragrunt validate-inputs --terragrunt-strict-validate
        - run: terragrunt init --upgrade --reconfigure
        - run: terragrunt plan -no-color -lock=false -out=$PLANFILE
        - run: terragrunt show -json $PLANFILE > $SHOWFILE

[okgolove]

@tufitko thank you very much for your explanation and patience!

[jmgilman]

After reading this issue history, I'm a bit confused. It looks like #237 implemented support for execution_order_groups with a flag, but it's usage doesn't appear to be documented anywhere. Does setting this flag solve the original issue stated here?

We're investigating using Atlantis and this tool to manage our mono-repo, but we have dozens of modules with various inter-dependencies (i.e. run-all apply produces six separate groups). It's unclear if this flag would help in this situation.

[tufitko]

@jmgilman Yeah, it's working, no matter how many groups. Groups will be executed one by one asc https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#order-of-planning-applying