Terragrunt Atlantis Config Generation using Github Action

[angeloskaltsikis]

As you may have already faced, you need to run terragrunt-atlantis-config tool as soon as you have added/changed/deleted something from your IaC as you will most likely need to update the atlantis.yaml file with the changes introduced by your change. In order to solve that problem as we couldn't rely on human to remember to run this command we decided to use pre-commit and to be exact we have added a file .pre-commit-config.yaml in our Repo containing the following

repos:
    - repo: local
      hooks:
        -   id: run-terragrunt-atlantis-config-generate
            name: 'terragrunt-atlantis-config generate'
            entry: 'run-terragrunt-atlantis.sh'
            language: 'script'
            always_run: true
            stages: [post-commit]
            description: "Runs terragrunt-atlantis-config generate, requires https://github.com/transcend-io/terragrunt-atlantis-config"

and run-terragrunt-atlantis.sh is

#!/usr/bin/env bash
exec terragrunt-atlantis-config generate --ignore-parent-terragrunt --autoplan --workflow terragrunt --parallel=false --output ./atlantis.yaml

As you can imagine this is not obligatory and it requires each user to install the hook which from then runs automatically except for the case she instructs to SKIP it.

This sounds fine but if you infrastructure is described on more than one IaC repositories it starts getting a bit cumbersome to ask from different users to install the above hooks.

As a result i was thinking that having a Github action which runs on after a push on a branch (which has an open PR), this will generate the atlantis.yaml file and add a commit for it on the branch taking this responsibility from each user. This will be highly portable from repo-to-repo as easy as it is to include a new github action in your repo.

What i don't like with this approach is that it will create several commits for atlantis.yaml and if you don't use the Squash & Merge feature of Github you still depend on users good will to merge those commits onto one and don't create a mess in history.

I am writing this whole proposal here in order to discuss whether you had similar problems/ideas and if you would like to collaborate on that to create something that we can use among different repos with minimal configuration in a yaml file.

[dmattia]

Thanks for starting this discussion!

At Transcend, we use a monorepo with steps similarish to the following in a CircleCI job:

      - checkout
      - run: terragrunt-atlantis-config generate --output atlantis.yaml.expected --autoplan --parallel=false
      - run: diff atlantis.yaml atlantis.yaml.expected
      - slack/status:
              channel: some-channel
              fail_only: true
              mentions: user1,user2
              only_for_branches: master
              webhook: $SLACK_WEBHOOK_URL

Personally, I prefer the tool to just have some status check fail if terragrunt-atlantis-config hasn't been run, as opposed to it pushing commits itself, but I could see supporting either option.

Having a pre-commit option is also something we looked into (we use the same pre-commit library 😄 ), but in our case we have so many modules that the tool can take a few seconds, which can add up with lots of pre-commits in a monorepo. But that is also something I think would be great to document, as for most use cases a pre-commit is an excellent option.

So I think there's a few things to do here:

• [x] Document setting up a pre-commit hook in the README.md
• [x] Create a Github action that fails a status check if atlantis.yaml is not fully up to date
• [ ] Look into adding support for the action to actually push up commits

Does that seem like an okay plan on your end?

[dmattia]

@angeloskaltsikis I just created https://github.com/transcend-io/terragrunt-atlantis-config-github-action, which defines an action that will fail if the atlantis.yaml file is not up to date.

I'll add in some documentation for how to use it in this repo later, but the tldr is:

name: terragrunt-atlantis-config
on: [push]

jobs:
  terragrunt_atlantis_config:
    runs-on: ubuntu-latest
    name: Validate atlantis.yaml
    steps:
      - uses: actions/checkout@v2
      - name: Ensure atlantis.yaml is up to date using terragrunt-atlantis-config
        id: atlantis_validator
        uses: transcend-io/terragrunt-atlantis-config-github-action@v0.0.3
        with:
          version: v0.4.3
          extra_args: '--autoplan --parallel=false'

[angeloskaltsikis]

Sorry for taking long to respond. @dmattia Thats great news 🚀 I also think that was the easiest check and the first one we should have implemented as it gives a very clear check if someone forgot to re-generate the atlantis.yaml file. I have some ideas regarding this github action check and i will write those on its repo.

What makes me mad if we would implement the auto-commit github action is the part that the git history is going to be a mess in our case (we don't use Squash & Merge). I guess a future version of Github actions that auto-commit will support something similar in the future and possibly we can re-evaluate then.

Regarding the pre-commit hooks, i can tell you that we also have a huge number of directories ("modules") in our repo (over 1500+) so it also takes several seconds (around 13ish) to calculate the whole configuration. I have already tested hooks on stages pre-commit (users had to do hackish things on git staging area to commit atlantis config on different commits) & post-commit (which is better however it runs always as it doesn't understand files & also in a rebase it may create several minutes latency as the config generation runs on each commit). I am currently thinking of evaluating the pre-push stage as it seems a better fit to our use case. Doing that, theoretically the atlantis config generation will only run when it should. What do you think? I will be happy to document how you can install the hook as soon as we find a good-enough one.

[angeloskaltsikis]

Hello again, so some days ago i had some free time to try and experiment with the third suggestion but unfortunately this didn't work as i would like. I have added the following configuration for the github action in my repo

name: terragrunt-atlantis-config
on: [push]

jobs:
  terragrunt_atlantis_config:
    runs-on: ubuntu-latest
    name: Validate atlantis.yaml
    steps:
      - uses: actions/checkout@v2
        with:
            ref: ${{ github.head_ref }}
      - name: Ensure atlantis.yaml is up to date using terragrunt-atlantis-config
        id: atlantis_validator
        uses: angeloskaltsikis/terragrunt-atlantis-config-github-action@move-file-after-diff-to-upload
        with:
          version: v0.4.3
          extra_args: '--ignore-parent-terragrunt --autoplan --workflow terragrunt --parallel=false --output ./atlantis.yaml'
      - uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: 'chore: Auto-Update Atlantis config'

and also created a fork of your github action and modified it a bit.

Even if i managed to make a small POC for the auto-commit feature this was very disturbing to the end user as she would have to git fetch from origin the auto-commits all the time which wouldn't create the best user experience. I am not sure whether we can make that more user friendly or we should ditch this concept.

Another idea that i had was that possibly we could extend Atlantis to run a custom step (terragrunt-atlantis-config) just after the checkout of the branch in some repos, and hide this whole complexity from the end user while atlantis will have an updated configuration available without the need of having that in VCS. I think i should note here that i am not sure that this is easy and good solution but i would love to get some feedback on how you find it.

Thanks in advance @dmattia

[dmattia]

This was all wonderful to know, thanks for doing some great research!

I would love to integrate this repo into atlantis someday, but will want to make sure the API stabilizes a bit before proposing that. I am currently in the talks with Luke (Atlantis' maintainer) to try and at least add this repo to Atlantis' docs.

I can't think of a way auto-commits would work right now, so I'll close this issue for the moment as there is some Github Action that now works that I'm using in production at Transcend. I just added some documentation to this repo's README, using the awesome optimization ideas from your issue on the github action repo.

Feel free to re-open if you ever have a thought, and I'll gladly play around with it. Otherwise a separate issue would work fine too, whichever you think makes more sense.

Otherwise the "merge into Atlantis" approach might just be easier in the long run