[Feature request] Dynamically generate repo atlantis.yaml config

[mechastorm]

I am looking at way to better maintain our atlantis.yml on monolith repos. In some repositories I encounter, we have >500 terraform directories. Each directory is defined as a unique project stanza. This is because each directory needs to be assigned a predefined workflow. Example a TF directory at terraform/myservice/prod should use the production workflow. All of this creates an atlantis.yml that is >5000 lines at times. And it will only grow larger.

I am working on a script to automatically generate a new atlantis.yml config each time a new directory is detected / added. But the workflow to integrate this generated atlantis.yml in version control could get complicated.

The one idea is to allow Atlantis run arbitary command/script before it reads the atlantis.yml. I think that would be the best to insert any automation script needed by Atlantis including generating an update atlantis.yml file.

I am also open to any other suggetions on how we can better maintain our atlantis.yaml for a large monolith repo.

[lkysow]

I think something like buildkite has (https://buildkite.com/docs/pipelines/defining-steps#dynamic-pipelines) where you can have a .atlantis/config.sh or something that generates your atlantis.yaml (which is then immediately parsed) might be an easy fix to this.

As written above, it's not at all secure, but then it's just as secure as an atlantis.yaml (at the moment). I think there's some mitigations we could have once the server-side config lands.

[mechastorm]

I think doing the .atlantis hook directory would be good as its a generic entrypoint for other items. It would be good if we could make it flexible and avoid hardcoding .atlantis/config.sh.

This feature will definitely open up to a bunch of security concerns, so we will have to balance that together with the upcoming server-side configs.

[keepclean]

I am also open to any other suggestions on how we can better maintain our atlantis.yaml for a large monolith repo.

I was trying to approach this with a pre-commit git-hook, but it doesn't scale across the team properly. Might there any other possible intermediate solutions or workarounds?

I am working on a script to automatically generate a new atlantis.yml config each time a new directory is detected / added.

What do you use as a trigger for your script?

---

@lkysow In case of using Atlantis with Gitlab, is it possible somehow to execute own Gitbal CI pipeline before Atlantis will run its own?

[keepclean]

Seems, I was able to find workaround for updating atlantis.yaml at least during auto-plan phase.

My workaround is based on post-checkout git-hook:

• in gitconfig, which is deployed with my Atlantis installation, I added another option hooksPath = /usr/local/bin/
• into my docker image, which is based on runatlantis/atlantis, I added a simple go program with name post-checkout, that traverses git repo with logic related to my case and writes config structure to atlantis.yaml file.

Not sure that my workaround is valid/legit though. I'll test it more.

[angeloskaltsikis]

Hello team, We are already using this amazing atlantis-config-generator from @dmattia to generate my Atlantis for using it with a custom workflow for Terragrunt. However this should happen on the user side which adds some unwanted complexity (users have to install pre-commit hooks and run them after some events). As far as i am concerned it would be totally valid if we could move this complexity inside Atlantis and generate the configuration that it needs when it receives a new commit for a PR. I am currently trying to use the @keepclean hack to hide the above complexity from our users but i would be totally up for a more elegant and proper solution. I wonder if anyone else experienced the same issue?

[lkysow]

I think the direction I'd like to go here is a server-side config that lets you specify a script to run for each repo id that will generate the repo-level atlantis.yaml that then gets parsed.

repos:
- id: your-repo
   repo_config:
   - run: ./my-script
   - run: |
        my bash script inline

[angeloskaltsikis]

I managed to proceed with a solution with post-checkout hooks on the Atlantis side for now (thanks @keepclean for your amazing hackish idea 😄 ). It seems to work as it should for now but i totally agree with @lkysow that defining that in the server-side config seems the best approach here.

PS. I will supply the solution i did in order to help others till the above functionality is properly inserted in Atlantis.

[angeloskaltsikis]

Hello friends, As promised (sorry for the very late response), my team and i have gathered all the information needed in order to make the dynamic generation of the atlantis.yaml for each repository. We have posted a small medium post in order to help anyone solve this and some other problems.

I just found out that @msarvar have created a proper PR to add such functionality but i believe that till then you can take advantage of the workaround method that we are currently using.

cc. @dmattia @mechastorm P.S. Sorry again for the late reply.

[tsunamishaun]

@angeloskaltsikis I found your post extremely helpful in getting multi-account terragrunt working with atlantis. I have switched to using @msarvar 's pre workflow hook though it doesn't run when atlantis plan is issued on the PR after being updated with new code (in my case from another folder/account). I think your post-checkout hook is still useful (for me at least) since it executes every time.

[Jeroe37]

Orqa 447 add server configuration to run prehooks.

[nitrocode]

Isn't this already supported? At the moment, you can generate the atlantis.yaml from a pre-workflow-run and atlantis will read it on the fly before determining which projects to plan.

Please correct me if I'm mistaken and this can be reopened.

[aairey]

Do you mean this: https://www.runatlantis.io/docs/pre-workflow-hooks.html#usage

That script is not included with Atlantis itself AFAIK. As far as I understood, the goal of this issue is to integrate a way to generate the config like with atlantis-generate-config, which we indeed run as a pre-workflow hook.

Example:

    pre_workflow_hooks:
      - run: >
          terragrunt-atlantis-config generate --output ./atlantis.yaml
          --autoplan --automerge --parallel --create-workspace

[marcportabellaclotet-mt]

As noted by @nitrocode, this feature is already available in the pre-workflow-run stage.

To address our requirements, we've developed an atlantis-yaml-generator tool designed for GitHub projects.

This tool seamlessly integrates into the pre-workflow-run stage and dynamically creates the atlantis.yaml file.

It specifically generates projects that are pertinent to the changes made in the pull request. In cases of monorepos, when your PR impacts just one project, there's no need for Atlantis to run on all the projects.

You can also exclude some project via regex filtering.

Thanks again to all atlantis contributors for this project!!

Sample usage:

    pre_workflow_hooks:
      - run: >
          atlantis-yaml-generator  -w single-workspace --pattern-detector main.tf           

Files changed in the PR:

project/one/main.tf
project/two/main.tf

It will render this atlantis.yaml file:

version: 3
automerge: true
parallel_apply: true
parallel_plan: true
projects:
    - name: project-one
      workspace: default
      workflow: single-workspace
      dir: project/one
      autoplan:
        enabled: true
        when_modified:
            - '**/*.tf'
            - '**/*.tfvars'
            - '**/*.json'
            - '**/*.tpl'
            - '**/*.tmpl'
            - '**/*.xml'
    - name: project-two
      workspace: default
      workflow: single-workspace
      dir: project/two
      autoplan:
        enabled: true
        when_modified:
            - '**/*.tf'
            - '**/*.tfvars'
            - '**/*.json'
            - '**/*.tpl'
            - '**/*.tmpl'
            - '**/*.xml'