Support incremental builds

[gmaghera]

We are trying to make the generation as fast as possible, by making it specific to the impact of the commit (on the user-side, perhaps via a pre-commit hook).

Generating with the option --filter against the directories changed by the commit is very, very fast, but it does not work with --output atlantis.yaml (where the file atlantis.yaml accurately represented the repository before the last commit). The resulting atlantis.yaml is not a merge of the small generation plus the previous file, it's just the small generation specific to the directories impacted by the commit.

If we had the option of using these two together, atlantis.yaml generation could be extremely fast, regardless of the size of ones Terragrunt mono-repo.

[dmattia]

Very interesting, I hadn't thought of doing this. The primary usage of --filter is for teams that have separate env-dev, env-staging, etc. directories that are managed by separate atlantis servers, so I do believe the existing flags work as I would expect them to (with overriding atlantis.yaml).

That being said, I would be interested to add support for this as a new feature. What would be your thoughts on having something like a --merge-with <filepath> flag that would let you supply in your old atlantis.yaml?

Then on a separate note, I have some questions about performance:

• how many modules are we talking about?
• Do you know what is actually taking a long time to run (maybe you have run_cmd calls or something)?

Our monorepo is about 650 modules large and the 15k line config takes under 1s to generate on CI (though closer to 6 seconds locally)

[gmaghera]

Thanks for entertaining the idea. We have over 3,300 Terragrunt modules and growing. Some could be deprecated probably, but until we do, the generator works on all of them. It takes maybe 16-18 seconds, which is not bad, but not fast enough to make it a pre-commit hook.

We have also considered moving it server-side where Atlantis runs, but there it performs a bit slower due to the Atlantis pod not having as much in resources as users' laptops do. And I'm also bit resistant, wondering if that would make atlantis.yaml more obscure to users, who are supposed to set variables like extra_atlantis_dependencies and atlantis_workflow within their terragrunt.hcl files.

[dmattia]

That's very fair! Also, thank you for the benchmark, that really helps a ton.

I'll see if I can get to this in the next two or so weeks once I finish off one other project here at work

[gmaghera]

In the interest of correctness... I have to correct the figure I left last time. It was from memory and incorrect.

We are now up to 4,718 Terragrunt modules and it takes just a hair under 2 minutes to generate.

[dmattia]

@gmaghera I cannot add you as a reviewer formally, but if you wouldn't mind reviewing https://github.com/transcend-io/terragrunt-atlantis-config/pull/207 and seeing if that would cover your use cases, I would appreciate it