atlantis apply does not apply my atlantis plan'ed projects.

[esn89]

I have projects set up like so:

- name: bastionhost-prod-project
  dir: prod/bastion/
  terraform_version: v0.12.21
  autoplan:
    when_modified:
    - "terragrunt.hcl"
    - "modules/bastionhost/**"
    enabled: false
  apply_requirements: [mergeable, approved]
  workflow: prod-workflow

- name: webappinfra-prod-project
  dir: prod/webapp
  terraform_version: v0.12.21
  autoplan:
    when_modified:
    - "terragrunt.hcl"
    - "modules/webapp/**"
    enabled: false
  apply_requirements: [mergeable, approved]
  workflow: prod-workflow

  ....
  ....

And this repeats for staging workflow and dev workflow.

When I make a change in the when_modified file, and run atlantis plan there are no problems whatsoever. I see a list of all my projects. The problem is when I do an atlantis apply afterwards. I get 0 projects applied:

https://imgur.com/a/GGnfAda

Any ideas? Do I have a wonky config preventing this?

[esn89]

Bump. Any insights?

[msarvar]

It is hard to figure out what is wrong with more context. Looks like you're using a custom workflow, are you doing anything special in it? Did you check if you plan is generating plan files?

[esn89]

@msarvar

Yes, I am doing something special with my custom workflow. Whenever I do a change, multiple projects are planned, so I have many projects which are waiting on an apply. (I have multiple Terraform regions).

I did do a check when I ran a plan, and I made sure to generate a plan file like this:

 - run: terragrunt plan -out=test.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "test.tfplan"

Now, this has been planned for all 5 of my regions and environments (development, staging, production) and the plans all look good. In each of my project directories for atlantis where I ran the atlantis plan:

prod/bastion/ dev/bastion/ stg/bastion/ prod/webapp/ dev/webapp/ stg/webapp/

they all have a dir similar to: .terragrunt-cache/NeFkClsN2EdepIe6dPn_fphls_E/ZglRs72-h59bhO0NDWvIEqgWwVg/test.tfplan

However, once again when I do, atlantis apply, it continues to say Ran Apply for 0 projects:

This is what my apply command looks like:

- run: terragrunt apply "test.tfplan"  --terragrunt-non-interactive -auto-approve

[esn89]

I am also think that atlantis apply and atlantis apply -p $PROJECT commands have very different behaviours. My guess is that atlantis apply just looks for a top-level plan file in the root of the repo and tries to apply that.

Where as atlantis apply -p $PROJECT will cd into the $PROJECT directory and apply from there.

I was hoping that the atlantis apply for multi-project plans and applies would be iterative whereby it goes into each planned $PROJECT and runs an apply there.

[msarvar]

@esn89 I think the issue is due to .terrafagrunt-cache/ folder. Atlantis expects certain folder hierarchy to find the plan files. If terragrunt hijacking it your plans will pass but apply will fail to find the plan files in expected place. Can terragrunt save the plan files in the terraform's root directory?

[esn89]

Hi @msarvar ,

Do you happen to know what the folder hierarchy is?

At the moment, I don't think it can save it in the root directory; what I'm trying to accomplish is a multi-plan and multi-apply of multiple projects.

If I put all of them in the root, it will just overwrite each other and I will end up having only one.

[msarvar]

@esn89 Here is the hierarchy Server flag --data-dir provides root directory for Atlantis server, default is ~/.atantis. When Atlantis clones a repo for a PR it will be located at data-dir/repos/<repo-org>/<repo-name>/<workspace-name>/<pr-num>. And your repo-config provides the location of your terraform roots.

I didn't mean to save it in the repo's root directory, rather in your project directory(terraform root). If you can make terragrunt to save the plan output in the project directory. Try to use this command instead of -out=test.tfplan. For plan: - run: terragrunt plan -no-color -out $PLANFILE For apply: - run: terragrunt apply -no-color $PLANFILE

[msarvar]

.terragrunt-cache folders are explicitly ignored when looking for a tfplans https://github.com/runatlantis/atlantis/pull/517.

[esn89]

@msarvar

You are right, I think it is explicitly ignored.

So I tried to do it with the -out flag like you showed me. Now the problem is that I am able to change the $PLANFILE name, but it still goes back into the .terragrunt-cache/abcdef123/ directory no matter how hard I try.

I looked at the #517 issue, and I was wondering if there is anyway to bring that feature back, to look into the .terragrunt-cache dir.

[esn89]

@msarvar

Update, I managed to get terragrunt plan to output the plan file to my Atlantis $DIR which is great and Atlantis picks up the .tfplan file.

However, now there is another problem which says that there are too many arguments:

[terragrunt] 2021/03/23 17:30:45 Running command: /usr/local/bin/terraform/terraform apply -var credentials=/atlantis-data/repos/prod/bastion/credentials.json -var statebucket=mybucket -var-file /atlantis-data/repos/prod/bastion/common.tfvars -var-file /atlantis-data/repos/prod/bastion/environment.tfvars *test.tfplan* -no-color -auto-approve
Too many command line arguments. Configuration path expected.
[terragrunt] 2021/03/23 17:30:45 Hit multiple errors:
exit status 1

[msarvar]

@esn89 there seem to be couple issues related to that error message in the terragrunt repo https://github.com/gruntwork-io/terragrunt/issues/1271. I would also recommend trying to run that apply command manually from the atlantis volume and try to debug what exactly is causing the issue, maybe it is using regex as tfplan.

[esn89]

@msarvar

I have checked with TG and it has been traced to a bug in TG.