plan-all fails if root has a parent terraform.tfvars

[dpetzel]

This feels like a bug to me, but I admit maybe its simply not a supported configuration, but figured I'd open this for clarification. I have this tree:

.
├── dev
│   ├── thing1
│   │   ├── backend.tf
│   │   └── terraform.tfvars
│   ├── thing2
│   │   ├── backend.tf
│   │   └── terraform.tfvars
│   └── terraform.tfvars

The parrent terraform.tfvars is only setting remote state configuration:

terragrunt = {
  remote_state {
    backend = "s3"
    config {
      bucket  = "mine"
      key     = "state/${path_relative_to_include()}/terraform.tfstate"
      region  = "us-east-1"
      encrypt = true
    }
  }
}

Children are like this:

terragrunt = {
  include {
    path = "${find_in_parent_folders()}"
  }
  terraform {
    source = "blah"
  }
}

When I run a terragrunt plan-all while sitting in the dev folder I get the following exception:

[terragrunt] [/my_path] 2017/05/03 16:37:28 Module /my_path has finished successfully!
[terragrunt] 2017/05/03 16:37:28 configstack.MultiError Encountered the following errors:
exit status 1
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:142 (0x4a924c)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:126 (0x4a903f)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/configstack/running_module.go:59 (0x4a86c9)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/configstack/stack.go:40 (0x4aa9c9)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:276 (0x45d3ed)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:215 (0x45c94c)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:169 (0x45c374)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:152 (0x45c097)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:485 (0x488e34)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:259 (0x486c4f)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/main.go:20 (0x4010c4)
/usr/local/go/src/runtime/proc.go:183 (0x42a864)
    main: }
/usr/local/go/src/runtime/asm_amd64.s:2086 (0x458e51)
    goexit: MOVQ DI, (R8)

Looking through the logs I think the crux of the issue is coming from this:

[terragrunt] [my_path] 2017/05/03 16:36:56 Running command: terraform init -backend-config=bucket=mine -backend-config=key=state/./terraform.tfstate -backend-config=region=us-east-1 -backend-config=encrypt=true

Specifically this snippet here: state/./terraform.tfstate -backend-config with the the .

If I create an intermediate directory, like the following, and run the plan-all from inside the stuff directory it works properly

.
├── dev
│   ├── stuff
│   │   ├── thing1
│   │   │   ├── backend.tf
│   │   │   └── terraform.tfvars
│   │   └── thing2
│   │       ├── backend.tf
│   │       └── terraform.tfvars
│   └── terraform.tfvars

I totally understand the reason I get the . is stemming from my usage path_relative_to_include, but I'm not how else I might go about structuring this (aside from the extra dir level).

I'd love to have the original tree style working as it lines up with some of the other examples in the README.

Thanks

[zot24]

The same thing happened to me check my comment on https://github.com/gruntwork-io/terragrunt/issues/174#issuecomment-296738987 it's because Terragrunt check for terraform.tfvars on the directory you are and exec terraform in there and subfolders as I comment on the previous link.

I ended doing what you mention an intermediate folder as I explain here https://github.com/gruntwork-io/terragrunt/issues/174#issuecomment-296820584

Hope that helps it's good to know I'm not the only one misunderstanding the folder structure

[dpetzel]

I'm no where near confident enough with go, or terragrunt to offer this up as a PR, but I did managed to get this working with the following diff. If something like this was the right approach, I'm guessing it would need to be applied to the *-all cli commands.

diff --git a/cli/cli_app.go b/cli/cli_app.go
index ca0e570..a1123fb 100644
--- a/cli/cli_app.go
+++ b/cli/cli_app.go
@@ -271,6 +271,18 @@ func planAll(terragruntOptions *options.TerragruntOptions) error {
        if err != nil {
                return err
        }
+       i := 0
+       for _, module := range stack.Modules {
+               if module.Path == terragruntOptions.WorkingDir {
+                       stack.Modules = append(stack.Modules[:i], stack.Modules[i+1:]...)
+                       terragruntOptions.Logger.Printf("Ignoring %s from list of modules in stack", module.Path)
+               } else {
+                       i++
+               }
+
+       }
+       stack.Modules = stack.Modules[:i]
+       fmt.Println(stack.Modules)

        terragruntOptions.Logger.Printf("%s", stack.String())
        return stack.Plan(terragruntOptions)

[brikis98]

Thanks for reporting. Definitely a bug. I think the reason we haven't seen it is that we do have an "intermediate" folder between the parent .tfvars file and the children. Usually that folder is the name of the environment (e.g. root -> prod -> child and we run plan-all in the prod folder).

The *-all commands should probably ignore folders that only contain a .tfvars file, but no .tf files and no source setting in the .tfvars file. PRs to fix this are welcome! The fix probably needs to happen in this method.

[dpetzel]

I'd be interested in taking a crack at the PR, but I'm a go noob.. Looking at that method, what would that method return when the ignore condition is met?

[brikis98]

It would probably log something and return nil, nil.

[dpetzel]

The *-all commands should probably ignore folders that only contain a .tfvars file, but no .tf files and no source setting in the .tfvars file.

So I hacked up this logic (I think), but there are a number of test failures (one example is below) In the fixture-modules many of these have no source, so they seem to incorrectly match the criteria?

As an example module-a:

• contains only a terraform.tfvars
• Has no .tf
• Has no source setting in the .tfvars

  
  test/fixture-modules/module-a
  └── terraform.tfvars

cat test/fixture-modules/module-a/terraform.tfvars terragrunt = {

Intentionally empty

}

Test Failure Output:

[terragrunt] 2017/05/04 11:15:56 terragrunt/test/fixture-modules/module-a has no source, ignoring [terragrunt] 2017/05/04 11:15:56 terragrunt/test/fixture-modules/module-b/module-b-child has no source, ignoring [terragrunt] 2017/05/04 11:15:56 terragrunt/test/fixture-modules/module-c has no source, ignoring [terragrunt] 2017/05/04 11:15:56 terragrunt/test/fixture-modules/module-d has no source, ignoring --- FAIL: TestResolveTerraformModulesMultipleModulesWithDependencies (0.00s) assertions.go:226: ^M ^M Error Trace: test_helpers.go:30 ^M ^M module_test.go:129 ^M Error: Not equal: ^M expected: 4 ^M received: 0

[dpetzel]

The following seems to work, and doesn't fail any tests. Thoughts?

diff --git a/configstack/stack.go b/configstack/stack.go
index 1e9f28a..2388d4d 100644
--- a/configstack/stack.go
+++ b/configstack/stack.go
@@ -95,7 +95,17 @@ func FindStackInSubfolders(terragruntOptions *options.TerragruntOptions) (*Stack
        if err != nil {
                return nil, err
        }
-
+       i := 0
+       for _, configFile := range terragruntConfigFiles {
+               if configFile == terragruntOptions.TerragruntConfigPath {
+                       // https://github.com/gruntwork-io/terragrunt/issues/208
+                       terragruntConfigFiles = append(terragruntConfigFiles[:i], terragruntConfigFiles[i+1:]...)
+                       terragruntOptions.Logger.Printf("Ignoring %s from list of modules in stack", configFile)
+               } else {
+                       i++
+               }
+       }
+       terragruntConfigFiles = terragruntConfigFiles[:i]
        return createStackForTerragruntConfigPaths(terragruntOptions.WorkingDir, terragruntConfigFiles, terragruntOptions)
 }

[brikis98]

Not sure I understand the logic. You're taking the first i config files, where i is the number of files not equal to terragruntOptions.TerragruntConfigPath? Not sure I understand the reasoning, and that seems like it would only eliminate the last file at best, and only work if that last file just happened to be the root one...

[dpetzel]

As I mentioned before, I'm a total GO noob, so I can't really explain it, but this was the approach that show up multiple times when I searched for "removing an item from a slice". I think if it were not for the if configFile == terragruntOptions.TerragruntConfigPath you'd be exactly right and it would remove the last.

Technically it is removing the last, but only when the condition is met. So at the time the condition is met, the one we want gone is in fact always last.

[dpetzel]

I realized today I'm still running on the custom build I mentioned above. Wondering what your thoughts are on taking this change, or if you think I should try and tackle it another way?

[JohnEmhoff]

Any updates on this front? Would be great to get a fix in.

[brikis98]

No updates at the moment. We're a bit overloaded at the moment, and given that there is a workaround, we have not been able to prioritize this one. PRs are very welcome though!

[okgolove]

It seems it still doesn't work. I have to create an extra directory.