Upgrading to Terraform 0.12: separate configuration file for Terragrunt?

apparentlymart commented 6 years ago

Hi! I'm one of the engineers at HashiCorp who works on Terraform Core.

As you might be aware, we've been working for some time now on various improvements to the Terraform configuration language, to address a number of usability problems that have been reported over the years. The new changes are broadly compatible with existing configurations, and particularly those using the common idioms reflected in the Terraform documentation.

Unfortunately resolving some of these problems required tightening up some things that were previously rather loose. Current and previous versions of Terraform ignore certain things that are considered invalid, often producing surprising results, and so to improve usability the new implementation has some more error checking in order to catch mistakes and produce error messages that are hopefully more actionable.

We're planning to include a migration helper tool in the next release in order to help detect and fix some of these rough edges. The details of what this tool does will be largely informed by real-world usage, and so we're planning to release an early preview of the new implementation as soon as it is in a usable state in the hope that we can gather as many examples as possible of patterns in the wild that the migration helper tool should be looking for.

With all of that context in mind, we know that there are lots of users wrapping Terraform with Terragrunt, and so we want to make sure there's a good upgrade path. I'm opening this issue to discuss a specific problem I've noticed, but I'd like to discuss any other potential hurdles too, in whatever forum makes sense.

Terragrunt currently uses an extension of the terraform.tfvars file as its own configuration file, relying on the fact that Terraform itself ignores definitions in that file that are not defined as variables, and then using a special variable name terragrunt whose value is not, by Terraform's rules, valid: it includes mixed-typed nested structures, has nested blocks, etc.

The new implementation has a few differences that will cause friction here if we proceed naively:

Terraform will consider the mixed-type nested block structures as invalid and will generate error messages about them during validation, since they are invalid by Terraform's rules.
The new configuration parser distinguishes more strongly between object expressions and nested blocks, in order to resolve various confusing behaviors that resulted from the ambiguity of that in the original HCL design.
The expression evaluator and the structural language were previously entirely separate (HIL vs. HCL), but they are combined in the new implementation, which means that interpolation sequences ${ are interpreted in all contexts, with the difference now being whether variables and functions are available, rather than whether interpolations are possible at all.

Looking at the "live infrastructure" example, a naive syntax-level pass of migration would produce the following rewritten version:

# Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules,
# remote state, and locking: https://github.com/gruntwork-io/terragrunt
terragrunt = {
  # Configure Terragrunt to automatically store tfstate files in an S3 bucket
  remote_state = {
    backend = "s3"
    config = {
      encrypt        = true
      bucket         = "terragrunt-example-prod-terraform-state"
      key            = "$${path_relative_to_include()}/terraform.tfstate"
      region         = "us-west-2"
      dynamodb_table = "terraform-locks"
    }
  }
  # Configure root level variables that all resources can inherit
  terraform = {
    extra_arguments = {
      # (this part will need to do some heuristic processing to decide whether to use
      # direct objects or lists of objects, mimicking the behavior of the current hcl.Decode
      # into an interface{}, so the details may not exactly match this in practice.)
      bucket = {
        commands = ["$${get_terraform_commands_that_need_vars()}"]
        optional_var_files = [
          "$${get_tfvars_dir()}/$${find_in_parent_folders("account.tfvars", "ignore")}"
        ]
      }
    }
  }
}

Since Terragrunt is using the HCL parser directly to process its configuration file today, it has the freedom to interpret the tree of HCL AST objects inside here as desired, and so I expect there are expectations here that are different than how Terraform generally deals with HCL structures it has no specific schema for. Also, due to the fact that interpolation syntax is now accepted in all quoted strings (variables and functions are just not available in some contexts), the naive migration tool will escape the ${ sequences under the assumption that the user intended them to be treated literally, since that was Terraform's own interpretation of a .tfvars file before this change.

Therefore it feels to me like the most practical path to take here is for our migration tool to have a special case for a variable called terragrunt (possibly activated only if there is no variable "terragrunt" block in configuration) which can then have some light understanding of Terragrunt's configuration structure and make some allowances for it so that we can avoid unintentional breakage of the Terragrunt configuration.

Terragrunt is currently relying on syntax features that are considered invalid in the new implementation, so leaving the configuration untouched by the migration tool would inevitably lead to syntax errors during parsing.

In order to avoid conflicts between Terraform's and Terragrunt's interpretation of the same file, I'd like to propose having Terragrunt look for a Terragrunt-specific configuration filename (e.g. terragrunt.conf) in preference to terraform.tfvars, and then Terraform would entirely ignore this file and leave Terragrunt's own parser to deal with it. Terragrunt could continue to use the original implementation of HCL, avoiding the need to deal with any of the new processing rules from the new implementation.

If we were to go down this path, we could build the configuration migration tool so that its special case for the terragrunt variable name is to migrate the contents out into a terragrunt.conf file that retains the block structure and original-HCL syntax that Terragrunt is expecting, and then deletes this value from the terraform.tfvars file altogether so that Terraform itself won't try to parse it.

I understand that this is a pretty drastic shift in the design and usage of Terragrunt, so I wanted to start talking about this early so we have some time to think through the implications and any other alternative approaches. I hope that together we can find a reasonable path forward!

brikis98 commented 5 years ago

Alright, #731 is merged. Please give https://github.com/gruntwork-io/terragrunt/releases/tag/v0.19.0 a try (the new binaries should show up in a few minutes)!

ozbillwang commented 5 years ago

@brikis98

Thanks to make terragrunt ready for terraform 0.12

But I got error:

/apps # terragrunt --version
terragrunt version v0.19.0

/apps # terraform --version
Terraform v0.12.1

/apps # terragrunt init
[terragrunt] [/apps] 2019/06/11 06:20:44 Running command: terraform --version
[terragrunt] 2019/06/11 06:20:44 Reading Terragrunt config file at /apps/terragrunt.hcl
[terragrunt] 2019/06/11 06:20:44 Error reading file at path /apps/terragrunt.hcl: open /apps/terragrunt.hcl: no such file or directory
[terragrunt] 2019/06/11 06:20:44 Unable to determine underlying exit code, so Terragrunt will exit with error code 1
/apps #

Update 1

Seems the README and migration guide have been updated:

https://github.com/gruntwork-io/terragrunt#quick-start

https://github.com/gruntwork-io/terragrunt/blob/master/_docs/migration_guides/upgrading_to_terragrunt_0.19.x.md

We need prepare a new terragrunt.hcl file now. Let me have a try

Update 2

A real sample to deploy terraform 0.12 with terragrunt 0.19

https://github.com/gruntwork-io/terragrunt-infrastructure-live-example

Now it makes sense for me.

Really appreciated.

brikis98 commented 5 years ago

@ozbillwang Glad to hear you got it sorted!

ozbillwang commented 5 years ago

@brikis98

And I can confirm now, I run on a complex terraform stack with 7 terraform modules, the part to do 0.12upgrade in terraform takes time to make it work, but the part to run terragrunt after 0.12upgrade, it works perfect.

kamsz commented 5 years ago

@brikis98 While declaring an empty remote state in a way:

terraform {
  # The configuration for this backend will be filled in by Terragrunt
  backend "gcs" {}
}

I'm getting:

Error: Missing key/value separator

  on terraform.tfvars line 2:
   1:
   2:   terraform {

Expected an equals sign ("=") to mark the beginning of the attribute value.

Content of directory terragrunt.hcl:

terraform {
  source = "."

  extra_arguments "vars" {
    commands = [
      "apply",
      "plan",
      "import",
      "push",
      "refresh",
      "destroy",
      "validate",
    ]

    arguments = [
      "-var-file=${get_terragrunt_dir()}/../tfvars/terraform.tfvars",
    ]
  }
}

include {
  path = "${find_in_parent_folders()}"
}

Content of root terragrunt.hcl:

remote_state {
  backend = "gcs"
  config = {
    bucket = "terraform-state"
    prefix = "${path_relative_to_include()}/terraform.tfstate"
  }
}

terragrunt version v0.19.2
Terraform v0.12.2
+ provider.google v2.6.0

Edit: So apparently it tries to parse terragrunt.hcl too, that's why there's an error. After commenting out terraform block from terragrunt.hcl it works.

brikis98 commented 5 years ago

on terraform.tfvars line 2:

You still have a terraform.tfvars file hanging around?

kamsz commented 5 years ago

The only file that's called terraform.tfvars is included here:

"-var-file=${get_terragrunt_dir()}/../tfvars/terraform.tfvars",

brikis98 commented 5 years ago

And does it have a terraform { ... } block starting on line 2?

kamsz commented 5 years ago

No. For some weird reason the error is related to terragrunt.hcl file, but it's showing terraform.tfvars. I've managed to overcome this by moving all *.tf files to subdirectory and point to it in terragrunt.hcl.

brikis98 commented 5 years ago

That error is from Terraform itself, not Terragrunt...

Are you passing terragrunt.hcl as a var file to Terraform somewhere? Check all your extra_arguments declarations.

kamsz commented 5 years ago

Nevermind, I had leftover environment variables set... Excuse my stupidity.

brikis98 commented 5 years ago

No prob! Thx for following up.