gruntwork-io / terragrunt

Terragrunt is a flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale.
https://terragrunt.gruntwork.io/
MIT License
8.04k stars 975 forks source link

Avoid dreating duplicate resources with 0.12 update #735

Closed rverma-nikiai closed 5 years ago

rverma-nikiai commented 5 years ago

With the upgrade to 0.19

Current I had a structure like

aws
- terraform.tfvars [1]
- module-a
   - terraform.tfvars[2]

terraform.tfvars/terragrunt.hcl[1] looks like
remote_state {
  backend = "s3"
  config = {
    bucket         = get_env("TF_BUCKET", "")
    key            = "${path_relative_to_include()}/terraform.tfstate}"
    region         = get_env("TF_BUCKET_REGION", "")
    encrypt        = true
    dynamodb_table = get_env("TF_DYNAMODB_TABLE", "")
    profile        = get_env("ASSUME_ROLE", "")
  }
}

terraform {
  extra_arguments "crud" {
    commands = [
      "apply",
      "destroy",
      "plan",
    ]

    arguments = [
      "-lock-timeout=1m",
      "-input=false",
    ]

    env_vars = {
      TF_VAR_aws_assume_role_arn = get_env("TF_VAR_aws_assume_role_arn", "")
      AWS_DEFAULT_REGION         = get_env("TF_VAR_region", "ap-south-1")
      TF_VAR_namespace           = get_env("TF_VAR_namespace", "niki")
    }
  }
}

while terragrunt.tfvars[2] looks like
terraform {
  source = "git::https://gitlab.com/niki-infra/terraform-root-modules//aws/account-dns?ref=0.12"
}

include {
  path = "${find_in_parent_folders()}"
}

When I moved to equivalent terragrunt.hcl terragrunt is asking to create the resources as present in module-a. Ideally it should read the module-a/terraform.tstate file which should result in no diff and thus no resource creation massage.

brikis98 commented 5 years ago

Could you provide more info? i.e., the full log output?

rverma-nikiai commented 5 years ago

root terragrunt.hcl is like

remote_state {
  backend = "s3"
  config = {
    bucket         = get_env("TF_BUCKET", "")
    key            = "${path_relative_to_include()}/terraform.tfstate}"
    region         = get_env("TF_BUCKET_REGION", "")
    encrypt        = true
    dynamodb_table = get_env("TF_DYNAMODB_TABLE", "")
    profile        = get_env("ASSUME_ROLE", "")
  }
}

terraform {
  extra_arguments "crud" {
    commands = [
      "apply",
      "destroy",
      "plan",
    ]

    arguments = [
      "-lock-timeout=1m",
      "-input=false",
    ]

    env_vars = {
      TF_VAR_aws_assume_role_arn = get_env("TF_VAR_aws_assume_role_arn", "")
      AWS_DEFAULT_REGION         = get_env("TF_VAR_region", "ap-south-1")
      TF_VAR_namespace           = get_env("TF_VAR_namespace", "niki")
    }
  }
}

module one is like

terraform {
  source = "git::https://gitlab.com/niki-infra/terraform-root-modules.git//aws/account-dns?ref=0.12"
}

include {
  path = "${find_in_parent_folders()}"
}

result of plan is

terragrunt plan
[terragrunt] [/conf/account-dns] 2019/06/11 00:24:06 Running command: terraform --version
[terragrunt] 2019/06/11 00:24:06 Reading Terragrunt config file at /conf/account-dns/terragrunt.hcl
[terragrunt] 2019/06/11 00:24:06 Terraform files in /conf/account-dns/.terragrunt-cache/woG6qtajCmRGo7rrDUYE8aql1FE/OCXfreOXusBkApFxOWURCvaQL-Q/aws/account-dns are up to date. Will not download again.
[terragrunt] 2019/06/11 00:24:06 Copying files from /conf/account-dns into /conf/account-dns/.terragrunt-cache/woG6qtajCmRGo7rrDUYE8aql1FE/OCXfreOXusBkApFxOWURCvaQL-Q/aws/account-dns
[terragrunt] 2019/06/11 00:24:06 Setting working directory to /conf/account-dns/.terragrunt-cache/woG6qtajCmRGo7rrDUYE8aql1FE/OCXfreOXusBkApFxOWURCvaQL-Q/aws/account-dns
[terragrunt] 2019/06/11 00:24:06 Backend config has changed from map[iam_endpoint: kms_key_id: profile:audit-admin secret_key: skip_get_ec2_platforms:<nil> skip_metadata_api_check:%!s(bool=false) token: dynamodb_endpoint: skip_credentials_validation:%!s(bool=false) acl: bucket:niki-audit-terraform-state shared_credentials_file: external_id: key:account-dns/terraform.tfstate} max_retries:%!s(float64=5) workspace_key_prefix:env: dynamodb_table:niki-audit-terraform-state-lock encrypt:%!s(bool=true) force_path_style:%!s(bool=false) role_arn: skip_region_validation:%!s(bool=false) skip_requesting_account_id:<nil> sts_endpoint: assume_role_policy: endpoint: lock_table:<nil> region:ap-south-1 session_name: access_key:] to map[bucket:niki-audit-terraform-state dynamodb_table:niki-audit-terraform-state-lock encrypt:%!s(bool=true) key:account-dns/terraform.tfstate} profile:audit-admin region:ap-south-1]
[terragrunt] [/conf/account-dns] 2019/06/11 00:24:06 Backend config has changed from map[assume_role_policy: max_retries:%!s(float64=5) sts_endpoint: encrypt:%!s(bool=true) key:account-dns/terraform.tfstate} kms_key_id: profile:audit-admin dynamodb_table:niki-audit-terraform-state-lock force_path_style:%!s(bool=false) region:ap-south-1 shared_credentials_file: skip_get_ec2_platforms:<nil> access_key: acl: endpoint: skip_metadata_api_check:%!s(bool=false) skip_region_validation:%!s(bool=false) skip_requesting_account_id:<nil> secret_key: bucket:niki-audit-terraform-state dynamodb_endpoint: role_arn: token: external_id: iam_endpoint: lock_table:<nil> session_name: skip_credentials_validation:%!s(bool=false) workspace_key_prefix:env:] to map[encrypt:%!s(bool=true) key:account-dns/terraform.tfstate} profile:audit-admin region:ap-south-1 bucket:niki-audit-terraform-state dynamodb_table:niki-audit-terraform-state-lock]
[terragrunt] [/conf/account-dns] 2019/06/11 00:24:06 Initializing remote state for the s3 backend
[terragrunt] [/conf/account-dns] 2019/06/11 00:24:09 Running command: terraform init -backend-config=key=account-dns/terraform.tfstate} -backend-config=profile=audit-admin -backend-config=region=ap-south-1 -backend-config=bucket=niki-audit-terraform-state -backend-config=dynamodb_table=niki-audit-terraform-state-lock -backend-config=encrypt=true

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.14"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[terragrunt] 2019/06/11 00:24:19 Running command: terraform plan -lock-timeout=1m -input=false
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_route53_record.dns_zone_soa will be created
  + resource "aws_route53_record" "dns_zone_soa" {
      + allow_overwrite = (known after apply)
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "audit.niki.ai"
      + records         = (known after apply)
      + ttl             = 60
      + type            = "SOA"
      + zone_id         = (known after apply)
    }

  # aws_route53_zone.dns_zone will be created
  + resource "aws_route53_zone" "dns_zone" {
      + comment       = "Managed by Terraform"
      + force_destroy = false
      + id            = (known after apply)
      + name          = "audit.niki.ai"
      + name_servers  = (known after apply)
      + vpc_id        = (known after apply)
      + vpc_region    = (known after apply)
      + zone_id       = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
rverma-nikiai commented 5 years ago

existing tfstate file account-dns/terraform.tfstate as present in s3

{
    "version": 3,
    "terraform_version": "0.11.14",
    "serial": 13,
    "lineage": "420cdbb3-c93d-f345-b2a3-b5856c443f52",
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {
                "name_servers": {
                    "sensitive": false,
                    "type": "list",
                    "value": [
                        "ns-1282.awsdns-32.org",
                        "ns-1972.awsdns-54.co.uk",
                        "ns-324.awsdns-40.com",
                        "ns-919.awsdns-50.net"
                    ]
                },
                "parent_name_servers": {
                    "sensitive": false,
                    "type": "list",
                    "value": [
                        "ns-1186.awsdns-20.org",
                        "ns-14.awsdns-01.com",
                        "ns-1588.awsdns-06.co.uk",
                        "ns-943.awsdns-53.net"
                    ]
                },
                "parent_zone_id": {
                    "sensitive": false,
                    "type": "string",
                    "value": "Z25XNJAG95VW9V"
                },
                "zone_id": {
                    "sensitive": false,
                    "type": "string",
                    "value": "Z6H02QBLQUK3S"
                }
            },
            "resources": {
                "aws_route53_record.dns_zone_soa": {
                    "type": "aws_route53_record",
                    "depends_on": [
                        "aws_route53_zone.dns_zone"
                    ],
                    "primary": {
                        "id": "Z6H02QBLQUK3S_audit.niki.ai._SOA",
                        "attributes": {
                            "allow_overwrite": "true",
                            "fqdn": "audit.niki.ai",
                            "health_check_id": "",
                            "id": "Z6H02QBLQUK3S_audit.niki.ai._SOA",
                            "name": "audit.niki.ai",
                            "records.#": "1",
                            "records.578560122": "ns-1282.awsdns-32.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400",
                            "set_identifier": "",
                            "ttl": "60",
                            "type": "SOA",
                            "zone_id": "Z6H02QBLQUK3S"
                        },
                        "meta": {
                            "schema_version": "2"
                        },
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                },
                "aws_route53_record.parent_dns_zone_soa": {
                    "type": "aws_route53_record",
                    "depends_on": [
                        "aws_route53_zone.parent_dns_zone"
                    ],
                    "primary": {
                        "id": "Z25XNJAG95VW9V_niki.ai_SOA",
                        "attributes": {
                            "allow_overwrite": "true",
                            "fqdn": "niki.ai",
                            "health_check_id": "",
                            "id": "Z25XNJAG95VW9V_niki.ai_SOA",
                            "name": "niki.ai",
                            "records.#": "1",
                            "records.1927139077": "ns-1186.awsdns-20.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400",
                            "set_identifier": "",
                            "ttl": "172800",
                            "type": "SOA",
                            "zone_id": "Z25XNJAG95VW9V"
                        },
                        "meta": {
                            "schema_version": "2"
                        },
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                },
                "aws_route53_zone.dns_zone": {
                    "type": "aws_route53_zone",
                    "depends_on": [],
                    "primary": {
                        "id": "Z6H02QBLQUK3S",
                        "attributes": {
                            "comment": "Managed by Terraform",
                            "delegation_set_id": "",
                            "force_destroy": "false",
                            "id": "Z6H02QBLQUK3S",
                            "name": "audit.niki.ai.",
                            "name_servers.#": "4",
                            "name_servers.0": "ns-1282.awsdns-32.org",
                            "name_servers.1": "ns-1972.awsdns-54.co.uk",
                            "name_servers.2": "ns-324.awsdns-40.com",
                            "name_servers.3": "ns-919.awsdns-50.net",
                            "tags.%": "0",
                            "vpc.#": "0",
                            "vpc_id": "",
                            "vpc_region": "",
                            "zone_id": "Z6H02QBLQUK3S"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                },
                "aws_route53_zone.parent_dns_zone": {
                    "type": "aws_route53_zone",
                    "depends_on": [],
                    "primary": {
                        "id": "Z25XNJAG95VW9V",
                        "attributes": {
                            "comment": "Parent domain name",
                            "delegation_set_id": "",
                            "force_destroy": "false",
                            "id": "Z25XNJAG95VW9V",
                            "name": "niki.ai.",
                            "name_servers.#": "4",
                            "name_servers.0": "ns-1186.awsdns-20.org",
                            "name_servers.1": "ns-14.awsdns-01.com",
                            "name_servers.2": "ns-1588.awsdns-06.co.uk",
                            "name_servers.3": "ns-943.awsdns-53.net",
                            "tags.%": "0",
                            "vpc.#": "0",
                            "vpc_id": "",
                            "vpc_region": "",
                            "zone_id": "Z25XNJAG95VW9V"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                }
            },
            "depends_on": []
        }
    ]
}
rverma-nikiai commented 5 years ago

found the issue, it was a config issue

key = "${path_relative_to_include()}/terraform.tfstate}"

should be

key = "${path_relative_to_include()}/terraform.tfstate"