search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.58k stars 9.54k forks source link

terraform bug Sec Group dependency on destroy #1669

Closed stefancocora closed 9 years ago

stefancocora commented 9 years ago

Hi guys ! Amazing work on terraform , please keep it up ( use steroids , whathever it takes ... ) ;) It is absolutely amazing to see this project progressing so fast !

I've downloaded version 0.4.2 and have started using it.

I've encountered a dependency bug.

Short intro about the bug: I have in terraform for the AWS provider:

On create everything proceeds ok and the resources get created. On destroy however terraform doesn't delete the security group before the VPC ending up in an Error state If I delete, manually through the AWS UI the Sec Group, then terraform can finish the destroy operation without issues and will remove the remaining resources ( VPC )

I've tested terraform 0.4.2 and terraform master and both have this bug.

Terraform v0.5.0-dev (8c3fba6d24a89aaa79fe7079726a7f3202d6f08e)

The Sec Group declares a dependency on the vpc_id, terraform should be able to figure out, on destroy, to delete the SG first before the VPC. I've also added

I've added the code in a github repo: https://github.com/stefancocora/tf_sg_deps_bug

Here goes the verbose bug output:

File structure

tree

tree -d
.
├── terraform-module-vpc
└── tf_aws_secgr
    ├── sg_ssh
    └── sg_web
tree
.
├── graph-0.3.7.png
├── graph.png
├── main.tf
├── outputs.tf
├── plan.tf
├── provider.tf
├── terraform-module-vpc
│   ├── main.tf
│   ├── outputs.tf
│   ├── provider.tf
│   ├── README.md
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   ├── terraform.tfvars
│   └── variables.tf
├── terraform.tfstate
├── terraform.tfstate.backup
├── terraform.tfvars
├── tf_aws_secgr
│   ├── sg_ssh
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   ├── README.md
│   │   └── variables.tf
│   └── sg_web
│       ├── main.tf
│       ├── outputs.tf
│       ├── README.md
│       └── variables.tf
├── tf.sh
└── variables.tf

main.tf of the root module

module "vpc" {
  source            = "./terraform-module-vpc"
  account           = "${var.account}"
  domain            = "${var.domain}"
  vpccidr           = "10.208.64.0/19"
  region            = "${var.region}"
  envname           = "${var.envname}"
  secret_key        = "${var.secret_key}"
  access_key        = "${var.access_key}"
}

module "sg_web" {
  source                = "./tf_aws_secgr//sg_web"
  security_group_name   = "${var.sg_web}"
  vpc_id                = "${module.vpc.vpcid}"
  aws_access_key        = "${var.access_key}"
  aws_secret_key        = "${var.secret_key}"
  aws_region            = "${var.region}"
  source_cidr_block     = "${var.source_cidr_block}"
}

relevant section of main.tf of the sec group

resource "aws_security_group" "main_security_group" {
    name = "${var.security_group_name}"
    description = "Security Group ${var.security_group_name}"
    vpc_id = "${var.vpc_id}"

    // allows traffic from the SG itself for tcp
    ingress {
        from_port = 0

Create operation

plan

./tf.sh plan
Refreshing Terraform state prior to plan...

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan.tf

+ module.vpc.aws_internet_gateway.igw
    tags.#:    "0" => "1"
    tags.Name: "" => "eu-west-1-opsvpc-envtest-igw"
    vpc_id:    "" => "${aws_vpc.main.id}"

+ module.vpc.aws_route53_zone.main
    name:    "" => "testops.awseuwest1.example.zone"
    zone_id: "" => "<computed>"

+ module.vpc.aws_route_table.publicnet
    route.#:                                     "" => "1"
    route.~4123265270.cidr_block:                "" => "0.0.0.0/0"
    route.~4123265270.gateway_id:                "" => "${aws_internet_gateway.igw.id}"
    route.~4123265270.instance_id:               "" => ""
    route.~4123265270.vpc_peering_connection_id: "" => ""
    tags.#:                                      "" => "1"
    tags.Name:                                   "" => "eu-west-1-opsvpc-envtest-publicnet"
    vpc_id:                                      "" => "${aws_vpc.main.id}"

+ module.vpc.aws_vpc.main
    cidr_block:                "" => "10.208.64.0/19"
    default_network_acl_id:    "" => "<computed>"
    default_security_group_id: "" => "<computed>"
    enable_dns_hostnames:      "" => "1"
    enable_dns_support:        "" => "1"
    main_route_table_id:       "" => "<computed>"
    tags.#:                    "" => "1"
    tags.Name:                 "" => "eu-west-1-opsvpc-envtest-vpc"

+ module.sg_web.aws_security_group.main_security_group
    description:                          "" => "Security Group sg_web"
    egress.#:                             "" => "<computed>"
    ingress.#:                            "" => "4"
    ingress.1773539029.cidr_blocks.#:     "" => "1"
    ingress.1773539029.cidr_blocks.0:     "" => "10.10.10.0/24"
    ingress.1773539029.from_port:         "" => "443"
    ingress.1773539029.protocol:          "" => "tcp"
    ingress.1773539029.security_groups.#: "" => "0"
    ingress.1773539029.self:              "" => "0"
    ingress.1773539029.to_port:           "" => "443"
    ingress.3472572478.cidr_blocks.#:     "" => "0"
    ingress.3472572478.from_port:         "" => "0"
    ingress.3472572478.protocol:          "" => "udp"
    ingress.3472572478.security_groups.#: "" => "0"
    ingress.3472572478.self:              "" => "1"
    ingress.3472572478.to_port:           "" => "65535"
    ingress.3536822960.cidr_blocks.#:     "" => "1"
    ingress.3536822960.cidr_blocks.0:     "" => "10.10.10.0/24"
    ingress.3536822960.from_port:         "" => "80"
    ingress.3536822960.protocol:          "" => "tcp"
    ingress.3536822960.security_groups.#: "" => "0"
    ingress.3536822960.self:              "" => "0"
    ingress.3536822960.to_port:           "" => "80"
    ingress.3544538468.cidr_blocks.#:     "" => "0"
    ingress.3544538468.from_port:         "" => "0"
    ingress.3544538468.protocol:          "" => "tcp"
    ingress.3544538468.security_groups.#: "" => "0"
    ingress.3544538468.self:              "" => "1"
    ingress.3544538468.to_port:           "" => "65535"
    name:                                 "" => "sg_web"
    owner_id:                             "" => "<computed>"
    tags.#:                               "" => "1"
    tags.Name:                            "" => "HTTP-HTTPS"
    vpc_id:                               "" => "${var.vpc_id}"

apply

./tf.sh apply
module.vpc.aws_route53_zone.main: Creating...
  name:    "" => "testops.awseuwest1.example.zone"
  zone_id: "" => "<computed>"
module.vpc.aws_vpc.main: Creating...
  cidr_block:                "" => "10.208.64.0/19"
  default_network_acl_id:    "" => "<computed>"
  default_security_group_id: "" => "<computed>"
  enable_dns_hostnames:      "" => "1"
  enable_dns_support:        "" => "1"
  main_route_table_id:       "" => "<computed>"
  tags.#:                    "" => "1"
  tags.Name:                 "" => "eu-west-1-opsvpc-envtest-vpc"
module.vpc.aws_vpc.main: Creation complete
module.vpc.aws_internet_gateway.igw: Creating...
  tags.#:    "0" => "1"
  tags.Name: "" => "eu-west-1-opsvpc-envtest-igw"
  vpc_id:    "" => "vpc-72802a17"
module.vpc.aws_internet_gateway.igw: Creation complete
module.vpc.aws_route_table.publicnet: Creating...
  route.#:                                    "" => "1"
  route.3461892135.cidr_block:                "" => "0.0.0.0/0"
  route.3461892135.gateway_id:                "" => "igw-389a515d"
...
snip
...

  vpc_id:                               "" => "vpc-72802a17"
module.sg_web.aws_security_group.main_security_group: Creation complete

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

  account = opsvpc
  region  = eu-west-1

destroy operation that fails

complete destroy operation with a plan step

./tf.sh destroynow
here is the destroy plan that terraform will carry out
Refreshing Terraform state prior to plan...

module.vpc.aws_route53_zone.main: Refreshing state... (ID: Z2DCB9X6JJXOQR)
module.vpc.aws_vpc.main: Refreshing state... (ID: vpc-72802a17)
module.vpc.aws_internet_gateway.igw: Refreshing state... (ID: igw-389a515d)
module.vpc.aws_route_table.publicnet: Refreshing state... (ID: rtb-d837b2bd)
module.sg_web.aws_security_group.main_security_group: Refreshing state... (ID: sg-ecc39089)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan.tf

- module.vpc.aws_internet_gateway.igw

- module.vpc.aws_route53_zone.main

- module.vpc.aws_route_table.publicnet

- module.vpc.aws_vpc.main

- module.sg_web.aws_security_group.main_security_group

destroying infrastructure
aws_route53_zone.main: Destroying...
aws_route_table.publicnet: Destroying...
aws_route_table.publicnet: Destruction complete
aws_internet_gateway.igw: Destroying...
aws_route53_zone.main: Destruction complete
aws_internet_gateway.igw: Destruction complete
aws_vpc.main: Destroying...
aws_vpc.main: Error: 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-72802a17' has dependencies and cannot be deleted.
Error applying plan:

1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-72802a17' has dependencies and cannot be deleted.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

running the destroy operation again shows you the remaining resources

./tf.sh destroynow
here is the destroy plan that terraform will carry out
Refreshing Terraform state prior to plan...

module.vpc.aws_vpc.main: Refreshing state... (ID: vpc-72802a17)
module.sg_web.aws_security_group.main_security_group: Refreshing state... (ID: sg-ecc39089)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan.tf

- module.vpc.aws_vpc.main

- module.sg_web.aws_security_group.main_security_group

destroying infrastructure
aws_vpc.main: Destroying...
aws_vpc.main: Error: 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-72802a17' has dependencies and cannot be deleted.
Error applying plan:

1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* 1 error(s) occurred:

* Error deleting VPC: The vpc 'vpc-72802a17' has dependencies and cannot be deleted.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

User deletes the Sec Group in the AWS UI

./tf.sh destroynow
here is the destroy plan that terraform will carry out
Refreshing Terraform state prior to plan...

module.vpc.aws_vpc.main: Refreshing state... (ID: vpc-72802a17)
module.sg_web.aws_security_group.main_security_group: Refreshing state... (ID: sg-ecc39089)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan.tf

- module.vpc.aws_vpc.main

destroying infrastructure
aws_vpc.main: Destroying...
aws_vpc.main: Destruction complete

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Outputs:

  account = opsvpc
  region  = eu-west-1
stefancocora commented 9 years ago

By parsing the terraform issues in github much more carefully I can see that there are many examples of module-to-module or module-to-resource dependency bugs. I think in general dependency graphs that involve a module are not working at this time in terraform 0.4.2 or in terraform 0.5 master

Here are just a few issues related to dependency resolution either on create or on destroy:

https://github.com/hashicorp/terraform/issues/1472 https://github.com/hashicorp/terraform/issues/1582 https://github.com/hashicorp/terraform/issues/1637

I can also confirm that when I'm writing all resources in the same file , in the same main.tf without any modules whatsoever, then dependency resolution works both for create and destroy operations.

/edit - added more context to the story :)

stefancocora commented 9 years ago

terraform 0.5.0 fixes this issue #1669 as can be seen from the below destroy output. That means that module-to-resource and module-to-module dependency works now so I can start using modules again !

Thank you !

terraform version
Terraform v0.5.0

...

./tf.sh destroynow

here is the destroy plan that terraform will carry out
Refreshing Terraform state prior to plan...

module.vpc.aws_route53_zone.main: Refreshing state... (ID: Z3J1VRBHCBXAKV)
module.vpc.aws_vpc.main: Refreshing state... (ID: vpc-13cd7b76)
module.vpc.aws_internet_gateway.igw: Refreshing state... (ID: igw-56875333)
module.sg_web.aws_security_group.main_security_group: Refreshing state... (ID: sg-63bfe506)
module.vpc.aws_route_table.publicnet: Refreshing state... (ID: rtb-e460e081)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: plan.tf

- module.vpc.aws_internet_gateway.igw

- module.vpc.aws_route53_zone.main

- module.vpc.aws_route_table.publicnet

- module.vpc.aws_vpc.main

- module.sg_web.aws_security_group.main_security_group

destroying infrastructure
aws_route_table.publicnet: Destroying...
aws_route53_zone.main: Destroying...
aws_security_group.main_security_group: Destroying...
aws_route_table.publicnet: Destruction complete
aws_internet_gateway.igw: Destroying...
aws_route53_zone.main: Destruction complete
aws_security_group.main_security_group: Destruction complete
aws_internet_gateway.igw: Destruction complete
aws_vpc.main: Destroying...
aws_vpc.main: Destruction complete

Apply complete! Resources: 0 added, 0 changed, 5 destroyed.

Outputs:

  account = opsvpc
  region  = eu-west-1
ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.