search

f5devcentral / terraform-provider-bigip-old

This Repo is Deprecated please refer to https://github.com/terraform-providers/terraform-provider-bigip Provider is Published !
Mozilla Public License 2.0
32 stars 15 forks source link

vlan and selfip resources failed to delete in BigIP, however provider reports success #110

Closed dannyk81 closed 6 years ago

dannyk81 commented 6 years ago

@scshitole

Found this issue today, looks like a serious issue in error detection.

Terraform tried to delete vlan and selfip resources however the operations failed in bigip (due to dependencies, this was expected to fail indeed), here's the bigip Rest Audit log:

[I][1428][20 Jul 2018 14:50:23 UTC][ForwarderPassThroughWorker] {"user":"local/tfuser","method":"DELETE","uri":"http://localhost:8100/mgmt/tm/net/self/~DEV_SRE_DMZ~dmz_selfip","status":400,"from":"x.x.x.x"}
[I][1429][20 Jul 2018 14:50:23 UTC][ForwarderPassThroughWorker] {"user":"local/tfuser","method":"DELETE","uri":"http://localhost:8100/mgmt/tm/net/self/~DEV_SRE_APP~app_selfip","status":400,"from":"x.x.x.x"}
[I][1430][20 Jul 2018 14:50:23 UTC][ForwarderPassThroughWorker] {"user":"local/tfuser","method":"DELETE","uri":"http://localhost:8100/mgmt/tm/net/vlan/~DEV_SRE_APP~app_vlan","status":400,"from":"x.x.x.x"}
[I][1431][20 Jul 2018 14:50:23 UTC][ForwarderPassThroughWorker] {"user":"local/tfuser","method":"DELETE","uri":"http://localhost:8100/mgmt/tm/net/vlan/~DEV_SRE_DMZ~dmz_vlan","status":400,"from":"x.x.x.x"}

However, provider returned success and the resource were removed from state, although they are in fact still configured in bigip:

bigip_net_selfip.dmz_selfip: Destruction complete after 0s
bigip_net_selfip.app_selfip: Destruction complete after 0s
bigip_net_vlan.app_vlan: Destruction complete after 0s
bigip_net_vlan.dmz_vlan: Destruction complete after 0s
scshitole commented 6 years ago

@dannyk81 is it possible to share the .tf file I will try to reproduce it

dannyk81 commented 6 years ago

@scshitole sure, sent by email.

Not sure you'll be able to use it as-is though.

But you should be able to reproduce this setup easily:

1) setup a vlan and/or selfip resource(s) 2) define a route using the above 3) try to delete the vlan or selfip, F5 should fail this but the provider will say it's destroyed

scshitole commented 6 years ago

@dannyk81 not able to reproduce somehow it works for me 

cat master.tf
provider "bigip" {
  address = "10.192.74.73"
  username = "admin"
  password = "admin"
}

resource "bigip_net_vlan" "vlan1" {
    name = "/Common/internal"
    tag = 101
    interfaces = {
        vlanport = 1.2,
        tagged = false
    }   

}

resource "bigip_net_vlan" "vlan2" {
        name = "/Common/external"
        tag = 102
        interfaces = {
                vlanport = 1.1,
                tagged = false
        }

}

resource "bigip_net_selfip" "selfip1" {
    name = "/Common/internalselfIP"
    ip = "11.1.1.1/24"
    vlan = "/Common/internal"
    depends_on = ["bigip_net_vlan.vlan1"]
    }

resource "bigip_net_selfip" "selfip2" {
        name = "/Common/externalselfIP"
        ip = "100.1.1.1/24"
        vlan = "/Common/external"
        depends_on = ["bigip_net_vlan.vlan2"]
        }

resource "bigip_net_route" "route2" {
  name = "sanjay-route2"
  network = "10.10.10.0/24"
  gw      = "11.1.1.2"
  depends_on = ["bigip_net_selfip.selfip1"]
}

SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform apply -auto-approve
bigip_net_vlan.vlan2: Creating...
  interfaces.#:          "0" => "1"
  interfaces.0.tagged:   "" => "false"
  interfaces.0.vlanport: "" => "1.1"
  name:                  "" => "/Common/external"
  tag:                   "" => "102"
bigip_net_vlan.vlan1: Creating...
  interfaces.#:          "0" => "1"
  interfaces.0.tagged:   "" => "false"
  interfaces.0.vlanport: "" => "1.2"
  name:                  "" => "/Common/internal"
  tag:                   "" => "101"
bigip_net_vlan.vlan2: Creation complete after 0s (ID: /Common/external)
bigip_net_vlan.vlan1: Creation complete after 0s (ID: /Common/internal)
bigip_net_selfip.selfip2: Creating...
  ip:            "" => "100.1.1.1/24"
  name:          "" => "/Common/externalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/external"
bigip_net_selfip.selfip1: Creating...
  ip:            "" => "11.1.1.1/24"
  name:          "" => "/Common/internalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/internal"
bigip_net_selfip.selfip2: Creation complete after 0s (ID: /Common/externalselfIP)
bigip_net_selfip.selfip1: Creation complete after 0s (ID: /Common/internalselfIP)
bigip_net_route.route2: Creating...
  gw:      "" => "11.1.1.2"
  name:    "" => "sanjay-route2"
  network: "" => "10.10.10.0/24"
bigip_net_route.route2: Creation complete after 0s (ID: sanjay-route2)

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform destroy -target=bigip_net_selfip.selfip1
bigip_net_vlan.vlan1: Refreshing state... (ID: /Common/internal)
bigip_net_selfip.selfip1: Refreshing state... (ID: /Common/internalselfIP)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - bigip_net_route.route2

  - bigip_net_selfip.selfip1

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

bigip_net_route.route2: Destroying... (ID: sanjay-route2)
bigip_net_route.route2: Destruction complete after 0s
bigip_net_selfip.selfip1: Destroying... (ID: /Common/internalselfIP)
bigip_net_selfip.selfip1: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform apply -auto-approve
bigip_net_vlan.vlan2: Refreshing state... (ID: /Common/external)
bigip_net_vlan.vlan1: Refreshing state... (ID: /Common/internal)
bigip_net_selfip.selfip2: Refreshing state... (ID: /Common/externalselfIP)
bigip_net_selfip.selfip1: Creating...
  ip:            "" => "11.1.1.1/24"
  name:          "" => "/Common/internalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/internal"
bigip_net_selfip.selfip1: Creation complete after 0s (ID: /Common/internalselfIP)
bigip_net_route.route2: Creating...
  gw:      "" => "11.1.1.2"
  name:    "" => "sanjay-route2"
  network: "" => "10.10.10.0/24"
bigip_net_route.route2: Creation complete after 0s (ID: sanjay-route2)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform destroy -target=bigip_net_selfip.selfip2
bigip_net_vlan.vlan2: Refreshing state... (ID: /Common/external)
bigip_net_selfip.selfip2: Refreshing state... (ID: /Common/externalselfIP)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - bigip_net_selfip.selfip2

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

bigip_net_selfip.selfip2: Destroying... (ID: /Common/externalselfIP)
bigip_net_selfip.selfip2: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform apply -auto-approve
bigip_net_vlan.vlan2: Refreshing state... (ID: /Common/external)
bigip_net_vlan.vlan1: Refreshing state... (ID: /Common/internal)
bigip_net_selfip.selfip1: Refreshing state... (ID: /Common/internalselfIP)
bigip_net_route.route2: Refreshing state... (ID: sanjay-route2)
bigip_net_selfip.selfip2: Creating...
  ip:            "" => "100.1.1.1/24"
  name:          "" => "/Common/externalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/external"
bigip_net_selfip.selfip2: Creation complete after 1s (ID: /Common/externalselfIP)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform destroy -target=bigip_net_vlan.vlan1
bigip_net_vlan.vlan1: Refreshing state... (ID: /Common/internal)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - bigip_net_route.route2

  - bigip_net_selfip.selfip1

  - bigip_net_vlan.vlan1

Plan: 0 to add, 0 to change, 3 to destroy.

Do you really want to destroy?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

bigip_net_route.route2: Destroying... (ID: sanjay-route2)
bigip_net_route.route2: Destruction complete after 0s
bigip_net_selfip.selfip1: Destroying... (ID: /Common/internalselfIP)
bigip_net_selfip.selfip1: Destruction complete after 1s
bigip_net_vlan.vlan1: Destroying... (ID: /Common/internal)
bigip_net_vlan.vlan1: Destruction complete after 0s

Destroy complete! Resources: 3 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform destroy -target=bigip_net_vlan.vlan2
bigip_net_vlan.vlan2: Refreshing state... (ID: /Common/external)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - bigip_net_selfip.selfip2

  - bigip_net_vlan.vlan2

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

bigip_net_selfip.selfip2: Destroying... (ID: /Common/externalselfIP)
bigip_net_selfip.selfip2: Destruction complete after 0s
bigip_net_vlan.vlan2: Destroying... (ID: /Common/external)
bigip_net_vlan.vlan2: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ 
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform apply -auto-approve
bigip_net_vlan.vlan2: Creating...
  interfaces.#:          "0" => "1"
  interfaces.0.tagged:   "" => "false"
  interfaces.0.vlanport: "" => "1.1"
  name:                  "" => "/Common/external"
  tag:                   "" => "102"
bigip_net_vlan.vlan1: Creating...
  interfaces.#:          "0" => "1"
  interfaces.0.tagged:   "" => "false"
  interfaces.0.vlanport: "" => "1.2"
  name:                  "" => "/Common/internal"
  tag:                   "" => "101"
bigip_net_vlan.vlan1: Creation complete after 0s (ID: /Common/internal)
bigip_net_selfip.selfip1: Creating...
  ip:            "" => "11.1.1.1/24"
  name:          "" => "/Common/internalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/internal"
bigip_net_vlan.vlan2: Creation complete after 0s (ID: /Common/external)
bigip_net_selfip.selfip2: Creating...
  ip:            "" => "100.1.1.1/24"
  name:          "" => "/Common/externalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/external"
bigip_net_selfip.selfip1: Creation complete after 1s (ID: /Common/internalselfIP)
bigip_net_route.route2: Creating...
  gw:      "" => "11.1.1.2"
  name:    "" => "sanjay-route2"
  network: "" => "10.10.10.0/24"
bigip_net_selfip.selfip2: Creation complete after 1s (ID: /Common/externalselfIP)
bigip_net_route.route2: Creation complete after 0s (ID: sanjay-route2)

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.
SJC-ML-00028512:terraform-provider-bigip shitole$ terraform destroy -force
bigip_net_vlan.vlan2: Refreshing state... (ID: /Common/external)
bigip_net_vlan.vlan1: Refreshing state... (ID: /Common/internal)
bigip_net_selfip.selfip1: Refreshing state... (ID: /Common/internalselfIP)
bigip_net_selfip.selfip2: Refreshing state... (ID: /Common/externalselfIP)
bigip_net_route.route2: Refreshing state... (ID: sanjay-route2)
bigip_net_route.route2: Destroying... (ID: sanjay-route2)
bigip_net_selfip.selfip2: Destroying... (ID: /Common/externalselfIP)
bigip_net_selfip.selfip2: Destruction complete after 0s
bigip_net_vlan.vlan2: Destroying... (ID: /Common/external)
bigip_net_route.route2: Destruction complete after 1s
bigip_net_selfip.selfip1: Destroying... (ID: /Common/internalselfIP)
bigip_net_vlan.vlan2: Destruction complete after 1s
bigip_net_selfip.selfip1: Destruction complete after 0s
bigip_net_vlan.vlan1: Destroying... (ID: /Common/internal)
bigip_net_vlan.vlan1: Destruction complete after 0s

Destroy complete! Resources: 5 destroyed.
scshitole commented 6 years ago

@dannyk81 is it possible to dump the logs to see what is happening

TF_LOG=DEBUG terraform destroy -target=bigip_net_selfip.selfip1 -force
2018/07/20 15:27:35 [INFO] Terraform version: 0.11.7  41e50bd32a8825a84535e353c3674af8ce799161
2018/07/20 15:27:35 [INFO] Go runtime version: go1.10.1
2018/07/20 15:27:35 [INFO] CLI args: []string{"/usr/local/bin/terraform", "destroy", "-target=bigip_net_selfip.selfip1", "-force"}
2018/07/20 15:27:35 [DEBUG] Attempting to open CLI config file: /Users/shitole/.terraformrc
2018/07/20 15:27:35 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/07/20 15:27:35 [INFO] CLI command args: []string{"destroy", "-target=bigip_net_selfip.selfip1", "-force"}
2018/07/20 15:27:35 [INFO] command: empty terraform config, returning nil
2018/07/20 15:27:35 [DEBUG] command: no data state file found for backend config
2018/07/20 15:27:35 [DEBUG] New state was assigned lineage "9b3c979f-f348-8746-dd2f-67f8080acbbe"
dannyk81 commented 6 years ago

@scshitole could you remove the depends_on attribute from the bigip_net_route resource and try again to destroy vlan or selfip?

In this scenario, I expect the provider to fail to destroy since the dependent resources are still in place in F5.

EDIT: Just to clarify, in your example you've implicitly declared the dependency between the resources (using depends_on attribute), so Terraform will delete the dependent resource first (route) and then delete the vlan and selfip.

However, this is not the issue - in case the dependency was not implicitly declared and you attempt to destroy the vlan and/or selfip (while the route resource is still in place), F5 returns an error code 400 - however the provider treats this situation as though the operation succeeded and removes the resource from the state.

You test-case above doesn't actually test this scenario.

dannyk81 commented 6 years ago

@scshitole were you able to test this without the depends_on attribute?

scshitole commented 6 years ago

@dannyk81 depends_on is required else I cannot apply the configuration as the Self_IP is not created, for the route to exists selfIP has to be present

bigip_net_selfip.selfip1: Creating...
  ip:            "" => "11.1.1.1/24"
  name:          "" => "/Common/internalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/internal"
bigip_net_selfip.selfip2: Creating...
  ip:            "" => "100.1.1.1/24"
  name:          "" => "/Common/externalselfIP"
  traffic_group: "" => "traffic-group-local-only"
  vlan:          "" => "/Common/external"
bigip_net_selfip.selfip2: Creation complete after 1s (ID: /Common/externalselfIP)
bigip_net_selfip.selfip1: Creation complete after 1s (ID: /Common/internalselfIP)

Error: Error applying plan:

1 error(s) occurred:

* bigip_net_route.route2: 1 error(s) occurred:

* bigip_net_route.route2: 01070330:3: Static route gateway 11.1.1.2 is not directly connected via an interface.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
dannyk81 commented 6 years ago

@scshitole Yes, I realize you need the self-ip to configure the route and that depends_on ensure the correct order.

However, this is not the issue here! there's a problem with the error detection when a resource is attempted to be destroyed and F5 returns an error code (400).

To reproduce: 1) create self_ip resource $ terraform apply -target bigip_net_selfip.selfip1 2) create route resource: $ terraform apply -target bigip_net_route.route1 (without depends_on) 3) Destroy self_ip resource: $ terraform destroy -target bigip_net_selfip.selfip1

Step (3) above should succeed and the self_ip will be removed from terraform state, although the self-ip was not removed in F5, and an error code 400 was returned.

scshitole commented 6 years ago

@dannyk81 I am looking into the same

scshitole commented 6 years ago

@dannyk81 fixed this issue please test Terraform will perform the following actions:

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.

Enter a value: yes

bigip_net_route.route2: Creating... gw: "" => "11.1.1.2" name: "" => "sanjay-route2" network: "" => "10.10.10.0/24" bigip_net_route.route2: Creation complete after 0s (ID: sanjay-route2)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed. SJC-ML-00028512:terraform-provider-bigip shitole$ SJC-ML-00028512:terraform-provider-bigip shitole$ SJC-ML-00028512:terraform-provider-bigip shitole$ SJC-ML-00028512:terraform-provider-bigip shitole$ SJC-ML-00028512:terraform-provider-bigip shitole$ SJC-ML-00028512:terraform-provider-bigip shitole$ SJC-ML-00028512:terraform-provider-bigip shitole$ terraform destroy -target bigip_net_selfip.selfip1 bigip_net_vlan.vlan1: Refreshing state... (ID: /Common/internal) bigip_net_selfip.selfip1: Refreshing state... (ID: /Common/internalselfIP)

An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 1 to destroy.

Do you really want to destroy? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm.

Enter a value: yes

bigip_net_selfip.selfip1: Destroying... (ID: /Common/internalselfIP)

Error: Error applying plan:

1 error(s) occurred:

Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure.

SJC-ML-00028512:terraform-provider-bigip shitole$ cat master.tf provider "bigip" { address = "10.192.74.73" username = "admin" password = "admin" }

resource "bigip_net_vlan" "vlan1" { name = "/Common/internal" tag = 101 interfaces = { vlanport = 1.2, tagged = false }

}

resource "bigip_net_vlan" "vlan2" { name = "/Common/external" tag = 102 interfaces = { vlanport = 1.1, tagged = false }

}

resource "bigip_net_selfip" "selfip1" { name = "/Common/internalselfIP" ip = "11.1.1.1/24" vlan = "/Common/internal" depends_on = ["bigip_net_vlan.vlan1"] }

resource "bigip_net_selfip" "selfip2" { name = "/Common/externalselfIP" ip = "100.1.1.1/24" vlan = "/Common/external" depends_on = ["bigip_net_vlan.vlan2"] }

resource "bigip_net_route" "route2" { name = "sanjay-route2" network = "10.10.10.0/24" gw = "11.1.1.2" }

dannyk81 commented 6 years ago

Thanks @scshitole!

Sorry for the delay, was away on vacation.

This looks great, works as expected on my end as well so I'm closing this.