[CLOSED] Terraform crash when creating SSL certificate resources on F5 BIG-IP

RavinderReddyF5 commented 3 years ago

Issue by revog Thursday Mar 12, 2020 at 08:15 GMT Originally opened as https://github.com/terraform-providers/terraform-provider-bigip/issues/255

Terraform Version

Terraform v0.12.19

provider.bigip v1.1.1

Terraform Configuration Files

...
resource "bigip_ssl_certificate" "ecs_s3_crt" {
  name      = "ecs-s3"
  content    = file("../certs/ecs-s3.domain.tld.crt")
  partition  = "/Common"
}

resource "bigip_ssl_key" "ecs_s3_key" {
  name      = "ecs-s3"
  content   = file("../certs/ecs-s3.domain.tld.key")
  partition = "/Common"
}

resource "bigip_ssl_certificate" "swisssign_gold_g22_test" {
  name      = "Server_Gold_G22_2014-chain.pem"
  content   = file("../certs/Server_Gold_G22_2014-chain.pem")
  partition = "/Common"
}
...

Debug Output

bigip_ssl_certificate.swisssign_gold_g22: Creating...
bigip_ssl_key.host_s3_key: Creating...
bigip_ssl_key.host_s3_key: Creation complete after 0s [id=host-s3.key]

Error: rpc error: code = Canceled desc = context canceled

Error: rpc error: code = Unavailable desc = transport is closing

panic: runtime error: invalid memory address or nil pointer dereference
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdc7b3d]
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: goroutine 36 [running]:
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/bigip.resourceBigipSslCertificateRead(0xc000247490, 0x104c380, 0xc00042ecd0, 0xc00017b600, 0xcdc)
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/bigip/resource_bigip_ssl_certificate.go:75 +0x2bd
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/bigip.resourceBigipSslCertificateCreate(0xc000247490, 0x104c380, 0xc00042ecd0, 0x2, 0x1acd380)
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/bigip/resource_bigip_ssl_certificate.go:61 +0x3c3
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Resource).Apply(0xc000424480, 0xc000011ea0, 0xc0004aabe0, 0x104c380, 0xc00042ecd0, 0xee1501, 0xc0000176e8, 0xc00036d560)
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/resource.go:305 +0x363
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema.(*Provider).Apply(0xc000424580, 0xc00011b9f0, 0xc000011ea0, 0xc0004aabe0, 0xc00018a1a8, 0xc00000d470, 0xee3200)
2020-03-12T09:04:54.258+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/helper/schema/provider.go:294 +0x9c
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin.(*GRPCProviderServer).ApplyResourceChange(0xc00000cb78, 0x1215fa0, 0xc00036ce40, 0xc0001762a0, 0xc00000cb78, 0xc000397e90, 0xf04ec0)
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/helper/plugin/grpc_provider.go:885 +0x86a
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0x100dc60, 0xc00000cb78, 0x1215fa0, 0xc00036ce40, 0xc000176240, 0x0, 0x0, 0x0, 0xc000363500, 0x1a56)
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/github.com/hashicorp/terraform-plugin-sdk/internal/tfplugin5/tfplugin5.pb.go:3189 +0x23e
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000b2000, 0x121e000, 0xc00039ed80, 0xc000398f00, 0xc00010e3c0, 0x1aa11a0, 0x0, 0x0, 0x0)
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc/server.go:995 +0x485
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc.(*Server).handleStream(0xc0000b2000, 0x121e000, 0xc00039ed80, 0xc000398f00, 0x0)
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc/server.go:1275 +0xe02
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000034290, 0xc0000b2000, 0x121e000, 0xc00039ed80, 0xc000398f00)
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc/server.go:710 +0x9f
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4: created by github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
2020-03-12T09:04:54.259+0100 [DEBUG] plugin.terraform-provider-bigip_v1.1.1_x4:         /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-bigip/vendor/google.golang.org/grpc/server.go:708 +0xa1
2020-03-12T09:04:54.260+0100 [DEBUG] plugin: plugin process exited: path=/root/.terraform.d/plugins/linux_amd64/terraform-provider-bigip_v1.1.1_x4 pid=5999 error="exit status 2"
2020/03/12 09:04:54 [DEBUG] bigip_ssl_certificate.host_s3_crt: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Canceled desc = context canceled
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/03/12 09:04:54 [TRACE] EvalMaybeTainted: bigip_ssl_certificate.host_s3_crt encountered an error during creation, so it is now marked as tainted
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/03/12 09:04:54 [TRACE] EvalWriteState: removing state object for bigip_ssl_certificate.host_s3_crt
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners
2020/03/12 09:04:54 [TRACE] EvalApplyProvisioners: bigip_ssl_certificate.host_s3_crt has no state, so skipping provisioners
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/03/12 09:04:54 [TRACE] EvalMaybeTainted: bigip_ssl_certificate.host_s3_crt encountered an error during creation, so it is now marked as tainted
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/03/12 09:04:54 [TRACE] EvalWriteState: removing state object for bigip_ssl_certificate.host_s3_crt
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalIf
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalIf
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalWriteDiff
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalApplyPost
2020/03/12 09:04:54 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: rpc error: code = Canceled desc = context canceled
2020/03/12 09:04:54 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Canceled desc = context canceled
2020/03/12 09:04:54 [TRACE] [walkApply] Exiting eval tree: bigip_ssl_certificate.host_s3_crt
2020/03/12 09:04:54 [TRACE] vertex "bigip_ssl_certificate.host_s3_crt": visit complete
2020/03/12 09:04:54 [DEBUG] bigip_ssl_certificate.swisssign_gold_g22: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/03/12 09:04:54 [TRACE] EvalMaybeTainted: bigip_ssl_certificate.swisssign_gold_g22 encountered an error during creation, so it is now marked as tainted
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/03/12 09:04:54 [TRACE] EvalWriteState: removing state object for bigip_ssl_certificate.swisssign_gold_g22
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalApplyProvisioners
2020/03/12 09:04:54 [TRACE] EvalApplyProvisioners: bigip_ssl_certificate.swisssign_gold_g22 has no state, so skipping provisioners
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalMaybeTainted
2020/03/12 09:04:54 [TRACE] EvalMaybeTainted: bigip_ssl_certificate.swisssign_gold_g22 encountered an error during creation, so it is now marked as tainted
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalWriteState
2020/03/12 09:04:54 [TRACE] EvalWriteState: removing state object for bigip_ssl_certificate.swisssign_gold_g22
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalIf
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalIf
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalWriteDiff
2020/03/12 09:04:54 [TRACE] <root>: eval: *terraform.EvalApplyPost
2020/03/12 09:04:54 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2020/03/12 09:04:54 [ERROR] <root>: eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/03/12 09:04:54 [TRACE] [walkApply] Exiting eval tree: bigip_ssl_certificate.swisssign_gold_g22
2020/03/12 09:04:54 [TRACE] vertex "bigip_ssl_certificate.swisssign_gold_g22": visit complete
2020/03/12 09:04:54 [TRACE] dag/walk: upstream of "provider.bigip (close)" errored, so skipping
2020/03/12 09:04:54 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/03/12 09:04:54 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/03/12 09:04:54 [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2020/03/12 09:04:54 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 82
2020/03/12 09:04:54 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2020/03/12 09:04:54 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2020/03/12 09:04:54 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock
2020-03-12T09:04:54.266+0100 [DEBUG] plugin: plugin exited

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

terraform plan on GitHub Gist terraform apply output on GitHub Gist

Crash Output

crash.log on GitHub Gist

Expected Behavior

Resources should get created without any errors

Actual Behavior

Resources (SSL certificates - .crt, key & CA chain) get created on F5 BIG-IP but terraform process does crash. Strange part is, according to the terraform output, one ressource gets created without any problems - others throw an error:

bigip_ssl_key.host_s3_key: Creating...
bigip_ssl_key.host_s3_key: Creation complete after 0s [id=host-s3.key]

Error: rpc error: code = Canceled desc = context canceled

Error: rpc error: code = Unavailable desc = transport is closing
....

But on the F5 all 3 resources got created successfully and are usable!

Steps to Reproduce

terraform init
terraform apply

Additional Context

Second execution of terraform apply leads to the following errors:

bigip_ssl_certificate.ecs_s3_crt: Creating...
bigip_ssl_certificate.swisssign_gold_g22: Creating...
bigip_ssl_key.ecs_s3_key: Creating...

Error: Error in Importing certificate (ecs-s3.crt): 01020066:3: The requested Certificate File (/Common/ecs-s3.crt) already exists in partition Common.

  on bigip-f5.tf line 83, in resource "bigip_ssl_certificate" "ecs_s3_crt":
  83: resource "bigip_ssl_certificate" "ecs_s3_crt" {

Error: Error in Importing certificate key (ecs-s3.key): 01020066:3: The requested Certificate Key File (/Common/ecs-s3.key) already exists in partition Common.

  on bigip-f5.tf line 89, in resource "bigip_ssl_key" "ecs_s3_key":
  89: resource "bigip_ssl_key" "ecs_s3_key" {

Error: Error in Importing certificate (Server_Gold_G22_2014-chain.crt): 01020066:3: The requested Certificate File (/Common/Server_Gold_G22_2014-chain.crt) already exists in partition Common.

  on bigip-f5.tf line 95, in resource "bigip_ssl_certificate" "swisssign_gold_g22":
  95: resource "bigip_ssl_certificate" "swisssign_gold_g22" {

Why does Terraform think, that this resources do already exist and does not connect them to the terraform plan?

I also tried to do an import of an existing remote object - here terraform reports a non existing remote-object ?!

# terraform import bigip_ssl_certificate.ecs_s3_crt "/Common//Common/ecs-s3.crt"
bigip_ssl_certificate.ecs_s3_crt: Importing from ID "/Common//Common/ecs-s3.crt"...
bigip_ssl_certificate.ecs_s3_crt: Import prepared!
  Prepared bigip_ssl_certificate for import
bigip_ssl_certificate.ecs_s3_crt: Refreshing state... [id=/Common//Common/ecs-s3.crt]

Error: Cannot import non-existent remote object

While attempting to import an existing object to
bigip_ssl_certificate.ecs_s3_crt, the provider detected that no object exists
with the given id. Only pre-existing objects can be imported; check that the
id is correct and that it is associated with the provider's configured region
or endpoint, or use "terraform apply" to create a new remote object for this
resource.

RavinderReddyF5 commented 3 years ago

Comment by RavinderReddyF5 Tuesday Mar 17, 2020 at 10:53 GMT

@revog please use partition name without / , for getting certificate we are creating name like ~ + partition + ~ + name

RavinderReddyF5 commented 3 years ago

Comment by RavinderReddyF5 Tuesday Mar 17, 2020 at 17:22 GMT

@revog respective validation is added in :https://github.com/terraform-providers/terraform-provider-bigip/pull/258

RavinderReddyF5 commented 3 years ago

Comment by revog Wednesday Mar 18, 2020 at 10:08 GMT

@RavinderReddyF5 Great! Thanks for this information, indeed it does work now as expected :-)!

RavinderReddyF5 commented 3 years ago

Comment by sebbycorp Tuesday Jun 23, 2020 at 02:23 GMT

I am getting an error when using AS3 to create certs .. Terraform Version


Terraform v0.12.26
+ provider.bigip v1.2.1
+ provider.docker v2.7.1
provider.bigip v1.1.1

Using the f5 sample code from https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/examples.html#example-2-https-application https

**Terraform Configuration Files**

RavinderReddyF5 commented 3 years ago

Comment by papineni87 Tuesday Jun 23, 2020 at 03:47 GMT

Can you share the logs along with terraform config ?

RavinderReddyF5 / terraform-provider-bigip-version0.12