Provider crashes when correcting drift in a Destination

[apr-1985]

I have a webhook destination that is created by Terraform. I made a manual change to the destination to test some things and then ran the Terraform again to set it back to how it should be. However the provider just crashes with the error below.

I tried removing the destination from the state manually correcting the drift and re-importing however the provider still wanted to make changes to the resource and so crashed.

Bit stuck now as I cannot destroy the destination without destroying my whole alerting stack of policies and conditions etc.

tested with provider version 3.8.0.

 Error: Plugin did not respond
│ 
│   with module.setup_podinfo_golden_signals.newrelic_notification_destination.opsgenie-destination,
│   on .terraform/modules/setup_podinfo_golden_signals/new_relic.tf line 1, in resource "newrelic_notification_destination" "opsgenie-destination":
│    1: resource "newrelic_notification_destination" "opsgenie-destination" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-newrelic_v3.8.0 plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 75 [running]:
github.com/newrelic/terraform-provider-newrelic/v2/newrelic.expandNotificationDestinationAuthBasic(...)
        /home/runner/work/terraform-provider-newrelic/terraform-provider-newrelic/newrelic/structures_newrelic_notifications_destination.go:46
github.com/newrelic/terraform-provider-newrelic/v2/newrelic.expandNotificationDestinationUpdate(0x0?)
        /home/runner/work/terraform-provider-newrelic/terraform-provider-newrelic/newrelic/structures_newrelic_notifications_destination.go:74 +0x606
github.com/newrelic/terraform-provider-newrelic/v2/newrelic.resourceNewRelicNotificationDestinationUpdate({0x1c30f90, 0xc000206f00}, 0xc0004acb80, {0x18f2e80?, 0xc0008a61e0?})
        /home/runner/work/terraform-provider-newrelic/terraform-provider-newrelic/newrelic/resource_newrelic_notifications_destination.go:297 +0x91
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xc000419ce0, {0x1c30fc8, 0xc00072a960}, 0xd?, {0x18f2e80, 0xc0008a61e0})
        /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.20.0/helper/schema/resource.go:741 +0x12e
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc000419ce0, {0x1c30fc8, 0xc00072a960}, 0xc0000aca90, 0xc0004aca00, {0x18f2e80, 0xc0008a61e0})
        /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.20.0/helper/schema/resource.go:847 +0x82c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc00000d9b0, {0x1c30fc8?, 0xc00072a840?}, 0xc00004ae60)
        /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.20.0/helper/schema/grpc_provider.go:1021 +0xe3c
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc0003b1220, {0x1c30fc8?, 0xc00072a000?}, 0xc0001441c0)
        /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.12.0/tfprotov5/tf5server/server.go:813 +0x4fc
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x1a4fd60?, 0xc0003b1220}, {0x1c30fc8, 0xc00072a000}, 0xc000144150, 0x0)
        /home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.12.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:385 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000001e0, {0x1c348d0, 0xc0004829c0}, 0xc00024c000, 0xc000455320, 0x2204740, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1301 +0xb0b
google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x1c348d0, 0xc0004829c0}, 0xc00024c000, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:1642 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:938 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.49.0/server.go:936 +0x28a

Error: The terraform-provider-newrelic_v3.8.0 plugin crashed!

[lzaga-newrelic]

Hi @apr-1985 , we'll investigate this issue. Can you share more details about the changes you made to the webhook destination?

[apr-1985]

Hi thanks for picking this up. It looks to be an issue with the 3.7.1 plugin and a destination with no basic auth.

The minimum TF to reproduce is

terraform {
  required_version = ">= 1.2.0"
  required_providers {
    newrelic = {
      source  = "newrelic/newrelic"
      version = "3.7.1"
    }
  }
}

provider "newrelic" {
  account_id = local.newrelic_account_number
  api_key    = local.newrelic_api_key # usually prefixed with 'NRAK'
  region     = "US"                   # Valid regions are US and EU
}

resource "newrelic_notification_destination" "test-destination" {
  name = "test-destination-webhook"
  type = "WEBHOOK"

  property {
    key   = "url"
    value = "https://my-url.com"
  }

  lifecycle {
    ignore_changes = [
      auth_basic
    ]
  }
}

Run this and the destination is setup. Then in the NR One UI change the url to https://my-url1234.com and apply the TF again and it will crash the plugin. Upgrade the plugin to 3.8.0 and apply and it will crash the 3.8.0 plugin.

However. If you start off with the 3.8.0 plugin for the initial apply the problem does not surface. Seems some basic auth changes were made in the file newrelic/structures_newrelic_notifications_destination.go between the versions https://github.com/newrelic/terraform-provider-newrelic/compare/v3.7.1...v3.8.0#diff-546e1a3f9ce991f6dc58616554de9154a3f4360f8a1d8bd20b48a30a0d8d8ab9

As it is working when using 3.8.0 initially (although I cannot figure out why it still crashes when upgrading to 3.8.0) I will leave it to you as to the next steps.

[lzaga-newrelic]

Hi @apr-1985 , We did add a bug fix to version 3.8.0, which causes this error when you have the ignore_changes flag on your resource.

We opened a bug for it and will work on it in the near future. In the meanwhile you can re-import the destination to your state file with the following steps:

1. terraform state show newrelic_notification_destination.test-destination - Copy the destination id
2. terraform state rm newrelic_notification_destination.test-destination
3. terraform import newrelic_notification_destination.test-destination <destination_id>

Hope this helps and I'll update when this bug is fixed.

[lzaga-newrelic]

The solution mentioned above is the best we can do. Closing this issue.