search

hashicorp / terraform-plugin-sdk

Terraform Plugin SDK enables building plugins (providers) to manage any service providers or custom in-house solutions
https://developer.hashicorp.com/terraform/plugin
Mozilla Public License 2.0
438 stars 232 forks source link

The authorization header is removed intermittently while using WaitForStateContext #820

Closed neil-yechenwei closed 2 years ago

neil-yechenwei commented 3 years ago

terraform-plugin-sdk version: v2.7.0 Terraform version: 1.0.9 AzureRM provider version: 2.82.0

Impacted resource:

https://github.com/hashicorp/terraform-provider-azurerm/blob/v2.82.0/internal/services/monitor/monitor_metric_alert_resource.go

The code snippet for WaitForStateContext in resource Monitor Metric Alert:

image image

Terraform configuration file:

provider "azurerm" {
  features {}
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-mmc-test01"
  location = "eastus"
}

resource "azurerm_key_vault" "test" {
  for_each = {
    name1 = "kvntest01"
  }
  name = each.value

  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  sku_name = "premium"

  access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.client_id

    key_permissions = [
      "create",
    ]

    secret_permissions = [
      "set",
    ]
  }
}

resource "azurerm_monitor_action_group" "test" {
  name                = "acctestmagtest01"
  resource_group_name = azurerm_resource_group.test.name
  short_name          = "P0"

  email_receiver {
    name          = "ishantdevops"
    email_address = "devops@example.com"
  }
}

resource "azurerm_monitor_metric_alert" "test" {
  for_each            = azurerm_key_vault.test

  name                = "Keyvault Availability (${each.value.name}) - test01"
  resource_group_name = azurerm_resource_group.test.name
  scopes              = [each.value.id]
  description         = "Action will be triggered whe the availability of the vault is less than 98%."
  severity            = 2

  criteria {
    metric_namespace = "Microsoft.KeyVault/vaults"
    metric_name      = "Availability"
    aggregation      = "Average"
    operator         = "LessThan"
    threshold        = 98
  }

  action {
    action_group_id = azurerm_monitor_action_group.test.id
  }
}

Debug log:

2021-10-26T09:15:02.123+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: AzureRM Request: 
PUT /subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx?api-version=2018-03-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.16.7 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v58.0.0 insights/2021-07-01-preview HashiCorp Terraform/1.0.9 (+https://www.terraform.io) Terraform Plugin SDK/2.7.0 terraform-provider-azurerm/2.82.0 pid-xx-xx-xx
Content-Length: 1356
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: xx-xx-xx-xx
Accept-Encoding: gzip

2021-10-26T09:15:06.900+0100 [TRACE] dag/walk: vertex "module.xxx (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:07.534+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: AzureRM Response for https://management.azure.com/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx?api-version=2018-03-01: 
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Api-Supported-Versions: 2017-09-01-preview, 2018-03-01
Arr-Disable-Session-Affinity: true
Cache-Control: no-cache, proxy-revalidate
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Date: Tue, 26 Oct 2021 08:15:06 GMT
Expires: -1
Pragma: no-cache
Proxy-Connection: Keep-Alive
Server: Microsoft-IIS/10.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding,Accept-Encoding
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: xx-xx-xx-xx
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 299
X-Ms-Request-Id: xx-xx-xx-xx
X-Ms-Request-Id: xx-xx-xx-xx
X-Ms-Routing-Request-Id: xxxx
X-Powered-By: ASP.NET

{
  "id": "/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx",
  "name": "xx-xx-xx-xx",
  "type": "Microsoft.Insights/metricAlerts",
  "location": "global",
  "tags": {
    "lv_application": "xxx",
    "lv_costcentre": "xx",
    "lv_environment": "xx",
    "lv_sbu": "xxx",
    "lv_team": "xxx",
    "lv_usecase": "xxx",
    "lv_service": "xxx"
  },
  "properties": {
    "description": "xxxx",
    "severity": 3,
    "enabled": true,
    "scopes": [
      "/subscriptions/xx-xx-xx-xx/resourcegroups/xx-xx-xx-xx/providers/Microsoft.ContainerService/managedClusters/xxxx"
    ],
    "evaluationFrequency": "xxx",
    "windowSize": "xxx",
    "criteria": {
      "allOf": [
        {
          "threshold": 0.0,
          "name": "xxx",
          "metricNamespace": "Microsoft.ContainerService/managedClusters",
          "metricName": "xxx",
          "dimensions": [
            {
              "name": "xxx",
              "operator": "Include",
              "values": [
                "NotReady",
                "Unknown"
              ]
            }
          ],
          "operator": "GreaterThan",
          "timeAggregation": "Total",
          "skipMetricValidation": false,
          "criterionType": "xxxx"
        }
      ],
      "odata.type": "xxxx"
    },
    "autoMitigate": true,
    "targetResourceType": "",
    "targetResourceRegion": "",
    "actions": [
      {
        "actionGroupId": "/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/actionGroups/xxxxx",
        "webHookProperties": {}
      }
    ]
  }
}
0: timestamp=2021-10-26T09:15:07.533+0100
2021-10-26T09:15:07.534+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: Waiting for Monitor Metric Alert "xx-xx-xx-xx" (Resource Group "xx-xx-xx-xx") to be created: timestamp=2021-10-26T09:15:07.533+0100
2021-10-26T09:15:07.534+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: Waiting for state to become: [200]: timestamp=2021-10-26T09:15:07.533+0100
2021-10-26T09:15:07.534+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: AzureRM Request: 
GET /subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx?api-version=2018-03-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.16.7 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v58.0.0 insights/2021-07-01-preview HashiCorp Terraform/1.0.9 (+https://www.terraform.io) Terraform Plugin SDK/2.7.0 terraform-provider-azurerm/2.82.0 pid-xx-xx-xx-xx
X-Ms-Correlation-Request-Id: xx-xx-xx-xx
Accept-Encoding: gzip: timestamp=2021-10-26T09:15:07.533+0100
2021-10-26T09:15:07.615+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: AzureRM Response for https://management.azure.com/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx?api-version=2018-03-01: 
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Api-Supported-Versions: 2017-09-01-preview, 2018-03-01
Arr-Disable-Session-Affinity: true
Cache-Control: no-cache, proxy-revalidate
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Date: Tue, 26 Oct 2021 08:15:06 GMT
Expires: -1
Pragma: no-cache
Proxy-Connection: Keep-Alive
Server: Microsoft-IIS/10.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding,Accept-Encoding
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: xx-xx-xx-xx
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11987
X-Ms-Request-Id: xx-xx-xx-xx
X-Ms-Request-Id: xx-xx-xx-xx
X-Ms-Routing-Request-Id: xxxxx
X-Powered-By: ASP.NET

{
  "id": "/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx",
  "name": "xx-xx-xx-xx",
  "type": "Microsoft.Insights/metricAlerts",
  "location": "global",
  "tags": {
    "lv_application": "xxx",
    "lv_costcentre": "xxx",
    "lv_environment": "xxx",
    "lv_sbu": "xxx",
    "lv_team": "xxx",
    "lv_usecase": "xxx",
    "lv_service": "xxxx"
  },
  "properties": {
    "description": "xxx",
    "severity": 3,
    "enabled": true,
    "scopes": [
      "/subscriptions/xx-xx-xx-xx/resourcegroups/xx-xx-xx-xx/providers/Microsoft.ContainerService/managedClusters/xxxx"
    ],
    "evaluationFrequency": "xxx",
    "windowSize": "xxx",
    "criteria": {
      "allOf": [
        {
          "threshold": 0.0,
          "name": "Metric1",
          "metricNamespace": "Microsoft.ContainerService/managedClusters",
          "metricName": "xxxx",
          "dimensions": [
            {
              "name": "xxx",
              "operator": "Include",
              "values": [
                "NotReady",
                "Unknown"
              ]
            }
          ],
          "operator": "GreaterThan",
          "timeAggregation": "Total",
          "skipMetricValidation": false,
          "criterionType": "xxx"
        }
      ],
      "odata.type": "xxxx"
    },
    "autoMitigate": true,
    "targetResourceType": "",
    "targetResourceRegion": "",
    "actions": [
      {
        "actionGroupId": "/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/actionGroups/xxxx",
        "webHookProperties": {}
      }
    ]
  }
}
0: timestamp=2021-10-26T09:15:07.615+0100
2021-10-26T09:15:07.615+0100 [TRACE] provider.terraform-provider-azurerm_v2.82.0_x5: Waiting 15s before next try: timestamp=2021-10-26T09:15:07.615+0100
2021-10-26T09:15:08.279+0100 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.xxxx (close)"
2021-10-26T09:15:08.285+0100 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:08.291+0100 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2021-10-26T09:15:11.901+0100 [TRACE] dag/walk: vertex "module.xxxx (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:13.279+0100 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.xxxx (close)"
2021-10-26T09:15:13.286+0100 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:13.292+0100 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2021-10-26T09:15:16.902+0100 [TRACE] dag/walk: vertex "module.xxxx (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:18.280+0100 [TRACE] dag/walk: vertex "meta.count-boundary (EachMode fixup)" is waiting for "module.xxxx (close)"
2021-10-26T09:15:18.287+0100 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:18.292+0100 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2021-10-26T09:15:21.903+0100 [TRACE] dag/walk: vertex "module.xxxx (close)" is waiting for "module.xxxx.azurerm_monitor_metric_alert.xxxx"
2021-10-26T09:15:22.619+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: AzureRM Request: 
GET /subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx?api-version=2018-03-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.16.7 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v58.0.0 insights/2021-07-01-preview HashiCorp Terraform/1.0.9 (+https://www.terraform.io) Terraform Plugin SDK/2.7.0 terraform-provider-azurerm/2.82.0 pid-xx-xx-xx-xx
X-Ms-Correlation-Request-Id: xx-xx-xx-xx
Accept-Encoding: gzip: timestamp=2021-10-26T09:15:22.619+0100
2021-10-26T09:15:22.796+0100 [DEBUG] provider.terraform-provider-azurerm_v2.82.0_x5: AzureRM Response for https://management.azure.com/subscriptions/xx-xx-xx-xx/resourceGroups/xx-xx-xx-xx/providers/Microsoft.Insights/metricAlerts/xx-xx-xx-xx?api-version=2018-03-01: 
HTTP/1.1 401 Unauthorized
Content-Length: 115
Cache-Control: no-cache, proxy-revalidate
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Date: Tue, 26 Oct 2021 08:15:22 GMT
Expires: -1
Pragma: no-cache
Proxy-Connection: Keep-Alive
Proxy-Support: Session-based-authentication
Set-Cookie: xxxxx
Strict-Transport-Security: max-age=31536000; includeSubDomains
Www-Authenticate: Bearer authorization_uri="https://login.windows.net/xxxxx", error="invalid_token", error_description="The authentication failed because of missing 'Authorization' header."
X-Bluecoat-Authorization: xxxxx
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: xx-xx-xx-xx
X-Ms-Failure-Cause: gateway
X-Ms-Request-Id: xxxx
X-Ms-Routing-Request-Id: xxxxx

{"error":{"code":"AuthenticationFailed","message":"Authentication failed. The 'Authorization' header is missing."}}: timestamp=2021-10-26T09:15:22.795+0100
2021-10-26T09:15:22.796+0100 [TRACE] maybeTainted: module.xxxx.azurerm_monitor_metric_alert.xxxx encountered an error during creation, so it is now marked as tainted
2021-10-26T09:15:22.796+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.xxxx.azurerm_monitor_metric_alert.xxxx
2021-10-26T09:15:22.796+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.xxxx.azurerm_monitor_metric_alert.xxxx
2021-10-26T09:15:22.796+0100 [TRACE] evalApplyProvisioners: module.xxxx.azurerm_monitor_metric_alert.xxxx is tainted, so skipping provisioning
2021-10-26T09:15:22.796+0100 [TRACE] maybeTainted: module.xxxx.azurerm_monitor_metric_alert.xxxx was already tainted, so nothing to do
2021-10-26T09:15:22.796+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.xxxx.azurerm_monitor_metric_alert.xxxx
2021-10-26T09:15:22.796+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: removing state object for module.xxxx.azurerm_monitor_metric_alert.xxxx
2021-10-26T09:15:22.797+0100 [TRACE] vertex "module.xxxx.azurerm_monitor_metric_alert.xxxx": visit complete

Expected Behavior:

The resource azurerm_monitor_metric_alert should be created successfully without error while using WaitForStateContext.

Actual Behavior:

It intermittently failed to create the resource azurerm_monitor_metric_alert while using WaitForStateContext and threw below error message. Per the error message, seems the authorization header would be removed intermittently while creating this resource with WaitForStateContext.

Error message:

09:15:02 module.xxx.azurerm_monitor_metric_alert.xxx: Creating... 09:15:12 module.xxx.azurerm_monitor_metric_alert.xxx: Still creating... [10s elapsed] 09:15:22 module.xxx.azurerm_monitor_metric_alert.xxx: Still creating... [20s elapsed] 09:15:23 Error: waiting for Monitor Metric Alert: (Name "xxxxx" / Resource Group "xxxxx") to finish provisioning: retrieving Monitor Metric Alert "xxxxx" (Resource Group "xxxxx"): insights.MetricAlertsClient#Get: Failure responding to request: StatusCode=401 -- Original Error: autorest/azure: Service returned an error. Status=401 Code="AuthenticationFailed" Message="Authentication failed. The 'Authorization' header is missing."

Steps to Reproduce:

  1. terraform init
  2. terraform apply
bflad commented 2 years ago

Hi @neil-yechenwei 👋 Thank you for raising this and apologies for the delayed response.

As far as I know, the SDK only calls and does not modify a given StateChangeConf.Refresh function, so it is very unlikely that the SDK itself is doing anything different over time. If the API client is dependent on underlying authentication or authorization information on the system in which it is running, it could be that information changed between executions and the API client reacted to those changes. That would be outside this SDK though, which is only just calling the refresh function given.

If you are still experiencing issues with this, my best recommendation would be to open an issue in the Terraform AzureRM Provider repository as those maintainers may know additional details about the behaviors of the particular API client.

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.