IBM-Cloud / terraform-provider-ibm

https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs
Mozilla Public License 2.0
342 stars 673 forks source link

Cached token expired when setting CORS config during cloudant instance deployment #5155

Open vbontempi opened 9 months ago

vbontempi commented 9 months ago

Creation of Cloudant instance using ibm_cloudant is failing when the instance deployment takes more than one hour to complete the deployment.

After one hour the cached token expires and it isn't refreshed Logs below of local execution with terraform debug log enabled: as found from the logs the token expiration is correctly handled with catalog APIs, but the same is not done with cloudant instance APIs when triggering the PUT request to set CORS config

2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: [DEBUG] REQUEST: [2024-02-23T17:54:45+01:00] GET /api/v1/cloudant HTTP/1.1
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Host: globalcatalog.cloud.ibm.com
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Accept: application/json
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Accept-Language: en
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Authorization: [PRIVATE DATA HIDDEN]
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Type: application/json
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: User-Agent: Bluemix-go SDK 0.1 / darwin
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Original-User-Agent: terraform-provider-ibm/1.62.0
2024-02-23T17:54:45.358+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: [DEBUG] RESPONSE: [2024-02-23T17:54:45+01:00] Elapsed: 206ms HTTP/1.1 401 Unauthorized
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Length: 133
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cf-Cache-Status: DYNAMIC
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cf-Ray: 85a0fe16fb6cbac1-MXP
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Connection: keep-alive
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Type: application/json; charset=utf-8
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Date: Fri, 23 Feb 2024 16:54:45 GMT
2024-02-23T17:54:45.541+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Server: cloudflare
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Strict-Transport-Security: max-age=31536000
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Content-Type-Options: nosniff
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Correlation-Id: 885ccdd2-df4b-4f88-a456-caf853de8f17
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Envoy-Upstream-Service-Time: 19
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Gc-Instance-Id: globalcatalog-eu-de-prod-resource-catalog-7b5557dc48-rr5mz
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Global-Transaction-Id: 885ccdd2-df4b-4f88-a456-caf853de8f17
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Request-Id: 885ccdd2-df4b-4f88-a456-caf853de8f17
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: {"message":"Cached token expired at 2024-02-23T16:31:05Z","code":401,"global_transaction_id":"885ccdd2-df4b-4f88-a456-caf853de8f17"}
2024-02-23T17:54:45.557+0100 [INFO]  provider.terraform-provider-ibm_v1.62.0: Authentication failed. Trying token refresh: timestamp=2024-02-23T17:54:45.541+0100
2024-02-23T17:54:45.557+0100 [INFO]  provider.terraform-provider-ibm_v1.62.0: Retrying authentication using API Key: timestamp=2024-02-23T17:54:45.541+0100
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: [DEBUG] REQUEST: [2024-02-23T17:54:45+01:00] POST /identity/token HTTP/1.1
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Host: iam.cloud.ibm.com
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Accept: application/json
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Accept-Language: en
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Authorization: [PRIVATE DATA HIDDEN]
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Type: application/x-www-form-urlencoded
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: User-Agent: Bluemix-go SDK 0.1 / darwin
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Original-User-Agent: terraform-provider-ibm/1.62.0
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:45.557+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: apikey=[PRIVATE DATA HIDDEN]&grant_type=urn%3Aibm%3Aparams%3Aoauth%3Agrant-type%3Aapikey&response_type=cloud_iam
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: [DEBUG] RESPONSE: [2024-02-23T17:54:45+01:00] Elapsed: 268ms HTTP/1.1 200 OK
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Length: 3175
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Akamai-Grn: 0.5d346868.1708707285.5e88ae2a
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cache-Control: no-cache, no-store, must-revalidate
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Connection: keep-alive
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Language: en-US
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Type: application/json
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Date: Fri, 23 Feb 2024 16:54:45 GMT
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Expires: 0
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Pragma: no-cache
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Strict-Transport-Security: max-age=31536000; includeSubDomains
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Transaction-Id: YzQ2cXQ-7944e12ade44428fa0f8e2224e70e677
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Content-Type-Options: nosniff
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Correlation-Id: YzQ2cXQ-7944e12ade44428fa0f8e2224e70e677
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Proxy-Upstream-Service-Time: 114
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Request-Id: 1b5c9531-be14-411c-bc87-62d3ef5aeda7
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: {"access_token":"[PRIVATE DATA HIDDEN]","refresh_token":"[PRIVATE DATA HIDDEN]","ims_user_id":10332705,"token_type":"[PRIVATE DATA HIDDEN]","expires_in":3600,"expiration":1708710882,"refresh_token_expiration":1708966482,"scope":"ibm openid"}
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: [DEBUG] REQUEST: [2024-02-23T17:54:45+01:00] GET /api/v1/cloudant HTTP/1.1
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Host: globalcatalog.cloud.ibm.com
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Accept: application/json
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Accept-Language: en
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Authorization: [PRIVATE DATA HIDDEN]
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Type: application/json
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: User-Agent: Bluemix-go SDK 0.1 / darwin
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Original-User-Agent: terraform-provider-ibm/1.62.0
2024-02-23T17:54:45.811+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: [DEBUG] RESPONSE: [2024-02-23T17:54:46+01:00] Elapsed: 264ms HTTP/1.1 200 OK
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Transfer-Encoding: chunked
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cache-Control: private, must-revalidate
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cf-Cache-Status: DYNAMIC
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Cf-Ray: 85a0fe194ef3bac1-MXP
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Connection: keep-alive
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Content-Type: application/json; charset=utf-8
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Date: Fri, 23 Feb 2024 16:54:46 GMT
2024-02-23T17:54:46.083+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Etag: W/"6-3a20987232008ae561ea7083c9dcd72c--notfull--"
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Server: cloudflare
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Strict-Transport-Security: max-age=31536000
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Vary: Accept-Encoding, Origin, Access-Control-Request-Headers, Access-Control-Request-Method
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Content-Type-Options: nosniff
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Correlation-Id: 66ea1898-9fc1-4355-b29f-7fa2bb0786ef
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Envoy-Upstream-Service-Time: 202
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Gc-Instance-Id: globalcatalog-eu-de-prod-resource-catalog-7b5557dc48-mhgfl
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Global-Transaction-Id: 66ea1898-9fc1-4355-b29f-7fa2bb0786ef
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: X-Request-Id: 66ea1898-9fc1-4355-b29f-7fa2bb0786ef
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:46.107+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: 20d8
(...)
2024-02-23T17:54:46.169+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:46.169+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: 0
2024-02-23T17:54:46.169+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0:
2024-02-23T17:54:46.169+0100 [INFO]  provider.terraform-provider-ibm_v1.62.0: 2024/02/23 17:54:46 [Debug] Request:
PUT /_api/v2/user/config/cors HTTP/1.1
Host: bea21813-9abd-46e0-9ece-c6427e1b26b3-bluemix.cloudantnosqldb.appdomain.cloud
User-Agent: cloudant-terraform/1.62.0
Transfer-Encoding: chunked
Accept: application/json
Authorization: [redacted]
Content-Encoding: gzip
Content-Type: application/json
X-IBMCloud-SDK-Analytics: service_name=cloudant;service_version=V1;operation_id=PutCorsConfiguration
Accept-Encoding: gzip

54
[binary content]
: timestamp=2024-02-23T17:54:46.169+0100
2024-02-23T17:54:47.265+0100 [INFO]  provider.terraform-provider-ibm_v1.62.0: 2024/02/23 17:54:47 [Debug] Response:
HTTP/2.0 401 Unauthorized
Cache-Control: no-cache, no-store, must-revalidate
Cf-Cache-Status: DYNAMIC
Cf-Ray: 85a0fe204b302cde-DFW
Content-Type: application/json
Date: Fri, 23 Feb 2024 16:54:47 GMT
Server: Cloudant
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
Via: 2.0 lb3.bm-cc-us-south-29 (Glum)
X-Cloudant-Action: cloudantnosqldb.sapi.usercors
X-Cloudant-Backend: sapi.lb3.bm-cc-us-south-29.cloudant.net
X-Cloudant-Request-Class: unlimited
X-Content-Type-Options: nosniff
X-Envoy-Upstream-Service-Time: 228
X-Global-Transaction-Id: 7c33563f-796c-41c9-9c32-1a8de1fb8535

{"message": "The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required."}: timestamp=2024-02-23T17:54:47.265+0100
2024-02-23T17:54:47.266+0100 [DEBUG] provider.terraform-provider-ibm_v1.62.0: Error updating CORS settings: The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required.
{
    "StatusCode": 401,
    "Headers": {
        "Cache-Control": [
            "no-cache, no-store, must-revalidate"
        ],
        "Cf-Cache-Status": [
            "DYNAMIC"
        ],
        "Cf-Ray": [
            "85a0fe204b302cde-DFW"
        ],
        "Content-Type": [
            "application/json"
        ],
        "Date": [
            "Fri, 23 Feb 2024 16:54:47 GMT"
        ],
        "Server": [
            "Cloudant"
        ],
        "Strict-Transport-Security": [
            "max-age=31536000"
        ],
        "Vary": [
            "Accept-Encoding"
        ],
        "Via": [
            "2.0 lb3.bm-cc-us-south-29 (Glum)"
        ],
        "X-Cloudant-Action": [
            "cloudantnosqldb.sapi.usercors"
        ],
        "X-Cloudant-Backend": [
            "sapi.lb3.bm-cc-us-south-29.cloudant.net"
        ],
        "X-Cloudant-Request-Class": [
            "unlimited"
        ],
        "X-Content-Type-Options": [
            "nosniff"
        ],
        "X-Envoy-Upstream-Service-Time": [
            "228"
        ],
        "X-Global-Transaction-Id": [
            "7c33563f-796c-41c9-9c32-1a8de1fb8535"
        ]
    },
    "Result": {
        "message": "The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required."
    },
    "RawResult": null
}: timestamp=2024-02-23T17:54:47.265+0100
2024-02-23T17:54:47.266+0100 [ERROR] provider.terraform-provider-ibm_v1.62.0: Response contains error diagnostic: tf_resource_type=ibm_cloudant @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_severity=ERROR diagnostic_summary="[ERROR] Error updating CORS settings: The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required." tf_proto_version=5.4 tf_req_id=9df9f636-cc67-b278-5c33-4f06c3037229 tf_rpc=ApplyResourceChange @module=sdk.proto diagnostic_detail= tf_provider_addr=provider timestamp=2024-02-23T17:54:47.266+0100
2024-02-23T17:54:47.289+0100 [ERROR] vertex "module.cloudant.ibm_cloudant.cloudant_instance" error: [ERROR] Error updating CORS settings: The server could not verify that you are authorized to access the URL requested. You either supplied the wrong credentials (e.g. a bad password), or your browser doesn't understand how to supply the credentials required.

Community Note

Terraform CLI and Terraform IBM Provider Version

terraform version 1.7.4.

terraform ibm provider

ibm = {
      source  = "IBM-Cloud/ibm"
      version = ">= 1.55.0"
    }

Affected Resource(s)

Debug Output

pasted above

Steps to Reproduce

template to debug the issue

locals {
  sm_guid = var.existing_sm_instance_guid
  sm_crn = var.existing_sm_instance_crn
  sm_region = var.existing_sm_instance_region
}

module "resource_group" {
  source  = "terraform-ibm-modules/resource-group/ibm"
  version = "1.1.0"
  resource_group_name          = var.resource_group == null ? "${var.prefix}-resource-group" : null
  existing_resource_group_name = var.resource_group
}
module "cloudant" {
  source            = "terraform-ibm-modules/cloudant/ibm"
  version           = "1.1.7"
  resource_group_id = module.resource_group.resource_group_id
  instance_name     = "${var.prefix}-cloudant"
  access_tags       = []
  region            = var.region
  tags              = var.resource_tags
  plan              = "lite"
  database_config = []
}

# load cloudant instance details when ready
data "ibm_cloudant" "instance" {
  depends_on        = [module.cloudant]
  name              = module.cloudant.instance_name
  resource_group_id = module.resource_group.resource_group_id
}

# create resource key for cloudant instance
resource "ibm_resource_key" "resource_key" {
  name                 = "cd-resource-key"
  role                 = "Manager"
  resource_instance_id = data.ibm_cloudant.instance.id
  timeouts {
    create = "15m"
    delete = "15m"
  }
}

# create secrets group for secrets
module "secrets_manager_group" {
  source                   = "terraform-ibm-modules/secrets-manager-secret-group/ibm"
  version                  = "1.1.0"
  region                   = local.sm_region
  secrets_manager_guid     = local.sm_guid
  secret_group_name        = "${var.prefix}-secret-group"                   high entropy string as is static value
  secret_group_description = "Secret-Group for storing account credentials" 
  providers = {
    ibm = ibm.ibm-sm
  }
}

# Creates the arbitrary secret to store the cloudant resource key in secrets manager
module "sm_arbitrary_cloudant_secret" {
  source               = "terraform-ibm-modules/secrets-manager-secret/ibm"
  version              = "1.1.1"
  region               = local.sm_region
  secrets_manager_guid = local.sm_guid
  secret_group_id      = module.secrets_manager_group.secret_group_id
  secret_type          = "arbitrary"
  secret_name             = "${var.prefix}-cloudant-rk-secret"                   
  secret_description      = "example secret in existing secret manager instance" plaintext-exposure
  secret_payload_password = ibm_resource_key.resource_key.credentials["apikey"]
  providers = {
    ibm = ibm.ibm-sm
  }
}
  1. terraform apply
ricellis commented 9 months ago

When initializing the Cloudant client for the newly provisioned service the resource does: https://github.com/IBM-Cloud/terraform-provider-ibm/blob/819a586e652748b9682e060412d89845d89c5f45/ibm/service/cloudant/resource_ibm_cloudant.go#L384

to later obtain a bearer token https://github.com/IBM-Cloud/terraform-provider-ibm/blob/819a586e652748b9682e060412d89845d89c5f45/ibm/service/cloudant/resource_ibm_cloudant.go#L390

the meta used to get the session is passed into resourceIBMCloudantCreate when the resource create starts.

On Feb 23rd instances were created as normal, but for a short window there were long delays in them being recognized as provisioned. In this case there was longer than the token lifetime inbetween the start of resourceIBMCloudantCreate and the call to get the BluemixSession(). Whilst this is extremely unusual it would be good to handle it better if it does happen and refresh the token if necessary.

I'm not sure on the expected pattern in the provider for this, it seems wasteful to call RefreshToken(session) every time when the token will nearly always be valid. Equally the Cloudant resource does not seem like the right place to be decoding a session bearer token and checking if it is expired.

Some possible options:

It would be good to hear the maintainers' thoughts on this.