Unexpected status code 401 from API https://private.us-south.kms.cloud.ibm.com/api/v2/keys

[ocofaigh]

I observed the following error when deploying the DA:

 2024/08/29 20:51:21 Terraform apply | Error: unexpected response code '401': {"statusCode":401,"message":"Unexpected status code 401 from API https://private.us-south.kms.cloud.ibm.com/api/v2/keys"}
 2024/08/29 20:51:21 Terraform apply | 
 2024/08/29 20:51:21 Terraform apply |   with module.storage_delegation[0].restapi_object.storage_delegation,
 2024/08/29 20:51:21 Terraform apply |   on storage_delegation/main.tf line 36, in resource "restapi_object" "storage_delegation":
 2024/08/29 20:51:21 Terraform apply |   36: resource "restapi_object" "storage_delegation" {
 2024/08/29 20:51:21 Terraform apply | 
 2024/08/29 20:51:21 Terraform APPLY error: Terraform APPLY errorexit status 1

Re-apply hit the same error

[ocofaigh]

cc @andreainnocenti

[ocofaigh]

The error is coming from module.storage_delegation[0].restapi_object.storage_delegation. This corresponds to:

resource "restapi_object" "storage_delegation" {
  provider       = restapi.restapi_watsonx_admin
  depends_on     = [resource.ibm_iam_authorization_policy.cos_s2s_keyprotect, data.ibm_kms_key.kms_key]
  path           = "//dataplatform.cloud.ibm.com/api/rest/v1/storage-delegations"
  read_path      = "//dataplatform.cloud.ibm.com/api/rest/v1/storage-delegations/{id}"
  read_method    = "GET"
  create_path    = "//dataplatform.cloud.ibm.com/api/rest/v1/storage-delegations"
  create_method  = "POST"
  id_attribute   = var.cos_guid
  object_id      = var.cos_guid
  destroy_method = "DELETE"
  destroy_path   = "//dataplatform.cloud.ibm.com/api/rest/v1/storage-delegations/{id}"
  data           = <<-EOT
                  {
                    "cos_instance_id": "${var.cos_guid}",
                    "kms_key_crn": "${data.ibm_kms_key.kms_key.keys[0].crn}",
                    "catalogs": true,
                    "projects": true
                  }
                  EOT
}

Is COS getting a 401 when trying to access the KMS key? I can see the code is creating an s2s auth policy between COS and KMS to allow COS reader access, so not sure why the 401?

[ocofaigh]

I tried creating an auth policy to allow Watson Discovery service reader access to Key Protect, but it still failed

[ocofaigh]

Quick update. I think the 401 is happening because of a known timing issue, and a workaround needs to be added to the code that we have in all other DAs to add a sleep after auth policy is created to allow the policy to be replicated on the backend (same as this code)

On retry (after auth policy replication occurrs on backend) its actually now giving a 403 error: Error: unexpected response code '403': {"statusCode":403,"message":"Insufficient account entitlements."}

This would indicate that the user provising the DA is missing some kind of entitlement. Need to find out what that is and how to add it.

[ocofaigh]

I created a new issue to track the missing entitlement https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/issues/154

[ocofaigh]

Going to create a PR today with the workaround for the 401

[ocofaigh]

PR with workaround: https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/pull/156

[ocofaigh]

Fixed in https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/releases/tag/v1.4.11

[ocofaigh]

Issue has been reproduced:

[PROJECTS] Apply Error: {map[error_msg: unexpected response code '401': {"statusCode":401,"message":"Unexpected status code 401 from API https://private.us-south.kms.cloud.ibm.com/api/v2/keys"} resource_name:storage_delegation resource_type:restapi_object]}

I don't know why the us-south KMS endpoint is being used. The test was using eu-de. Need to see dig into the code..

[ocofaigh]

Oh wow I think I found it. According to the code here, each region has a different endpoint:

dataplatform_ui_mapping = {
    "us-south" = "https://dataplatform.cloud.ibm.com/",
    "eu-gb"    = "https://eu-uk.dataplatform.cloud.ibm.com/",
    "eu-de"    = "https://eu-de.dataplatform.cloud.ibm.com/",
    "jp-tok"   = "https://jp-tok.dataplatform.cloud.ibm.com/"
  }

However the restapi call here is hard coded to dataplatform.cloud.ibm.com which is the us-south endpoint. That might explain why the api is hitting the us-south KMS endpoint and getting a 401 since the KMS instance is actually in eu-de. The mapping probably needs to be used in that restapi call too?

[ocofaigh]

Fix is in https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/releases/tag/v1.4.12