Updating existing_kms_instance_crn from KP CRN to HPCS CRN causing issues.

[Ak-sky]

deploy-many.sh script which failed with below error- for 2b---Security-Service---Secret-Manager

2024/06/04 14:48:44 Terraform apply | module.secrets_manager[0].ibm_resource_instance.secrets_manager_instance: Modifying... [id=crn:v1:bluemix:public:secrets-manager:us-south:a/abac0df06b644a9cabc6e44f55b3880e:6e7d89c5-6bbd-42f7-b46e-52e85dbd52f0::]
 2024/06/04 14:48:45 Terraform apply | 
 2024/06/04 14:48:45 Terraform apply | Error: [ERROR] Error updating resource instance: Please contact the Service Provider for this error. [400, Bad Request] The request body is in wrong format with resp code: {
 2024/06/04 14:48:45 Terraform apply |     "StatusCode": 422,
 2024/06/04 14:48:45 Terraform apply |     "Headers": {
 2024/06/04 14:48:45 Terraform apply |         "Cache-Control": [
 2024/06/04 14:48:45 Terraform apply |             "max-age=0, no-cache, no-store"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Content-Length": [
 2024/06/04 14:48:45 Terraform apply |             "319"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Content-Type": [
 2024/06/04 14:48:45 Terraform apply |             "application/json; charset=utf-8"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Date": [
 2024/06/04 14:48:45 Terraform apply |             "Tue, 04 Jun 2024 14:48:44 GMT"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Expires": [
 2024/06/04 14:48:45 Terraform apply |             "Tue, 04 Jun 2024 14:48:44 GMT"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Pragma": [
 2024/06/04 14:48:45 Terraform apply |             "no-cache"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Request-Id": [
 2024/06/04 14:48:45 Terraform apply |             "bss-5da410cdfd89c372"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Retry-After": [
 2024/06/04 14:48:45 Terraform apply |             "0"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Server": [
 2024/06/04 14:48:45 Terraform apply |             "istio-envoy"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Strict-Transport-Security": [
 2024/06/04 14:48:45 Terraform apply |             "max-age=31536000;includeSubDomains"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "Transaction-Id": [
 2024/06/04 14:48:45 Terraform apply |             "bss-c46905de9c3cb414"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Content-Type-Options": [
 2024/06/04 14:48:45 Terraform apply |             "nosniff"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Correlation-Id": [
 2024/06/04 14:48:45 Terraform apply |             "bss-c46905de9c3cb414"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Envoy-Upstream-Service-Time": [
 2024/06/04 14:48:45 Terraform apply |             "143"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Op-Completion-Time": [
 2024/06/04 14:48:45 Terraform apply |             ""
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Ratelimit-Limit": [
 2024/06/04 14:48:45 Terraform apply |             "100"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Ratelimit-Remaining": [
 2024/06/04 14:48:45 Terraform apply |             "96"
 2024/06/04 14:48:45 Terraform apply |         ],
 2024/06/04 14:48:45 Terraform apply |         "X-Ratelimit-Reset": [
 2024/06/04 14:48:45 Terraform apply |             "0"
 2024/06/04 14:48:45 Terraform apply |         ]
 2024/06/04 14:48:45 Terraform apply |     },
 2024/06/04 14:48:45 Terraform apply |     "Result": {
 2024/06/04 14:48:45 Terraform apply |         "details": "{\"error\":\"BadRequest\",\"description\":\"The request body is in wrong format\"}",
 2024/06/04 14:48:45 Terraform apply |         "error_code": "RC-ServiceBrokerErrorResponse",
 2024/06/04 14:48:45 Terraform apply |         "message": "Please contact the Service Provider for this error. [400, Bad Request] The request body is in wrong format",
 2024/06/04 14:48:45 Terraform apply |         "status_code": 422,
 2024/06/04 14:48:45 Terraform apply |         "transaction_id": "bss-c46905de9c3cb414"
 2024/06/04 14:48:45 Terraform apply |     },
 2024/06/04 14:48:45 Terraform apply |     "RawResult": null
 2024/06/04 14:48:45 Terraform apply | }
 2024/06/04 14:48:45 Terraform apply | 
 2024/06/04 14:48:45 Terraform apply | 
 2024/06/04 14:48:45 Terraform apply |   with module.secrets_manager[0].ibm_resource_instance.secrets_manager_instance,
 2024/06/04 14:48:45 Terraform apply |   on ../../main.tf line 21, in resource "ibm_resource_instance" "secrets_manager_instance":

• After waiting for sometime, re-ran the shell script, above error has gone but it failed at 3---Observability---Logging-Monitoring-Activity-Tracker with the error

2024/06/04 18:06:23 Terraform apply | module.cos_bucket[0].module.buckets["sky1-rag-log-archive-cos-bucket"].ibm_cos_bucket.cos_bucket[0]: Creating...
 2024/06/04 18:06:23 Terraform apply | module.cos_bucket[0].module.buckets["sky1-rag-at-events-cos-bucket"].ibm_cos_bucket.cos_bucket[0]: Creating...
 2024/06/04 18:06:23 Terraform apply | 
 2024/06/04 18:06:23 Terraform apply | Error: BucketAlreadyExists: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.
 2024/06/04 18:06:23 Terraform apply |  status code: 409, request id: e74b4508-071f-413a-891b-a08b7feb5eb4, host id: 
 2024/06/04 18:06:23 Terraform apply | 
 2024/06/04 18:06:23 Terraform apply |   with module.cos_bucket[0].module.buckets["sky1-rag-at-events-cos-bucket"].ibm_cos_bucket.cos_bucket[0],
 2024/06/04 18:06:23 Terraform apply |   on .terraform/modules/cos_bucket/main.tf line 127, in resource "ibm_cos_bucket" "cos_bucket":

and at 6---Sample-RAG-app-configuration with below error

 2024/06/04 18:25:51 Terraform plan | Error: GetSecretWithContext failed Service Unavailable
 2024/06/04 18:25:51 Terraform plan | {
 2024/06/04 18:25:51 Terraform plan |     "StatusCode": 503,
 2024/06/04 18:25:51 Terraform plan |     "Headers": {
 2024/06/04 18:25:51 Terraform plan |         "Cache-Control": [
 2024/06/04 18:25:51 Terraform plan |             "no-cache, max-age=0, no-store"
 2024/06/04 18:25:51 Terraform plan |         ],
 2024/06/04 18:25:51 Terraform plan |         "Content-Length": [
 2024/06/04 18:25:51 Terraform plan |             "19"
 2024/06/04 18:25:51 Terraform plan |         ],
 2024/06/04 18:25:51 Terraform plan |         "Content-Security-Policy": [
 2024/06/04 18:25:51 Terraform plan |             "default-src 'none'"
 2024/06/04 18:25:51 Terraform plan |         ],
 2024/06/04 18:25:51 Terraform plan |         "Content-Type": [
 2024/06/04 18:25:51 Terraform plan |             "text/plain"
 2024/06/04 18:25:51 Terraform plan |         ],
 2024/06/04 18:25:51 Terraform plan |         "Date": [
 2024/06/04 18:25:51 Terraform plan |             "Tue, 04 Jun 2024 18:25:51 GMT"
 2024/06/04 18:25:51 Terraform plan |         ],
 2024/06/04 18:25:51 Terraform plan |         "Server": [
 2024/06/04 18:25:51 Terraform plan |             "istio-envoy"
 2024/06/04 18:25:51 Terraform plan |         ]
 2024/06/04 18:25:51 Terraform plan |     },
 2024/06/04 18:25:51 Terraform plan |     "Result": null,
 2024/06/04 18:25:51 Terraform plan |     "RawResult": "bm8gaGVhbHRoeSB1cHN0cmVhbQ=="
 2024/06/04 18:25:51 Terraform plan | }
 2024/06/04 18:25:51 Terraform plan | 
 2024/06/04 18:25:51 Terraform plan | 
 2024/06/04 18:25:51 Terraform plan |   with module.secrets_manager_secret_signing_key[0].ibm_sm_arbitrary_secret.arbitrary_secret[0],
 2024/06/04 18:25:51 Terraform plan |   on .terraform/modules/secrets_manager_secret_signing_key/main.tf line 37, in resource "ibm_sm_arbitrary_secret" "arbitrary_secret":
 2024/06/04 18:25:51 Terraform plan |   37: resource "ibm_sm_arbitrary_secret" "arbitrary_secret" {

• I have also noticed that same resources are still associated with KP instance (which was passed earlier) and nothing is associated with HPCS instance though I have updated the existing_kms_instance_crn to HPCS CRN.

[ocofaigh]

@Ak-sky can we switch focus to getting this to work in the core security services stack (tracked at https://github.com/terraform-ibm-modules/stack-ibm-core-security-services/issues/77)? The genai stack consumes core security services stack, and RAG stack will then consume genai stack so any testing here should be done directly on the core security services stack