"invalid bearer token" error while deploying SaaS DA repo.

[rajatagarwal-ibm]

Hitting the invalid bearer token error for SaaS DA deployment in the stack.

Error log:

2024/05/02 11:11:41 Terraform apply | Error: unexpected response code '401': Invalid bearer token: Access token is invalid.
2024/05/02 11:11:41 Terraform apply | 
2024/05/02 11:11:41 Terraform apply |   with module.configure_project.restapi_object.configure_project[0],
2024/05/02 11:11:41 Terraform apply |   on configure_project/main.tf line 3, in resource "restapi_object" "configure_project":
2024/05/02 11:11:41 Terraform apply |    3: resource "restapi_object" "configure_project" {
2024/05/02 11:11:41 Terraform apply | 

Steps to reproduce:

1. Follow the steps mentioned in this readme - https://github.com/terraform-ibm-modules/stack-retrieval-augmented-generation
2. If not passing an actual value for watsonx_admin_api_key, it failed with the above error.

Below are the scenarios I have tested:

• FAILED when setting "watsonx_admin_api_key": null,
• FAILED when omitting watsonx_admin_api_key from the .def.json file.
• FAILED when setting same value as ibmcloud_api_key.

P.S. - I tried to deploy it in the GE Ops account.

[Ak-sky]

Looks like Bearer token was missing when passed to the these 2 shell scripts- https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/blob/78c16f84a3fbfd3b4dd8ec26e9e780fdb1e89851/configure_user/scripts/add_user.sh and https://github.com/terraform-ibm-modules/terraform-ibm-watsonx-saas-da/blob/78c16f84a3fbfd3b4dd8ec26e9e780fdb1e89851/configure_user/scripts/enforce_account_restriction.sh

and jq was failing due to this.

[ocofaigh]

@brendankellyibm could this be the issue I explained was going to happen sooner or later where the token has expired by the time the script try to run them? We have some people testing now in accounts where the token is valid for a short period of time (maybe 60mins?)

[brendankellyibm]

How long was the DA running for?

[Ak-sky]

• After some debugging what I noticed is missing word "Bearer" in Bearer <TOKEN> while running the cmd configure_user/scripts/add_user.sh causing the shell script to fail (<stdin>:1): Malformed BOM (while parsing '��')) at this line and this line.
• Also failure (401] Unauthorized: Invalid bearer token: Access token is invalid) at POST API call due to the wrong TOKEN generated here.

[vburckhardt]

Do we have a theory why this happens only for some users or some runs? I do not think I've seen this one - deploying 1-2 times a day for the past months or so. I use the same values for ibmcloud_api_key and watsonx_admin_api_key in my testing.

[ocofaigh]

Stale - closing for now - not been reproduced in number of runs.