ibm_resource_instance returning 401 Token expired (intermittent)

ocofaigh commented 2 years ago

We are intermittently seeing errors like:

Error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-12840474957746303036"}

Error: Error retrieving plan: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-11961725898272431839"}

Error: Error retrieving deployment for plan standard : Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-12084307182901635149"}

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

Terraform v1.0.11 ibm-cloud/ibm v1.36.0

Affected Resource(s)

ibm_resource_instance

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please share a link to the ZIP file.

Debug Output

{"id":"crn:v1:bluemix:public:kms:us-south:a/abac0df06b644a9cabc6e44f55b3880e:04196601-1d12-4cd8-9439-b334fd1c60ec::","guid":"04196601-1d12-4cd8-9439-b334fd1c60ec","url":"/v2/resource_instances/04196601-1d12-4cd8-9439-b334fd1c60ec","created_at":"2021-12-13T06:06:57.907283654Z","updated_at":"2021-12-13T06:06:58.885906754Z","deleted_at":null,"created_by":"IBMid-666000KAO3","updated_by":"","deleted_by":"","scheduled_reclaim_at":null,"restored_at":null,"scheduled_reclaim_by":"","restored_by":"","name":"s3backend-test-iak2bl-kms","region_id":"us-south","account_id":"abac0df06b644a9cabc6e44f55b3880e","reseller_channel_id":"","resource_plan_id":"eedd3585-90c6-4c8f-be3d-062069e99fc3","resource_group_id":"1bdebd2987214dfb952ca965f19eb0ce","resource_group_crn":"crn:v1:bluemix:public:resource-controller::a/abac0df06b644a9cabc6e44f55b3880e::resource-group:1bdebd2987214dfb952ca965f19eb0ce","target_crn":"crn:v1:bluemix:public:globalcatalog::::deployment:eedd3585-90c6-4c8f-be3d-062069e99fc3%3Aus-south","allow_cleanup":false,"crn":"crn:v1:bluemix:public:kms:us-south:a/abac0df06b644a9cabc6e44f55b3880e:04196601-1d12-4cd8-9439-b334fd1c60ec::","state":"active","type":"service_instance","sub_type":"kms","resource_id":"ee41347f-b18e-4ca6-bf80-b5467c63f9a6","dashboard_url":"/keyprotect","last_operation":{"type":"create","state":"succeeded","async":false,"description":"Completed create instance operation"},"resource_aliases_url":"/v2/resource_instances/04196601-1d12-4cd8-9439-b334fd1c60ec/resource_aliases","resource_bindings_url":"/v2/resource_instances/04196601-1d12-4cd8-9439-b334fd1c60ec/resource_bindings","resource_keys_url":"/v2/resource_instances/04196601-1d12-4cd8-9439-b334fd1c60ec/resource_keys","plan_history":[{"resource_plan_id":"eedd3585-90c6-4c8f-be3d-062069e99fc3","start_date":"2021-12-13T06:06:57.907283654Z","requestor_id":"IBMid-666000KAO3"}],"migrated":false,"extensions":{"endpoints":{"private":"https://private.us-south.kms.cloud.ibm.com","public":"https://us-south.kms.cloud.ibm.com"}},"controlled_by":"","locked":false}: timestamp=2021-12-13T06:07:59.679Z
2021-12-13T06:07:59.949Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2021/12/13 06:07:59 tagList:  []: timestamp=2021-12-13T06:07:59.948Z
2021-12-13T06:08:00.033Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2021/12/13 06:08:00 tagList:  []: timestamp=2021-12-13T06:08:00.032Z
2021-12-13T06:08:00.041Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2021/12/13 06:08:00 Authentication failed. Trying token refresh: timestamp=2021-12-13T06:08:00.040Z
2021-12-13T06:08:00.041Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2021/12/13 06:08:00 Retrying authentication using API Key: timestamp=2021-12-13T06:08:00.041Z
2021-12-13T06:08:00.110Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2021/12/13 06:08:00 Authentication failed. Trying token refresh: timestamp=2021-12-13T06:08:00.110Z
2021-12-13T06:08:00.243Z [INFO]  provider.terraform-provider-ibm_v1.36.0: 2021/12/13 06:08:00 Retrying authentication using API Key: timestamp=2021-12-13T06:08:00.242Z
2021-12-13T06:08:00.244Z [ERROR] vertex "module.bootstrap[0].ibm_resource_instance.kms_instance" error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-5481268334657177060"}
2021-12-13T06:08:00.244Z [TRACE] vertex "module.bootstrap[0].ibm_resource_instance.kms_instance": visit complete, with errors
2021-12-13T06:08:00.244Z [TRACE] vertex "module.bootstrap[0].ibm_resource_instance.kms_instance": dynamic subgraph encountered errors: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-5481268334657177060"}
2021-12-13T06:08:00.244Z [ERROR] vertex "module.bootstrap[0].ibm_resource_instance.kms_instance" error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-5481268334657177060"}
2021-12-13T06:08:00.245Z [TRACE] vertex "module.bootstrap[0].ibm_resource_instance.kms_instance": visit complete, with errors
2021-12-13T06:08:00.245Z [TRACE] vertex "module.bootstrap.ibm_resource_instance.kms_instance (expand)": dynamic subgraph encountered errors: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-5481268334657177060"}
2021-12-13T06:08:00.245Z [ERROR] vertex "module.bootstrap.ibm_resource_instance.kms_instance (expand)" error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-5481268334657177060"}
2021-12-13T06:08:00.245Z [TRACE] vertex "module.bootstrap.ibm_resource_instance.kms_instance (expand)": visit complete, with errors
2021-12-13T06:08:00.245Z [TRACE] dag/walk: upstream of "module.bootstrap.ibm_kms_key.cos_key (expand)" errored, so skipping
2021-12-13T06:08:00.416Z [ERROR] vertex "module.bootstrap[0].ibm_resource_instance.cos_instance" error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-9684266306605302235"}
2021-12-13T06:08:00.416Z [TRACE] vertex "module.bootstrap[0].ibm_resource_instance.cos_instance": visit complete, with errors
2021-12-13T06:08:00.416Z [TRACE] vertex "module.bootstrap[0].ibm_resource_instance.cos_instance": dynamic subgraph encountered errors: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-9684266306605302235"}
2021-12-13T06:08:00.416Z [ERROR] vertex "module.bootstrap[0].ibm_resource_instance.cos_instance" error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-9684266306605302235"}
2021-12-13T06:08:00.416Z [TRACE] vertex "module.bootstrap[0].ibm_resource_instance.cos_instance": visit complete, with errors
2021-12-13T06:08:00.417Z [TRACE] vertex "module.bootstrap.ibm_resource_instance.cos_instance (expand)": dynamic subgraph encountered errors: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-9684266306605302235"}
2021-12-13T06:08:00.417Z [ERROR] vertex "module.bootstrap.ibm_resource_instance.cos_instance (expand)" error: Error retrieving service offering: Request failed with status code: 401, ServerErrorResponse: {"message":"Token expired","code":401,"global_transaction_id":"gend-9684266306605302235"}
2021-12-13T06:08:00.417Z [TRACE] vertex "module.bootstrap.ibm_resource_instance.cos_instance (expand)": visit complete, with errors
2021-12-13T06:08:00.417Z [TRACE] dag/walk: upstream of "module.bootstrap.ibm_iam_authorization_policy.policy (expand)" errored, so skipping
2021-12-13T06:08:00.417Z [TRACE] dag/walk: upstream of "module.bootstrap.ibm_cos_bucket.state_bucket (expand)" errored, so skipping
2021-12-13T06:08:00.417Z [TRACE] dag/walk: upstream of "module.bootstrap.output.s3_region (expand)" errored, so skipping
2021-12-13T06:08:00.417Z [TRACE] dag/walk: upstream of "module.bootstrap.output.s3_endpoint_private (expand)" errored, so skipping
2021-12-13T06:08:00.417Z [TRACE] dag/walk: upstream of "module.bootstrap.ibm_resource_key.cos_key (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap.output.aws_secret_access_key (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap.output.bucket_name (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap.output.s3_endpoint_public (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap.local.cos_endpoint (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap.null_resource.bootstrap_switch_backend (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/ibm-cloud/ibm\"] (close)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap.output.aws_access_key_id (expand)" errored, so skipping
2021-12-13T06:08:00.418Z [TRACE] dag/walk: upstream of "module.bootstrap (close)" errored, so skipping
2021-12-13T06:08:00.419Z [TRACE] dag/walk: upstream of "null_resource.do_something (expand)" errored, so skipping
2021-12-13T06:08:00.419Z [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/null\"] (close)" errored, so skipping
2021-12-13T06:08:00.419Z [TRACE] dag/walk: upstream of "root" errored, so skipping
2021-12-13T06:08:00.433Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-12-13T06:08:00.436Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/null/3.1.0/linux_amd64/terraform-provider-null_v3.1.0_x5 pid=3900
2021-12-13T06:08:00.436Z [DEBUG] provider: plugin exited
2021-12-13T06:08:00.436Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-12-13T06:08:00.442Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/ibm-cloud/ibm/1.36.0/linux_amd64/terraform-provider-ibm_v1.36.0 pid=3893
2021-12-13T06:08:00.442Z [DEBUG] provider: plugin exited

Full log (stdout + trace): logs.zip

Panic Output

Expected Behavior

No error

Actual Behavior

401 token expired error

Steps to Reproduce

terraform apply

Important Factoids

References

0000

kavya498 commented 2 years ago

@ocofaigh , looks like there was IAM downtime on 13th Dec and that would have resulted this error.. /many has reported similar issues.. Did you encounter this issue on any other date? If so, please provide those logs..

Thanks

ocofaigh commented 2 years ago

Here are the timestamps where we saw this issue (in UTC):

2021-12-09 12:04
2021-12-09 12:04
2021-12-09 12:05
2021-12-13 06:07

Here are the logs from the 9th Dec: observability-instances-module-test-logs-211209-120422.tar.gz

ocofaigh commented 2 years ago

Just letting you know this issue still occurs intermittently. Todays timestamp was 2022-02-11 06:37:31 UTC

IBM-Cloud / terraform-provider-ibm