[BUG] Rate limit exceeded errors still occurr

holgpar commented 1 month ago

Is there an existing issue for this?

[X] #903

What version of the Terraform provider are you using?

1.6.0

What version of the Terraform CLI are you using?

tofu v1.7.2

What type of issue are you facing

bug report

Describe the bug

We still see failing plan commands with outputs like this from time to time:

│ Error: API Error Reading Resource Entitlement (Subaccount) │ │ with module.cloudci.module.sap-btp-entitlements.btp_subaccount_entitlement.entitlement["cis-xrs"], │ on .terraform/modules/cloudci.sap-btp-entitlements/main.tf line 21, in resource "btp_subaccount_entitlement" "entitlement": │ 21: resource "btp_subaccount_entitlement" "entitlement" { │ │

I think this is unexpected, because there was a recent change in the provider to add some retry logic.

Expected Behavior

Rate Limits are handled by the provider via back-off and retrying.

Steps To Reproduce

Execute a "plan" against a subaccount with a configuration containing about 20 entitlements. We currently execute three such plans in parallel.

User's Role Collections

No response

Add screenshots to help explain your problem

No response

Additional context

No response

lechnerc77 commented 1 month ago

The adjustment we made was for the CREATE, UPDATE and DELETE operations which explains why the error occurs now during the READ.

Did you make any changes with respect to the parallelism (parameter [-parallelism=n](https://developer.hashicorp.com/terraform/cli/commands/plan#parallelism-n))?
Is it canary or live landscape?

holgpar commented 1 month ago

The adjustment we made was for the CREATE, UPDATE and DELETE operations which explains why the error occurs now during the READ.

yeah, that figures.

Did you make any changes with respect to the parallelism.

Nope, did not touch that.

Is it canary or live landscape?

It happens on live landscapes (but we also have much more regions on live landscapes).

lechnerc77 commented 1 month ago

[x] Loop that we have in the modifying operations of the resource needs to be transferred to the READ operation in the resource i.e. the complete GET call must be embedded in such a loop
[x] Reaction on throttling error must be implemented accordingly

lechnerc77 commented 1 month ago

Attempts to reproduce the error on prod landscape with identical set of entitlements as well as leveraging the module sap-btp-entitlements:

Local execution for 4 subaccounts in parallel via terramate orchestrating the Terraform CLI -> no error
Local execution for 4 subaccounts in parallel via terramate orchestrating the OpenTofu CLI -> no error
Execution via GH Action workflow for 4 subaccounts in parallel via terramate orchestrating the Terraform CLI -> no error (link)
Execution via GH Action workflow for 4 subaccounts in parallel via terramate orchestrating the OpenTofu CLI -> no error (link)
Execution via GH Action workflow and parallel execution via matrix -> no error (link)

For reference see: https://github.com/lechnerc77/tf-btp-loadtest

lechnerc77 commented 1 month ago

Update: Parallel execution of apply and destroy with same setup in region eu10 -> the destroy has 10 parallel reads on the entitlements. Execution was successful see https://github.com/lechnerc77/tf-btp-loadtest/actions/runs/10843825292

lechnerc77 commented 1 month ago

Update: Parallel execution of apply and destroy with same setup in region AP10 -> the destroy has 10 parallel reads on the entitlements. Execution was successful see https://github.com/lechnerc77/tf-btp-loadtest/actions/runs/10850499262

CHERIANS commented 1 month ago

@holgpar we had the ver 1.7.0 of Terraform provider for BTP released yesterday which contains the fix

SAP / terraform-provider-btp