feat: add retry logic for rate limit for entitlement

[lechnerc77]

Purpose

• Add retry logic in case of rate limiting to the following resources:
  • subaccount_entitlement
  • subaccount_service_instance
  • subaccount_subscription
• Increase waiting time for entitlement assignment/creation and deletion to avoid unnecessary retries when hitting a rate limit
• Closes #872

Does this introduce a breaking change?

[ ] Yes
[X] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[X] Feature
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

• Test the code via automated test

go test ./...

• Additional test steps

Execute assignment of entitlements in parallel to hit rate limit error

What to Check

Verify that the following are valid:

• Automated tests are executed successfully
• Verify that rate limit error is mitigated by retries

Other Information

• Adding the retry context which anyway wraps the state change context is not necessary as we can add the logic to the existing waiting logic

Checklist for reviewer

The following organizational tasks must be completed before merging this PR:

• [X] The PR is assigned to the Terraform project and a status is set (typically "in review").
• [X] The PR has the matching labels assigned to it.
• [X] The PR has a milestone assigned to it.
• [X] If the PR closes an issue, the issue is referenced.
• [X] Possible follow-up items are created and linked.

[lechnerc77]

Sonar Cloud fails, which is acceptable, as we cannot test the rate limit error with our test fixture recordings

[sp1goyal]

Hello Christian

We tried to reproduce the issue by adding 19 entitlements to 6 sub-accounts and then 12 sub-accounts, in both the cases the entitlements are being added and the rate limit issue couldn't be reproduced. Attaching the configuration for your reference.

variable "entitlements" {
  type        = map(list(string))
  description = "Map with all expected entitlements"
  default = {
    # Alert Notification Service
    "alert-notification" = ["standard", "lite", "free"]

    # Audit Log Viewer Service
    "auditlog-viewer" = ["free", "default"]

    # BTP LLM Proxy
    "azure-openai-service-demo" = ["default"]

    # Cloud Management Service
    "cis" = ["local", "cloud-automation", "central-viewer", "local-viewer"]

    # Cloud Portal Service
    "PortalApplication" = ["standard"]

    # Event Mesh
    "enterprise-messaging" = ["default"]

    # Forms Service by Adobe
    "ads" = ["free", "standard"]

    # SAP AI Core
    "aicore" = ["standard"]

    # SAP HANA Schemas & HDI Containers
    "hana" = ["hdi-shared"]

    #SAPLaunchpad
    "SAPLaunchpad" = ["foundation","standard"]
  }
}

Attaching the terraform script as well to run this configuration via the module mentioned in the link.

resource "btp_subaccount" "my_project" {
  count = 12
  name      = "sarthak_test${count.index+1}"
  subdomain = "sgoyal-${count.index+1}"
  region    = "eu12"
}

module "sap-btp-entitlements" {
  count = 12
  source  = "aydin-ozcan/sap-btp-entitlements/btp"
  version = "1.0.1"
  subaccount = btp_subaccount.my_project[count.index].id
  entitlements = var.entitlements
}

Please let us know if anything needs to be updated so as to reproduce the issue. Thanks & Regards Sarthak Goyal