[BUG] Undefined API error when deleting subaccount entitlements

[mpern]

Is there an existing issue for this?

• [X] I have searched the existing issues

What version of Terraform are you using?

1.5.7

What type of issue are you facing

bug report

Describe the bug

Destroying a subaccount fails when deleting the CF runtime entitlement.

Expected Behavior

Destroy "just works'

Steps To Reproduce

1. provision a new subaccount with an entitlement to CF runtime, e.g.
2. delete the subaccount or change the region to force a recreate
3. :boom: entitlement cannot be deleted

module.foo.btp_subaccount_entitlement.entitlements["runtime"]: Destroying... [id=applicationruntime]
module.foo.btp_subaccount_entitlement.entitlements["runtime"]: Still destroying... [id=applicationruntime, 10s elapsed]
module.foo.btp_subaccount_entitlement.entitlements["runtime"]: Still destroying... [id=applicationruntime, 20s elapsed]
╷
│ Error: API Error Deleting Resource Entitlement (Subaccount)
│
│ undefined API error during entitlement processing
╵

Add screenshots to help explain your problem

No response

Additional context

No response

[mpern]

trace log output, I hope it helps

2023-11-03T09:45:54.061+0100 [TRACE] dag/walk: vertex "module.foo.btp_subaccount_entitlement.entitlements[\"credstore\"]" is waiting for "module.foo.btp_subaccount.frontliners"
2023-11-03T09:45:57.131+0100 [TRACE] provider.terraform-provider-btp_v0.5.0-beta1: Called provider defined Resource Delete: @caller=github.com/hashicorp/terraform-plugin-framework@v1.4.0/internal/fwserver/server_deleteresource.go:91 @module=sdk.framework tf_resource_type=btp_subaccount_entitlement tf_provider_addr=registry.terraform.io/sap/btp tf_req_id=8ef71ed6-4654-46be-9cc3-4dcf72d14361 tf_rpc=ApplyResourceChange timestamp=2023-11-03T09:45:57.131+0100
2023-11-03T09:45:57.131+0100 [TRACE] provider.terraform-provider-btp_v0.5.0-beta1: Received downstream response: @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov6/internal/tf6serverlogging/downstream_request.go:40 diagnostic_error_count=1 tf_resource_type=btp_subaccount_entitlement tf_req_duration_ms=28069 tf_req_id=8ef71ed6-4654-46be-9cc3-4dcf72d14361 tf_rpc=ApplyResourceChange @module=sdk.proto diagnostic_warning_count=0 tf_proto_version=6.4 tf_provider_addr=registry.terraform.io/sap/btp timestamp=2023-11-03T09:45:57.131+0100
2023-11-03T09:45:57.131+0100 [ERROR] provider.terraform-provider-btp_v0.5.0-beta1: Response contains error diagnostic: @module=sdk.proto tf_provider_addr=registry.terraform.io/sap/btp tf_resource_type=btp_subaccount_entitlement tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov6/internal/diag/diagnostics.go:58 diagnostic_detail="undefined API error during entitlement processing" diagnostic_severity=ERROR diagnostic_summary="API Error Deleting Resource Entitlement (Subaccount)" tf_proto_version=6.4 tf_req_id=8ef71ed6-4654-46be-9cc3-4dcf72d14361 timestamp=2023-11-03T09:45:57.131+0100
2023-11-03T09:45:57.131+0100 [TRACE] provider.terraform-provider-btp_v0.5.0-beta1: Served request: tf_resource_type=btp_subaccount_entitlement @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov6/tf6server/server.go:870 tf_provider_addr=registry.terraform.io/sap/btp tf_req_id=8ef71ed6-4654-46be-9cc3-4dcf72d14361 tf_rpc=ApplyResourceChange @module=sdk.proto tf_proto_version=6.4 timestamp=2023-11-03T09:45:57.131+0100
2023-11-03T09:45:57.132+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.foo.btp_subaccount_entitlement.entitlements["runtime"]
2023-11-03T09:45:57.132+0100 [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.foo.btp_subaccount_entitlement.entitlements["runtime"]
2023-11-03T09:45:57.132+0100 [DEBUG] State storage *remote.State declined to persist a state snapshot
2023-11-03T09:45:57.132+0100 [ERROR] vertex "module.foo.btp_subaccount_entitlement.entitlements[\"runtime\"] (destroy)" error: API Error Deleting Resource Entitlement (Subaccount)
2023-11-03T09:45:57.132+0100 [TRACE] vertex "module.foo.btp_subaccount_entitlement.entitlements[\"runtime\"] (destroy)": visit complete, with errors
2023-11-03T09:45:57.132+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount.frontliners (destroy)" errored, so skipping
2023-11-03T09:45:57.132+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount.frontliners" errored, so skipping
2023-11-03T09:45:57.132+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i545703\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i319024\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"application-logs\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i555107\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"auditlog-management\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"auditlog-viewer\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i327130\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.output.subaccount (expand)" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"credstore\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i073468\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"kyma-runtime\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"credstore-proxy\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i554734\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_environment_instance.kyma" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"xsuaa-broker\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i550468\"]" errored, so skipping
2023-11-03T09:45:57.133+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_role_collection_assignment.admins[\"i066492\"]" errored, so skipping
2023-11-03T09:45:57.134+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"xsuaa-application\"]" errored, so skipping
2023-11-03T09:45:57.134+0100 [TRACE] dag/walk: upstream of "module.foo.btp_subaccount_entitlement.entitlements[\"saas-registry\"]" errored, so skipping
2023-11-03T09:45:57.134+0100 [TRACE] dag/walk: upstream of "output.frontliners (expand)" errored, so skipping
2023-11-03T09:45:57.134+0100 [TRACE] dag/walk: upstream of "module.foo (close)" errored, so skipping
2023-11-03T09:45:57.134+0100 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/sap/btp\"] (close)" errored, so skipping
2023-11-03T09:45:57.134+0100 [TRACE] dag/walk: upstream of "root" errored, so skipping

[v0lkc]

@mpern could you please do me a favour and paste the code snippet which you use to set the entitlement?

[mpern]

Sure, here you go (reduced to the failing entitlement):

resource "btp_subaccount" "foo" {
  name      = "foo"
  subdomain = "foo"
  region    = "eu12"
}

locals {
  entitlements = {
    runtime = {
      service = "APPLICATION_RUNTIME"
      plan    = "MEMORY"
      quota   = 5
    }
  }
}

resource "btp_subaccount_entitlement" "entitlements" {
  for_each = local.entitlements

  subaccount_id = btp_subaccount.foo.id
  service_name  = each.value.service
  plan_name     = each.value.plan
  amount        = each.value.quota
}

[lechnerc77]

Retest with referenced shortened script was successful - entitlement could be deleted without issues.

[lechnerc77]

Scheduled meeting to check config to make error reproduceable

[lechnerc77]

@mpern Thanks for sharing the repository for your setup. I could finally reproduce the error. The error arises due to a dependency between the APPLICATION_RUNTIME and the Cloud Foundry environment you create in your original setup.

For the setup to work consistently you must model the dependency between this specific entitlement and the creation of the environment instance of Cloud Foundry.

Enhancing your reduced setup the following configurations works:

resource "btp_subaccount" "foo" {
  name      = "retest-issue-511"
  subdomain = "retest-issue-511"
  region    = "eu12"
}

locals {
  entitlements = {
    runtime = {
      service = "APPLICATION_RUNTIME"
      plan    = "MEMORY"
      quota   = 1
    }
  }
}

resource "btp_subaccount_entitlement" "entitlements" {
  for_each = local.entitlements

  subaccount_id = btp_subaccount.foo.id
  service_name  = each.value.service
  plan_name     = each.value.plan
  amount        = each.value.quota
}

data "btp_subaccount_environments" "all" {
  subaccount_id = btp_subaccount.foo.id
}

resource "null_resource" "cache_target_environment" {
  triggers = {
    label = [for env in data.btp_subaccount_environments.all.values : env if env.service_name == "cloudfoundry" && env.environment_type == "cloudfoundry"][0].landscape_label
  }

  lifecycle {
    ignore_changes = all
  }
}

resource "btp_subaccount_environment_instance" "cloudfoundry" {
  depends_on = [btp_subaccount_entitlement.entitlements]

  subaccount_id    = btp_subaccount.foo.id
  name             = "issue-511-cf-environment"
  environment_type = "cloudfoundry"
  service_name     = "cloudfoundry"
  plan_name        = "standard"
  landscape_label  = null_resource.cache_target_environment.triggers.label
  # some regions offer multiple environments of a kind and you must explicitly select the target environment in which
  # the instance shall be created. 
  # available environments can be looked up using the btp_subaccount_environments datasource
  parameters = jsonencode({
    instance_name = "issue-511-cf-org-name"
  })
}

This corresponds to the situation of creating a service instance where you must model the dependency between the entitlement of the service and the creation of the service instance.

Can you please validate if this resolves your issue?

[github-actions[bot]]

This issue is stale because it has been open 15 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.