[BUG]Not able to apply terraform after deleting a subaccount manually from BTP cockpit

TieyanFu commented 9 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

What version of the Terraform provider are you using?

1.0.0.rc1

What version of the Terraform CLI are you using?

1.6.6

What type of issue are you facing

bug report

Describe the bug

I tried to go through sample , and run into a rookie mistake. Basically I tried to terraform apply first, the subaccount got created correctly, however the free-tier services are not entitled in my global account, so I tried to delete the subaccount manually in BTP Cockpit (to have a clean slate), however, in my next try terraform reported that my user does not have sufficient authorisation, although the subaccount is indeed created by my user (through Terraform). The error log is following:

time_sleep.wait_a_few_seconds: Refreshing state... [id=2024-01-08T10:24:59Z]
btp_subaccount.project: Refreshing state... [id=0b358f3c-ca9c-4bf4-8e68-061cf270518a]
btp_subaccount_entitlement.hana-hdi-shared: Refreshing state... [id=hana-hdi-shared]
btp_subaccount_role_collection_assignment.subaccount-admins["tieyan.fu@sap.com"]: Refreshing state... [id=0b358f3c-ca9c-4bf4-8e68-061cf270518a,Subaccount Administrator,tieyan.fu@sap.com]
module.cloudfoundry_environment.null_resource.cache_target_environment: Refreshing state... [id=1875508953518508515]
module.cloudfoundry_environment.btp_subaccount_environment_instance.cf: Refreshing state... [id=3A1B91BA-F2BB-4FC4-B1C0-E2852A6AD55F]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # btp_subaccount.project has been deleted
  - resource "btp_subaccount" "project" {
      - id            = "0b358f3c-ca9c-4bf4-8e68-061cf270518a" -> null
        name          = "d049740-terraform"
        # (9 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement
 <= read (data resources)

Terraform planned the following actions, but then encountered a problem:

  # btp_subaccount.project will be created
  + resource "btp_subaccount" "project" {
      + beta_enabled    = (known after apply)
      + created_by      = (known after apply)
      + created_date    = (known after apply)
      + description     = (known after apply)
      + id              = (known after apply)
      + last_modified   = (known after apply)
      + name            = "d049740_terraform1"
      + parent_features = (known after apply)
      + parent_id       = (known after apply)
      + region          = "eu10"
      + state           = (known after apply)
      + subdomain       = (known after apply)
      + usage           = (known after apply)
    }

  # btp_subaccount_entitlement.bas will be created
  + resource "btp_subaccount_entitlement" "bas" {
      + amount        = (known after apply)
      + category      = (known after apply)
      + created_date  = (known after apply)
      + id            = (known after apply)
      + last_modified = (known after apply)
      + plan_id       = (known after apply)
      + plan_name     = "free"
      + service_name  = "sapappstudio"
      + state         = (known after apply)
      + subaccount_id = (known after apply)
    }

  # btp_subaccount_entitlement.build_workzone will be created
  + resource "btp_subaccount_entitlement" "build_workzone" {
      + amount        = (known after apply)
      + category      = (known after apply)
      + created_date  = (known after apply)
      + id            = (known after apply)
      + last_modified = (known after apply)
      + plan_id       = (known after apply)
      + plan_name     = "free"
      + service_name  = "SAPLaunchpad"
      + state         = (known after apply)
      + subaccount_id = (known after apply)
    }

  # btp_subaccount_entitlement.hana-cloud will be created
  + resource "btp_subaccount_entitlement" "hana-cloud" {
      + amount        = (known after apply)
      + category      = (known after apply)
      + created_date  = (known after apply)
      + id            = (known after apply)
      + last_modified = (known after apply)
      + plan_id       = (known after apply)
      + plan_name     = "hana"
      + service_name  = "hana-cloud"
      + state         = (known after apply)
      + subaccount_id = (known after apply)
    }

  # btp_subaccount_entitlement.hana-cloud-tools will be created
  + resource "btp_subaccount_entitlement" "hana-cloud-tools" {
      + amount        = (known after apply)
      + category      = (known after apply)
      + created_date  = (known after apply)
      + id            = (known after apply)
      + last_modified = (known after apply)
      + plan_id       = (known after apply)
      + plan_name     = "tools"
      + service_name  = "hana-cloud-tools"
      + state         = (known after apply)
      + subaccount_id = (known after apply)
    }

  # btp_subaccount_entitlement.hana-hdi-shared will be created
  + resource "btp_subaccount_entitlement" "hana-hdi-shared" {
      + amount        = (known after apply)
      + category      = (known after apply)
      + created_date  = (known after apply)
      + id            = (known after apply)
      + last_modified = (known after apply)
      + plan_id       = (known after apply)
      + plan_name     = "hdi-shared"
      + service_name  = "hana"
      + state         = (known after apply)
      + subaccount_id = (known after apply)
    }

  # btp_subaccount_role_collection_assignment.Business_Application_Studio_Administrator will be created
  + resource "btp_subaccount_role_collection_assignment" "Business_Application_Studio_Administrator" {
      + id                   = (known after apply)
      + origin               = "ldap"
      + role_collection_name = "Business_Application_Studio_Administrator"
      + subaccount_id        = (known after apply)
      + user_name            = "tieyan.fu@sap.com"
    }

  # btp_subaccount_role_collection_assignment.Business_Application_Studio_Developer will be created
  + resource "btp_subaccount_role_collection_assignment" "Business_Application_Studio_Developer" {
      + id                   = (known after apply)
      + origin               = "ldap"
      + role_collection_name = "Business_Application_Studio_Developer"
      + subaccount_id        = (known after apply)
      + user_name            = "tieyan.fu@sap.com"
    }

  # btp_subaccount_role_collection_assignment.launchpad_admin will be created
  + resource "btp_subaccount_role_collection_assignment" "launchpad_admin" {
      + id                   = (known after apply)
      + origin               = "ldap"
      + role_collection_name = "Launchpad_Admin"
      + subaccount_id        = (known after apply)
      + user_name            = "tieyan.fu@sap.com"
    }

  # btp_subaccount_role_collection_assignment.subaccount-admins["tieyan.fu@sap.com"] must be replaced
-/+ resource "btp_subaccount_role_collection_assignment" "subaccount-admins" {
      ~ id                   = "0b358f3c-ca9c-4bf4-8e68-061cf270518a,Subaccount Administrator,tieyan.fu@sap.com" -> (known after apply)
      ~ subaccount_id        = "0b358f3c-ca9c-4bf4-8e68-061cf270518a" # forces replacement -> (known after apply) # forces replacement
        # (3 unchanged attributes hidden)
    }

  # btp_subaccount_subscription.bas-subscribe will be created
  + resource "btp_subaccount_subscription" "bas-subscribe" {
      + additional_plan_features    = (known after apply)
      + app_id                      = (known after apply)
      + app_name                    = "sapappstudio"
      + authentication_provider     = (known after apply)
      + category                    = (known after apply)
      + commercial_app_name         = (known after apply)
      + created_date                = (known after apply)
      + customer_developed          = (known after apply)
      + description                 = (known after apply)
      + display_name                = (known after apply)
      + formation_solution_name     = (known after apply)
      + globalaccount_id            = (known after apply)
      + id                          = (known after apply)
      + labels                      = (known after apply)
      + last_modified               = (known after apply)
      + parameters                  = jsonencode({})
      + plan_name                   = "free"
      + platform_entity_id          = (known after apply)
      + quota                       = (known after apply)
      + state                       = (known after apply)
      + subaccount_id               = (known after apply)
      + subscribed_subaccount_id    = (known after apply)
      + subscribed_tenant_id        = (known after apply)
      + subscription_url            = (known after apply)
      + supports_parameters_updates = (known after apply)
      + supports_plan_updates       = (known after apply)
      + tenant_id                   = (known after apply)
    }

  # btp_subaccount_subscription.build_workzone_subscribe will be created
  + resource "btp_subaccount_subscription" "build_workzone_subscribe" {
      + additional_plan_features    = (known after apply)
      + app_id                      = (known after apply)
      + app_name                    = "SAPLaunchpad"
      + authentication_provider     = (known after apply)
      + category                    = (known after apply)
      + commercial_app_name         = (known after apply)
      + created_date                = (known after apply)
      + customer_developed          = (known after apply)
      + description                 = (known after apply)
      + display_name                = (known after apply)
      + formation_solution_name     = (known after apply)
      + globalaccount_id            = (known after apply)
      + id                          = (known after apply)
      + labels                      = (known after apply)
      + last_modified               = (known after apply)
      + parameters                  = jsonencode({})
      + plan_name                   = "free"
      + platform_entity_id          = (known after apply)
      + quota                       = (known after apply)
      + state                       = (known after apply)
      + subaccount_id               = (known after apply)
      + subscribed_subaccount_id    = (known after apply)
      + subscribed_tenant_id        = (known after apply)
      + subscription_url            = (known after apply)
      + supports_parameters_updates = (known after apply)
      + supports_plan_updates       = (known after apply)
      + tenant_id                   = (known after apply)
    }

  # btp_subaccount_subscription.hana-cloud-tools will be created
  + resource "btp_subaccount_subscription" "hana-cloud-tools" {
      + additional_plan_features    = (known after apply)
      + app_id                      = (known after apply)
      + app_name                    = "hana-cloud-tools"
      + authentication_provider     = (known after apply)
      + category                    = (known after apply)
      + commercial_app_name         = (known after apply)
      + created_date                = (known after apply)
      + customer_developed          = (known after apply)
      + description                 = (known after apply)
      + display_name                = (known after apply)
      + formation_solution_name     = (known after apply)
      + globalaccount_id            = (known after apply)
      + id                          = (known after apply)
      + labels                      = (known after apply)
      + last_modified               = (known after apply)
      + parameters                  = jsonencode({})
      + plan_name                   = "tools"
      + platform_entity_id          = (known after apply)
      + quota                       = (known after apply)
      + state                       = (known after apply)
      + subaccount_id               = (known after apply)
      + subscribed_subaccount_id    = (known after apply)
      + subscribed_tenant_id        = (known after apply)
      + subscription_url            = (known after apply)
      + supports_parameters_updates = (known after apply)
      + supports_plan_updates       = (known after apply)
      + tenant_id                   = (known after apply)
    }

  # module.cloudfoundry_environment.data.btp_subaccount_environments.all will be read during apply
  # (config refers to values not yet known)
 <= data "btp_subaccount_environments" "all" {
      + id            = (known after apply)
      + subaccount_id = (known after apply)
      + values        = (known after apply)
    }

Plan: 13 to add, 0 to change, 1 to destroy.
╷
│ Error: API Error Reading Resource Environment Instance (Subaccount)
│
│   with module.cloudfoundry_environment.btp_subaccount_environment_instance.cf,
│   on ../../../modules/environment/cloudfoundry/envinstance_cf/envinstance_cf.tf line 40, in resource "btp_subaccount_environment_instance" "cf":
│   40: resource "btp_subaccount_environment_instance" "cf" {
│
│ Access forbidden due to insufficient authorization. Make sure to have sufficient access rights. [Status: 403; Correlation ID: 3d142ef8-a164-6096-1f4d-9aaa9015443f]

I tried with "Terraform init" hoping to reinitialise everything locally, but clearly the server side is not cleaned up. So now I am not able to run terraform apply. How can I rectify the error? I assume it could happen that user changes the configuration (e.g. delete subaccount or service in BTP Cockpit), what is the expected behavior and how to rectify the situation?

Expected Behavior

Is above error expected? What is the step to rectify the situation?

Steps To Reproduce

Download the sample and follow the setup
Execute the terraform init command with free-tier plan for cloud foundry
Execute terraform apply.
Delete the subaccount and try to run terraform apply again.
rerun should encounter above error.

User's Role Collections

-Subaccount Admin

Add screenshots to help explain your problem

No response

Additional context

No response

lechnerc77 commented 9 months ago

Hi @TieyanFu ,

in general if you aim to manage the infrastructure via Terraform it is not recommended to manipulate the managed infrastructure manually as this causes state deviations. Especially if you delete some managed infrastructure the Terraform planning might cause issues in order to align the setup stored in the state and the one on the platform. The solution to resolve these issues depends on the scenario. I think in your setup a terraform destroy might have put you back to start.

Your state seems to try to delete and recreate role collection assignment which causes some errors down the line when an environment instance should be created. Be aware that if you are the one that executed the TF script, your user will automatically assigned as subaccount admin. Do not add yourself explicitly via the resource btp_subaccount_role_collection_assignment as subaccount admin, only assign this role to additional users. Otherwise follow-up errors will occur that would enforce you to make adjustments of your state file. See also this issue comment.

I do not know how our current Terraform state and your resources on BTP look like, but I would recommend the following procedure:

Destroy the resources you want to manage on SAP BTP
Delete your local Terraform state (*.tfstate*)
Adjust the role-collection assignment, so that the person authentication with the TF provider does not get assigned the subaccount admin role via the corresponding Terraform resource.
Execute a terraform plan to validate that the setup gets created from scratch. The planning should result in resources to be added, but no resources to be changed or deleted.

Please let me know if this works.

TieyanFu commented 9 months ago

Thanks @lechnerc77 , your feedback definitely describes the best practice. After deleting the state files, the apply works.

SAP / terraform-provider-btp