search

terraform-google-modules / terraform-google-project-factory

Creates an opinionated Google Cloud project by using Shared VPC, IAM, and Google Cloud APIs
https://registry.terraform.io/modules/terraform-google-modules/project-factory/google
Apache License 2.0
826 stars 535 forks source link

[Project factory] API dependency causing issues in terraform destroy #605

Closed vishramyadav-g closed 3 years ago

vishramyadav-g commented 3 years ago

Description

Project factory module has the option to provide APIs to be enabled. When dependent and parent APIs (which have dependent APIs) are provided as input, running terraform destroy fails.

Error

Error: Error when reading or editing Project Service prj-busunit-code-tool-d-e7d6/storage-api.googleapis.com: Error disabling service "storage-api.googleapis.com" for project "prj-busunit-code-tool-d-e7d6": Error waiting for api to disable: Error code 5, message: [Hook call/poll failed for service "file.googleapis.com".

Other details:

Terraform version: 0.13.6 Google provider version: google v3.61.0, google-beta v3.61.0 CFT version: 11.1.0 ( Can be reproduced with 10.3.2 as well, seems version agnostic ) Activated API: "bigquery.googleapis.com", "bigquerystorage.googleapis.com", "cloudapis.googleapis.com", "cloudbilling.googleapis.com", "cloudbuild.googleapis.com", "cloudkms.googleapis.com", "cloudresourcemanager.googleapis.com", "compute.googleapis.com", "container.googleapis.com", "containerregistry.googleapis.com", "deploymentmanager.googleapis.com", "dns.googleapis.com", "eventarc.googleapis.com", "file.googleapis.com", "iam.googleapis.com", "iamcredentials.googleapis.com", "iap.googleapis.com", "logging.googleapis.com", "monitoring.googleapis.com", "networkmanagement.googleapis.com", "oslogin.googleapis.com", "run.googleapis.com", "runtimeconfig.googleapis.com", "secretmanager.googleapis.com", "securitycenter.googleapis.com", "servicemanagement.googleapis.com", "serviceusage.googleapis.com", "stackdriver.googleapis.com", "storage-api.googleapis.com", "storage-component.googleapis.com", "storage.googleapis.com", "vpcaccess.googleapis.com", "websecurityscanner.googleapis.com",

Steps to Reproduce:

  1. Configure project factory module to create a project with dependent and independent APIs.
  2. Run terraform apply
  3. Run terraform destroy (On created project)

Debug log:

Github gist : `Debug logs`

vishramyadav-g commented 3 years ago

Github gist : Debug logs

morgante commented 3 years ago

Unfortunately there's not any easy way for us to track which services are dependent and therefore disable in the proper order. The three workarounds I suggest are:

  1. Only list top-level services in the API list (ie. not the services they are dependent on)
  2. Set disable_services_on_destroy = false
  3. Run terraform destroy multiple times. It should eventually converge.
bharathkkb commented 3 years ago

In the provider we do a retry for service activation, maybe worth opening an issue in the provider repo to retry this error a few times. This will get reduce the need to run destroy multiple times to converge.

maitreya-source commented 3 years ago

Great idea @bharathkkb, Thanks. Opened following issue in provider : Provider issue #9637.

bharathkkb commented 3 years ago

closing this since this needs to be addressed in the provider