Deleting a project that has manually deployed resources creates an undeleteable zombie project

periwinkleFTW commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.4.6 on darwin_amd64

provider registry.terraform.io/hashicorp/google v4.59.0
provider registry.terraform.io/hashicorp/google-beta v4.59.0
provider registry.terraform.io/hashicorp/null v3.2.1
provider registry.terraform.io/hashicorp/random v3.4.3
provider registry.terraform.io/hashicorp/time v0.9.1

Affected Resource(s)

google_apigateway*

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.

Debug Output

I dont know how to get debug output on Terraform Cloud

Panic Output

Error: Error when reading or editing Api: googleapi: Error 400: Resource '"projects/andrey-dev-project-07dd/locations/global/apis/andrey-dev-seeder-api-07dd"' has nested resources. If the API supports cascading delete, set 'force' to true to delete it and its nested resources.

Expected Behavior

I expected an error before terraform starts deleting resources and APIs or for terraform to nuke the project and all the resources. I would be ok with either option

Actual Behavior

Terraform deletes a lot of resources and disables a lot of APIs. It is impossible to get back into the project because even attempting to deploy IAM permissions for my account errors out

Steps to Reproduce

Deploy a project, enable API gateway service. Deploy API using terraform, deploy config and gateway manually. Run destroy

terraform apply

Important Factoids

After this error a zombie project is created. It is impossible to delete this project because console shows Unknown Error pop up when trying to delete the project (I have Project Deleter perms in the org).

I use TF Cloud to manage infrastructure.

Terraform configuration handles project creation, enabling of API, and deployment of resources.

Similar deletion behaviour occurred with VPC (https://issuetracker.google.com/issues/186792016) but this bug is resolved by nuking the whole project (sometimes takes 2-3 tries to delete the project though)

References

0000

b/299600741

edwardmedia commented 1 year ago

@periwinkleFTW not sure how much the provider can do for this. This is how the gcp internal works. Unless we know exactly what to send to the api, there is little the provider can do.

periwinkleFTW commented 1 year ago

@melinath seems like it's not a provider issue

melinath commented 1 year ago

Just to make sure I understand this right, it sounds like the problem is that the API gateway can't be deleted because it has things inside it which would need to be deleted first (or we would need to either always force a cascading delete or provide a deletion_policy field).

But this error doesn't show up until Terraform tries to delete the project containing the API gateway, and that makes the project deletion fail.

And at that point, the project becomes completely inaccessible, either for changing IAM permissions or deleting it, because of an "Unknown Error", even for users in Cloud Console who have appropriate permissions to access, modify, or delete projects in the entire org.

Is that accurate?

periwinkleFTW commented 1 year ago

@melinath that is correct

periwinkleFTW commented 1 year ago

@melinath @edwardmedia I have an update. It seems that destroying the project with api gateway resources that are not managed by terraform creates a zombie project because terraform disables GCP API services that API gateway relies on while api config/gateway are still there. I managed to get inside the zombie and when going into the 'API Gateway' I saw that 2 out of 3 GCP API services were disabled (bottom ones). Upon enabling those API services I was able to manually delete the API gateway

hashicorp / terraform-provider-google