search

vmware-archive / terraforming-gcp

use terraform, deploy yourself a pcf
Apache License 2.0
71 stars 87 forks source link

GCP - handle "Error 409: There were concurrent policy changes." errors #132

Closed markstokan closed 5 years ago

markstokan commented 5 years ago

While destroying a GCP PKS deployment with three service account (opsman, pks-worker-node, pks-master-node), we found we had to run leftovers v0.48.0 three times to delete all of the service accounts. Only one service account was deleted per run.

The errors we received were:

IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-opsman@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Deleting...
[IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-pks-worker-node@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Deleting...
[IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-pks-master-node@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Deleting...
[IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-opsman@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Remove IAM Policy Bindings: Set Project IAM Policy: googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted
[IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-pks-master-node@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Remove IAM Policy Bindings: Set Project IAM Policy: googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted
[IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-pks-worker-node@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Deleted!
[DNS Managed Zone: grape-staging-zone] Deleting...
[DNS Managed Zone: grape-staging-zone] Deleted!

2 errors occurred:
    * [IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-opsman@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Remove IAM Policy Bindings: Set Project IAM Policy: googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted
    * [IAM Service Account: projects/pcf-toolsmiths-dev-1/serviceAccounts/grape-staging-pks-master-node@pcf-toolsmiths-dev-1.iam.gserviceaccount.com] Remove IAM Policy Bindings: Set Project IAM Policy: googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted

Would it be possible for leftovers to add retry logic in the case of a 409?

genevieve commented 5 years ago

@markstokan Do you want to open this as an issue here https://github.com/genevieve/leftovers/issues?

markstokan commented 5 years ago

No idea why I opened this in the "terraforming-gcp" repo. Closing and re-opening in leftovers.