We recently saw a failure in our (CF Toolsmiths) CI to tear down a custom GCP environment.
The failure looked something like this:
google_compute_network.pcf-network: The network resource 'projects/myproject/global/networks/my-pcf-network' is already being used by 'projects/myproject/global/addresses/google-managed-services-my-pcf-network'
The google-managed-services-my-pcf-network resource is a global managed service address, which is created by a service external to the VPC (and possibly external to GCP itself). This address may have been created by a Cloud SQL instance with a private IP. These use VPC peering to connect the VPC to the Cloud SQL instance, which creates a dependency of the VPC on the Cloud SQL instance.
The order in which leftovers deletes these items is:
VPC network
global address (this is the link from the Cloud SQL instance to the VPC)
cloud SQL instance
So on the first run, the VPC network deletion will fail because the global address has not been deleted yet. The global address and cloud SQL instance will be deleted.
On the second run, the VPC network, which now has no dependencies, will be deleted.
Steps to replicate this:
create a VPC network named "deleteme" (or similar)
create a Cloud SQL instance named "deleteme" (or similar) with private IP and associated network "deleteme" (or whatever you named your VPC network)
run leftovers --iaas gcp --filter deleteme
Logs from the replication:
2019-06-26 10:08:43 → leftovers --iaas gcp --gcp-service-account-key ~/Downloads/gcp-service-account-key.json --filter deleteme --dry-run
[Subnetwork: deleteme-existing-network-rowan (Network:deleteme-existing-network-rowan)]
[Network: deleteme-existing-network-rowan]
[Global Address: google-managed-services-deleteme-existing-network-rowan]
[SQL Instance: deleteme-cloud-sql-rowan]
2019-06-26 10:11:33 → leftovers --iaas gcp --gcp-service-account-key ~/Downloads/gcp-service-account-key.json --filter deleteme -n
[Subnetwork: deleteme-existing-network-rowan (Network:deleteme-existing-network-rowan)] Deleting...
[Subnetwork: deleteme-existing-network-rowan (Network:deleteme-existing-network-rowan)] Deleted!
[Network: deleteme-existing-network-rowan] Deleting...
[Network: deleteme-existing-network-rowan] Delete: Operation error: The network resource 'projects/pcf-toolsmiths-dev-1/global/networks/deleteme-existing-network-rowan' is already being used by 'projects/pcf-toolsmiths-dev-1/global/addresses/google-managed-services-deleteme-existing-network-rowan'
[Global Address: google-managed-services-deleteme-existing-network-rowan] Deleting...
[Global Address: google-managed-services-deleteme-existing-network-rowan] Deleted!
[SQL Instance: deleteme-cloud-sql-rowan] Deleting...
Waiting 2s before next try.
Waiting 4s before next try.
Waiting 8s before next try.
Waiting 10s before next try.
[SQL Instance: deleteme-cloud-sql-rowan] Deleted!
1 error occurred:
* [Network: deleteme-existing-network-rowan] Delete: Operation error: The network resource 'projects/pcf-toolsmiths-dev-1/global/networks/deleteme-existing-network-rowan' is already being used by 'projects/pcf-toolsmiths-dev-1/global/addresses/google-managed-services-deleteme-existing-network-rowan'
2019-06-26 10:20:16 → leftovers --iaas gcp --gcp-service-account-key ~/Downloads/gcp-service-account-key.json --filter deleteme -n
[Subnetwork: deleteme-existing-network-rowan (Network:deleteme-existing-network-rowan)] Deleting...
[Subnetwork: deleteme-existing-network-rowan (Network:deleteme-existing-network-rowan)] Deleted!
[Network: deleteme-existing-network-rowan] Deleting...
Waiting 2s before next try.
Waiting 4s before next try.
Waiting 8s before next try.
Waiting 10s before next try.
[Network: deleteme-existing-network-rowan] Deleted!
Try leftovers --filter deleteme --dry-run to list remaining resources!
We recently saw a failure in our (CF Toolsmiths) CI to tear down a custom GCP environment.
The failure looked something like this:
The
google-managed-services-my-pcf-networkresource is a global managed service address, which is created by a service external to the VPC (and possibly external to GCP itself). This address may have been created by a Cloud SQL instance with a private IP. These use VPC peering to connect the VPC to the Cloud SQL instance, which creates a dependency of the VPC on the Cloud SQL instance.The order in which leftovers deletes these items is:
So on the first run, the VPC network deletion will fail because the global address has not been deleted yet. The global address and cloud SQL instance will be deleted.
On the second run, the VPC network, which now has no dependencies, will be deleted.
Steps to replicate this:
leftovers --iaas gcp --filter deletemeLogs from the replication: