`google_service_networking_connection` produce an error when trying to destroy the resource

botzkobg commented 1 month ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.8 on macOS Sonoma 14.6.1

provider registry.terraform.io/hashicorp/google v6.7.0

Affected Resource(s)

google_service_networking_connection

Terraform Configuration

data "google_compute_network" "cloud_run_vpc_default" {
  project = var.cloudRunProjectId
  name    = "default"
}

resource "google_compute_global_address" "private_ip_address" {
  name          = "private-ip-address"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 24
  network       = data.google_compute_network.cloud_run_vpc_default.id
}

resource "google_service_networking_connection" "default" {
  network                 = data.google_compute_network.cloud_run_vpc_default.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_address.name]
}

Debug Output

N/A

Expected Behavior

Running terraform destroy should successfully destroy the resources created by google_service_networking_connection

Actual Behavior

Running terraform destroy produces error

Error: Unable to remove Service Networking Connection, err: Error waiting for Delete Service Networking Connection: Error code 9, message: Failed to delete connection; Producer services (e.g. CloudSQL, Cloud Memstore, etc.) are still using this connection.
Help Token: AaUeqG5HQE87q1sKZj4eZ-XYOvVYDisRlH0rqXzhPJa0Kh51_TxkRA3JQ-_jDLsIGFakZdm49gX_OJ3YHjbY-sbBwXWawtAxE73FS2bR1ARmc7bJ

Steps to reproduce

terraform plan -var-file=vars.tfvars.json -out=execution.tfplan
terraform apply execution.tfplan
terraform destroy -var-file=vars.tfvars.json

Important Factoids

No response

References

No response

b/376331301

botzkobg commented 1 month ago

After terraform apply is executed servicenetworking-googleapis-com is visible in VPC Networks -> default -> Private services access -> Private connections to services

Additionally servicenetworking-googleapis-com is visible in VPC Network Peering

When you execute terraform destroy you will receive the error message reported above.

Using the UI I can successfully delete servicenetworking-googleapis-com from VPC Network Peering without any issues. This will also remove it from VPC Networks -> default -> Private services access -> Private connections to services . At that point terraform destroy will say that only the google_compute_global_address. private_ip_address needs to be destroyed.

However if I try to delete servicenetworking-googleapis-com from VPC Networks -> default -> Private services access -> Private connections to services I receive the same error reported above.

I'm not sure what is the difference between the 2 places but if these are 2 different resources then both should be deleted in the correct order because both are created by terraform. If they are the same thing then the issue will be with GCP.

wawrzek commented 1 month ago

I have the same problem when try to use Terraform to establish a private connection from the SQL instance to a VPC (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/sql_database_instance)

ggtisc commented 1 month ago

Hi @botzkobg!

I ran your code and I was able to successfully destroy the google_service_networking_connection resource with the command terraform destroy -target=google_service_networking_connection.default. However, it is possible that you have some resources that depend on your google_service_networking_connection as indicated by the error, so I suggest you check if the google_service_networking_connection is still being used and try the direct command again without variables.

botzkobg commented 1 month ago

Thank you @ggtisc

I will try again by using the command you have used. Could you confirm if you checked the UI and if you did where both resources created as on my screenshots?

botzkobg commented 1 month ago

@ggtisc, just tried as you suggested without any vars and with the command that will delete only this resource. I got the same error message. Here is the output of the commands: execution.log

I also tried on a new project. On this new project there is no problem to destroy the resource.

All resources in my project are created with terraform and nothing else was used to create or change resource (except me deleting google_service_networking_connection.default from the UI to be able to fully destroy the project). My terraform script creates a lot of resources. The destroy command fails to destroy only this resource. As seen from the attached output only the google_compute_global_address.private_ip_address and google_service_networking_connection.default are created but still google_service_networking_connection.default can't be deleted. All of this makes me think that there was something else created but is not destroyed and starts using the connection immediately after it's created.

@wawrzek I also use this for mysql instance.

ggtisc commented 1 month ago

yes @botzkobg I took the time to confirm that nothing would be left on the UI side. It should be noted that I used a project where I had no other resources other than those involved in this scenario

botzkobg commented 1 month ago

@ggtisc I have managed to reproduce it on a new project. It seems that the issue will happen if you additionally create a mysql database. After a database was created once you will be able to reproduce this issue every time on that project.

Steps to reproduce:

Create a new project
Enable compute.googleapis.com and servicenetworking.googleapis.com
Save the blow code to a file

locals {
  project_id      = "terraform-bug-test-delete-2"
}

data "google_compute_network" "cloud_run_vpc_default" {
  project = local.project_id
  name    = "default"
}

resource "google_compute_global_address" "private_ip_address" {
  project       = local.project_id
  name          = "private-ip-address"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 24
  network       = data.google_compute_network.cloud_run_vpc_default.id
}

resource "google_service_networking_connection" "default" {
  network                 = data.google_compute_network.cloud_run_vpc_default.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_address.name]
}

resource "google_sql_database_instance" "main" {
  depends_on = [google_service_networking_connection.default]

  database_version = "MYSQL_8_0_26"
  // TODO this should be `false` only fro `dev` env
  deletion_protection = false
  name                = "main"
  project             = local.project_id
  region              = "europe-west2"

  settings {
    activation_policy            = "ALWAYS"
    availability_type            = "ZONAL"
    collation                    = null
    connector_enforcement        = "NOT_REQUIRED"
    deletion_protection_enabled  = false
    disk_autoresize              = true
    disk_autoresize_limit        = 0
    disk_size                    = 10
    disk_type                    = "PD_SSD"
    edition                      = "ENTERPRISE"
    enable_dataplex_integration  = false
    enable_google_ml_integration = false
    pricing_plan                 = "PER_USE"
    tier                         = "db-custom-1-3840"

    backup_configuration {
      binary_log_enabled             = true
      enabled                        = true
      location                       = null
      point_in_time_recovery_enabled = false
      start_time                     = "18:49"
      transaction_log_retention_days = 7

      backup_retention_settings {
        retained_backups = 7
        retention_unit   = "COUNT"
      }
    }

    database_flags {
      name  = "log_bin_trust_function_creators"
      value = "on"
    }
    database_flags {
      name  = "long_query_time"
      value = "1"
    }
    database_flags {
      name  = "slow_query_log"
      value = "on"
    }
    database_flags {
      name  = "sql_mode"
      value = "NO_ENGINE_SUBSTITUTION"
    }

    insights_config {
      query_insights_enabled  = true
      query_plans_per_minute  = 5
      query_string_length     = 1024
      record_application_tags = true
      record_client_address   = true
    }

    ip_configuration {
      allocated_ip_range                            = null
      enable_private_path_for_google_cloud_services = false
      ipv4_enabled                                  = true
      private_network                               = data.google_compute_network.cloud_run_vpc_default.id
      server_ca_mode                                = "GOOGLE_MANAGED_INTERNAL_CA"
      ssl_mode                                      = "ALLOW_UNENCRYPTED_AND_ENCRYPTED"
    }

    location_preference {
      follow_gae_application = null
      secondary_zone         = null
      zone                   = "europe-west2-c"
    }

    sql_server_audit_config {
      bucket             = null
      retention_interval = "0s"
      upload_interval    = "0s"
    }
  }
}

Execute the following commands:

terraform plan -out=execution.tfplan
terraform apply execution.tfplan
terraform destroy

This will give you the reported error message:

│ Error: Unable to remove Service Networking Connection, err: Error waiting for Delete Service Networking Connection: Error code 9, message: Failed to delete connection; Producer services (e.g. CloudSQL, Cloud Memstore, etc.) are still using this connection.
│ Help Token: AYJSUtn_ttU-LUe0jSB2AG8da9chWzOTEjdqVolx79iHQKnvoHbDuQm-iGq8Kle7v-Ov36eTtamsGMFbUgQkv6l2A4-gK2HV4obK4iHl7i-_3oCw

wawrzek commented 4 weeks ago

You can use the test from this module (called postgres) to reproduce the issue.

Link to the repo with a module and the test: https://github.com/ivy-net/otofu-modules/tree/master/postgres

terraform test
tests/main.tftest.hcl... in progress
  run "standalone_plan"... pass
  run "setup1"... pass
  run "plan_standalone"... pass
  run "apply_standalone"... pass
  run "setup2"... pass
  run "plan_private"... pass
  run "apply_private"... pass
  run "apply_go_public"... fail
╷
│ Error: Unable to remove Service Networking Connection, err: Error waiting for Delete Service Networking Connection: Error code 9, message: Failed to delete connection; Producer services (e.g. CloudSQL, Cloud Memstore, etc.) are still using this connection.
│ Help Token: AYJSUtmpJM0MYPuxLJLy6wgEIFKDiqKXLC_0DypdRHhPnSBVBdgiLCwOfe91IhDxIPGlWpGh6kEwalShyr5ryMW-b-O3OmgnLFFmedduBEOYg0hJ
│
│
╵
tests/main.tftest.hcl... tearing down
Terraform encountered an error destroying resources created while executing tests/main.tftest.hcl/apply_go_public.
╷
│ Error: Unable to remove Service Networking Connection, err: Error waiting for Delete Service Networking Connection: Error code 9, message: Failed to delete connection; Producer services (e.g. CloudSQL, Cloud Memstore, etc.) are still using this connection.
│ Help Token: AYJSUtnO6uTSFxURFIJbOCgQmfuz0sYdNaSUhYWEuGqM3oAzaLiREzyvw9RcS1FueEGyulzjtZ1_gYrmvDbD5PB51yemKa-BhyT-E1BNd4Ve0Blz
│
│
╵

Terraform left the following resources in state after executing tests/main.tftest.hcl/apply_go_public, and they need to be cleaned up manually:
  - google_compute_global_address.this[0]
  - google_service_networking_connection.this[0]
Terraform encountered an error destroying resources created while executing tests/main.tftest.hcl/setup2.
╷
│ Error: Error waiting for Deleting Network: The network resource 'projects/ivynet-tests/global/networks/deciding-seal' is already being used by 'projects/ivynet-tests/global/addresses/deciding-seal-private-ip-address'
│
│
│
╵

Terraform left the following resources in state after executing tests/main.tftest.hcl/setup2, and they need to be cleaned up manually:
  - random_pet.name
  - module.network.google_compute_network.this
tests/main.tftest.hcl... fail

Failure! 7 passed, 1 failed.

ggtisc commented 4 weeks ago

Confirmed issue!

After creating the resources and then trying to destroy the google_service_networking_connection with the command terraform destroy -target=google_service_networking_connection.connection_name and terraform destroy. Terraform acts as if it will destroy both the google_sql_database_instance and the google_service_networking_connection in a chain action but after waiting it destroy the google_sql_database_instance but for some reason the google_service_networking_connection is not possible to be destroyed as if it continues to be used

StephenWithPH commented 3 weeks ago

This may be similar to #18834

hashicorp / terraform-provider-google