`resource_tags` cannot be destoryed if attached to a deleted `google_bigquery_dataset`

[glasnt]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.5 on darwin_arm64 (also Cloud Build)

• provider registry.terraform.io/hashicorp/google v5.43.0
• provider registry.terraform.io/hashicorp/google-beta v5.43.0

(Versions as provided by cft-developer-tools)

Affected Resource(s)

google_bigquery_table

Terraform Configuration

# Create tag keys and values
data "google_project" "default" {}

resource "google_tags_tag_key" "env_tag_key" {
  parent     = "projects/${data.google_project.default.project_id}"
  short_name = "env3"
}

resource "google_tags_tag_key" "department_tag_key" {
  parent     = "projects/${data.google_project.default.project_id}"
  short_name = "department3"
}

resource "google_tags_tag_value" "env_tag_value" {
  parent     = "tagKeys/${google_tags_tag_key.env_tag_key.name}"
  short_name = "prod"
}

resource "google_tags_tag_value" "department_tag_value" {
  parent     = "tagKeys/${google_tags_tag_key.department_tag_key.name}"
  short_name = "sales"
}

# Create a dataset
resource "google_bigquery_dataset" "default" {
  dataset_id                      = "MyDataset"
  default_partition_expiration_ms = 2592000000  # 30 days
  default_table_expiration_ms     = 31536000000 # 365 days
  description                     = "dataset description"
  location                        = "US"
  max_time_travel_hours           = 96 # 4 days
}

# Create a table
resource "google_bigquery_table" "default" {
  dataset_id          = google_bigquery_dataset.default.dataset_id
  table_id            = "mytable"
  description         = "table description"
  deletion_protection = false # set to "true" in production

  # Attach tags to the table
  resource_tags = {
    (google_tags_tag_key.env_tag_key.namespaced_name) : google_tags_tag_value.env_tag_value.short_name,
    (google_tags_tag_key.department_tag_key.namespaced_name) : google_tags_tag_value.department_tag_value.short_name
  }
}

Debug Output

https://gist.github.com/glasnt/627f9198eb84b6e78ff45963a7a9dfea

Expected Behavior

Should be able to cleanly destroy resource resources

Actual Behavior

Cannot delete tag value, tagValues/281483072758839, because it is still attached to resources in 'us' region. To delete this tag value, remove all holds and then remove it from all resources. At least one binding was found to an active or deleted resource in the 'us' region.

Steps to reproduce

1. terraform apply
2. terraform destory

Important Factoids

Related to https://github.com/terraform-google-modules/terraform-docs-samples/pull/734 (adding resource_tags to bigquery tables), but the same sample for adding resource_tags to bigquery datasets (https://github.com/terraform-google-modules/terraform-docs-samples/pull/733) didn't have an issue with the destroy

Speculation: Because google_bigquery_table has allow_resource_tags_on_deletion (which will be removed soon anyway), Terraform's order of operations doesn't delete the tags first, because it can have them attached to the table. However, you can't delete the resource tags if they're attached to a resource, deleted or not.

I've tried various depends_on, but I can't work out how to make sure the tags are deleted first.

References

https://github.com/terraform-google-modules/terraform-docs-samples/pull/734

https://github.com/GoogleCloudPlatform/magic-modules/pull/11393

[wj-chen]

This is currently by design to support table time travel: https://cloud.google.com/bigquery/docs/managing-tables#undeletetable

Quoted from a response from a related internal issue:

Once you issue a command to delete a table, the table is not immediately deleted. To support time travel, the table is actually deleted a few days later (2-7 days as customer configures it). So during this time the tag deletion will fail (since it is still linked to the table which is in pending delete state).

Meanwhile I'm not aware of any way to instruct Terraform to destroy resources in a custom order.

At the minimum, we can add some documentation on google_bigquery_table next to resource_tags that informs users of this behavior and asks them to clear the tags on the table first before removing the tag resources.

Also open to ideas if Terraform Google team has other ideas on how to manage this case.

[ggtisc]

According to the error messages after deleting the google_tags_tag_value and google_tags_tag_key resources 1st they need to be unattached from any other resources.

For example if the initial configuration is this:

resource "google_bigquery_table" "default" {
  dataset_id          = google_bigquery_dataset.default.dataset_id
  table_id            = "mytable"
  description         = "table description"
  deletion_protection = false # set to "true" in production

  # Attach tags to the table
  resource_tags = {
    (google_tags_tag_key.env_tag_key.namespaced_name) : google_tags_tag_value.env_tag_value.short_name,
    (google_tags_tag_key.department_tag_key.namespaced_name) : google_tags_tag_value.department_tag_value.short_name
  }
}

You 1st need to change for the next code removing the resource_tags before deleting the resources:

resource "google_bigquery_table" "default" {
  dataset_id          = google_bigquery_dataset.default.dataset_id
  table_id            = "mytable"
  description         = "table description"
  deletion_protection = false # set to "true" in production

  # Attach tags to the table
  resource_tags = {}
}

Then you should be able to delete the resources

[glasnt]

In the context of my problem, I don't have the ability to remove the tags before destroying the resources (see linked terraform-docs-samples PRs above).

[ggtisc]

@shuyama1 could you please check this issue to see if this is a bug or an enhancement?

[shuyama1]

@glasnt Thanks for bringing up the issue and for opening the ticket!

@wj-chen Thanks for your input from the service perspective. I do have a couple of questions about the table time travel feature:

1. Is this feature enabled by default? Is there a way to explicitly disable it and delete tables immediately?

   It mentions a retention window of "a minimum of two days to a maximum of seven days" in the doc, so I assume the answer is probably no? If that’s the case, is there a method to force delete tags linked to tables in the pending delete state?

2. Once a table is deleted after the time travel window expires, will the associated tags be automatically deleted as well?

[wj-chen]

• Is this feature enabled by default? Is there a way to explicitly disable it and delete tables immediately? It mentions a retention window of "a minimum of two days to a maximum of seven days" in the doc, so I assume the answer is probably no? If that’s the case, is there a method to force delete tags linked to tables in the pending delete state?
• Once a table is deleted after the time travel window expires, will the associated tags be automatically deleted as well?

I think it's no to both but I'm reaching out to the feature team to confirm and get more details. Will update here again once I hear back.

[wj-chen]

• Is this feature enabled by default? Is there a way to explicitly disable it and delete tables immediately? It mentions a retention window of "a minimum of two days to a maximum of seven days" in the doc, so I assume the answer is probably no? If that’s the case, is there a method to force delete tags linked to tables in the pending delete state?
• Once a table is deleted after the time travel window expires, will the associated tags be automatically deleted as well?

I think it's no to both but I'm reaching out to the feature team to confirm and get more details. Will update here again once I hear back.

Heard back and confirmed that the answer is that the current behavior is as expected by design (full context in b/336356803). The tags are intentionally preserved for customers to recover deleted tables and datasets. If customer wants to remove the tags too, they need to unset them from the dataset/table before removal.

I think we can start with this so the users are aware:

At the minimum, we can add some documentation on google_bigquery_table next to resource_tags that informs users of this behavior and asks them to clear the tags on the table first before removing the tag resources.