Editing BigQuery destination config does not force stream recreation

[norbertcyran]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v0.13 on Linux

• provider registry.terraform.io/hashicorp/google v5.37

Affected Resource(s)

google_datastream_stream

Expected Behavior

When editing most of bigquery_destination_config fields (e.g. source_hierarchy_datasets.dataset_template.dataset_id_prefix), stream should be recreated, as those fields are immutable

Actual Behavior

terraform apply fails with Error: Error updating Stream "<redacted>": googleapi: Error 400: Only `kms_key_name` and `data_freshness` can be modified for BigQuery destination config.

Steps to reproduce

1. Create Datastream stream with TF
2. Change something in bigquery_destination_config, for example source_hierarchy_datasets.dataset_template.dataset_id_prefix
3. Run terraform apply

Additional information

https://github.com/hashicorp/terraform-provider-google/blob/main/google/services/datastream/resource_datastream_stream.go#L143 we can see that ForceNew is missing from most of the fields in BQ destination config, except kms_key_name, which, according to the error message from the API, is one of the two fields allowed to change

Workaround available: tainting the resource

b/352330060

[zli82016]

It looks like a bug in the provider code. The issue will be forwarded to the service team to fix.

1. Service team need to double check if kms_key_name can be modified. If yes, kms_key_name should not have ForceNew: true.
2. Other immutable fields inside bigquery_destination_config should have ForceNew: true.

[ggtisc]

Confirmed issue!

It isn't possible to change other properties in the block code bigquery_destination_config other than kms_key_nameand data_freshness. After trying to apply changes terraform returns the next message:

Error: Error updating Stream "<redacted>": googleapi: Error 400: Onlykms_key_nameanddata_freshnesscan be modified for BigQuery destination config.

In both Terraform registry and Google Cloud documentation it is not specified that this change is not possible.

To test it, there was tried to change the first block of code for the second:

Original:

bigquery_destination_config {
    single_target_dataset {
      dataset_id = google_bigquery_dataset.bq_ds_18644.id
    }
    data_freshness = "900s"
}

Changed code:

bigquery_destination_config {
    source_hierarchy_datasets {
      dataset_template {
        location = "us-central1"
        dataset_id_prefix = "myprefix"
      }
    }
    data_freshness = "900s"
}