google_pubsub_subscription treats an empty cloud_storage_config.avro_config as if it is missing and yields textConfig instead

[KKamishima]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.6 on linux_amd64

• provider registry.terraform.io/hashicorp/google v6.4.0

Affected Resource(s)

google_pubsub_subscription

Terraform Configuration

provider "google" {
  project = "macnica-mcp-devmgr-sandbox"
  region  = "asia-northeast1"
}

resource "google_storage_bucket" "example" {
  name  = "kkamishima-tf-bucket"
  location = "ASIA-NORTHEAST1"
  uniform_bucket_level_access = true
}

resource "google_pubsub_topic" "example" {
  name = "kkamishima-tf-topic"
}

resource "google_pubsub_subscription" "example" {
  name  = "kkamishima-tf-subscription"
  topic = google_pubsub_topic.example.id

  cloud_storage_config {
    bucket = google_storage_bucket.example.name

    avro_config {
    }
  }
  depends_on = [
    google_storage_bucket.example,
    google_storage_bucket_iam_member.admin,
  ]
}

data "google_project" "project" {
}

resource "google_storage_bucket_iam_member" "admin" {
  bucket = google_storage_bucket.example.name
  role   = "roles/storage.admin"
  member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

It is basically a duplicate of Example Usage - Pubsub Subscription Push Cloudstorage Avro in the documentation, except that there are no write_metadata or use_topic_schema properties specified.

Debug Output

No response

Expected Behavior

Output from 'feedbackgcloud pubsub subscriptions describe' shall look like this. Please notice the existence of avroConfig: {} part and the absence of textConfig: {} part, when compared to the Actual Behavior.

cloudStorageConfig:
  bucket: kkamishima-tf-bucket
  state: ACTIVE
  avroConfig: {}

As a result, of course, the messages are stored in Avro format, not in Text format.

Actual Behavior

Output from 'feedbackgcloud pubsub subscriptions describe' looks like this.

cloudStorageConfig:
  bucket: kkamishima-tf-bucket
  state: ACTIVE
  textConfig: {}

Also messages are indeed stored in Text format, despite that the template does inlcude avro_config specification.

Steps to reproduce

1. terraform apply

Important Factoids

To make a subscription with avroConfig: {} , please use the following command line.

gcloud pubsub subscriptions update kkamishima-tf-subscription --cloud-storage-output-format=avro --cloud-storage-bucket=kkamishima-tf-bucket

References

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_subscription https://cloud.google.com/pubsub/docs/create-cloudstorage-subscription https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/create https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/update https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions#Subscription

b/372032557

[KKamishima]

Also as a probably related issue, when an existing subscription contains an empty avroConfig: {} and if you try to apply the template without avro_config specification (i.e. implicitly choosing Text as its output format), Terraform does not report any change. It should replace avroConfig: {} with textConfig: {} in such cases.

Current output from describe ↓

cloudStorageConfig:
  avroConfig: {}

Template to apply ↓

  cloud_storage_config {
    bucket = google_storage_bucket.example.name
# no avro_config
  }

Output from terraform apply ↓

No changes. Your infrastructure matches the configuration.

[KKamishima]

Another note: if you specify both 'write_metadata' and 'use_topic_schema' to false inside 'avro_config', the result would be the same with Actual Behavior in the opening message.

    avro_config {
      write_metadata = false
      use_topic_schema = false
    }

↓

cloudStorageConfig:
  bucket: kkamishima-tf-bucket
  state: ACTIVE
  textConfig: {}

[ggtisc]

Hi @KKamishima!

After executing your code everything was successful without errors and the avro_config was set correctly in the tfstate file after many tries.

If after removing the avro_config{} from your code and executing a new terraform apply it says that there are no changes this is because the avro_config is a default configuration even if you don't declare it at creation

[KKamishima]

Hi @ggtisc Thank you for looking into the matter. However, I still insist that the issue persists, I am afraid.

What matters here is the ACTUAL subscription resource created using a template which contains an empty avroConfig {} specification. When I apply such a template, and then I query the properties using gcloud, I see textConfig is present (and avroConfig is absent.)

$ gcloud pubsub subscriptions describe kkamishima-tf-subscription
ackDeadlineSeconds: 300
cloudStorageConfig:
  bucket: kkamishima-tf-bucket
  maxDuration: 300s
  state: ACTIVE
  textConfig: {}
expirationPolicy:
  ttl: 2678400s
labels:
  goog-terraform-provisioned: 'true'
messageRetentionDuration: 604800s
name: projects/macnica-mcp-devmgr-sandbox/subscriptions/kkamishima-tf-subscription
pushConfig: {}
state: ACTIVE
topic: projects/macnica-mcp-devmgr-sandbox/topics/kkamishima-tf-topic

[KKamishima]

Meanwhile, terraform.tfstate contains avroConfig as specified in the input template. So there are discrepancies between the state managed by Terraform and the actual resource state, I presume.

terraform.tfstate
...
            "cloud_storage_config": [
              {
                "avro_config": [],
...

[ggtisc]

I tried to replicate this issue with the next configuration a couple of times, but can't get the same result as the user. I'm forwarding it for a more in-depth review.

resource "google_storage_bucket" "sb_19658" {
  name  = "sb-19658"
  location = "us-central1"
  uniform_bucket_level_access = true
}

resource "google_pubsub_topic" "ps_topic_19658" {
  name = "ps-topic-19658"
}

data "google_project" "project" {}

resource "google_storage_bucket_iam_member" "sb_iam_member" {
  bucket = google_storage_bucket.sb_19658.name
  role   = "roles/storage.admin"
  member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

resource "google_pubsub_subscription" "ps_subs_19658" {
  name  = "ps-subs-19658"
  topic = google_pubsub_topic.ps_topic_19658.id

  cloud_storage_config {
    bucket = google_storage_bucket.sb_19658.name
    avro_config {}
  }
  depends_on = [
    google_storage_bucket.sb_19658,
    google_storage_bucket_iam_member.sb_iam_member,
  ]
}