flpablo commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.2.8 hashicorp/google v4.34.0

Affected Resource(s)

google_pubsub_lite_topic

Terraform Configuration Files

provider "google" {
 project = "example-project"
 region = "us-central1"
}

resource "google_pubsub_lite_reservation" "example" {
  name = "example-reservation"
  project = "project-name"
  throughput_capacity = 2
}

resource "google_pubsub_lite_topic" "example" {
  name = "example-topic"
  project = data.google_project.project.number

  partition_config {
    count = 1
    capacity {
      publish_mib_per_sec = 4
      subscribe_mib_per_sec = 8
    }
  }

  retention_config {
    per_partition_bytes = 32212254720
  }

  reservation_config {
    throughput_reservation = google_pubsub_lite_reservation.example.name
  }
}

Debug Output

https://gist.github.com/flpablo/f1e126a62a87dcae4d982f5d20a05d89

Expected Behavior

When creating a PubSub Topic Lite there should be two options for location: Regional or Zonal.

Actual Behavior

For deploying Zonal Topic, it is straight forward since you only need to provide a region and a zone. (No issue)
For deploying Regional Location, there is no clear way to choose this option. If the zone is removed from the arguments it throws and error that a zone must be provided. So it seems, regional location for PubSub Lite Topic is not an option at the moment.

References

https://cloud.google.com/pubsub/lite/docs/topics#overview_of_a_lite_topic

b/293325796

edwardmedia commented 2 years ago

@flpablo it is fine with me with either zone or region provided. Can you try one like below?

resource "google_pubsub_lite_topic" "example" {
  name = "example-topic"
  project = data.google_project.project.number

  partition_config {
    count = 1
    capacity {
      publish_mib_per_sec = 4
      subscribe_mib_per_sec = 8
    }
  }

  retention_config {
    per_partition_bytes = 32212254720
  }

  reservation_config {
    throughput_reservation = google_pubsub_lite_reservation.example.name
  }

  region = "us-west1"
  //zone = "us-west1-a"
}

flpablo commented 2 years ago

@edwardmedia I tried as mentioned but still I got the error of not determine the zone:

`% terraform apply google_pubsub_lite_reservation.example-reservation: Refreshing state... Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:

create

Terraform will perform the following actions:

google_pubsub_lite_topic.example will be created

resource "google_pubsub_lite_topic" "example" {
- id = (known after apply)
- name = "example-topic"
- project = "example-project"
- region = "us-west1"
- partition_config {
  - count = 1
  - capacity {
    - publish_mib_per_sec = 4
    - subscribe_mib_per_sec = 8 } }
- reservation_config {
  - throughput_reservation = "example-reservation" }
- retention_config {
  - per_partition_bytes = "32212254720" } }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.

Enter a value: yes

google_pubsub_lite_topic.example: Creating... ╷ │ Error: Cannot determine zone: set in this resource, or set provider-level zone. │ │ with google_pubsub_lite_topic.example, │ on main.tf line 21, in resource "google_pubsub_lite_topic" "example": │ 21: resource "google_pubsub_lite_topic" "example" { │ ╵ `

edwardmedia commented 2 years ago

@flpablo can you enable debug for your log with below environment variable?

export TF_LOG=DEBUG

flpablo commented 2 years ago

@edwardmedia Sure. These are the logs:

2022-09-04T15:26:05.327-0500 [INFO] backend/local: apply calling Apply 2022-09-04T15:26:05.328-0500 [DEBUG] Building and walking apply graph for NormalMode plan 2022-09-04T15:26:05.329-0500 [DEBUG] Resource state not found for node "google_pubsub_lite_topic.example-topic", instance google_pubsub_lite_topic.example-topic 2022-09-04T15:26:05.329-0500 [DEBUG] ProviderTransformer: "google_pubsub_lite_topic.example-topic (expand)" (terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"] 2022-09-04T15:26:05.329-0500 [DEBUG] ProviderTransformer: "google_pubsub_lite_topic.example-topic" (terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/hashicorp/google"] 2022-09-04T15:26:05.329-0500 [DEBUG] ReferenceTransformer: "google_pubsub_lite_topic.example-topic (expand)" references: [] 2022-09-04T15:26:05.330-0500 [DEBUG] ReferenceTransformer: "google_pubsub_lite_topic.example-topic" references: [] 2022-09-04T15:26:05.330-0500 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/google\"]" references: [] 2022-09-04T15:26:05.331-0500 [DEBUG] Starting graph walk: walkApply 2022-09-04T15:26:05.331-0500 [DEBUG] created provider logger: level=debug 2022-09-04T15:26:05.332-0500 [INFO] provider: configuring client automatic mTLS 2022-09-04T15:26:05.362-0500 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.34.0/darwin_amd64/terraform-provider-google_v4.34.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/google/4.34.0/darwin_amd64/terraform-provider-google_v4.34.0_x5] 2022-09-04T15:26:05.373-0500 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.34.0/darwin_amd64/terraform-provider-google_v4.34.0_x5 pid=3923 2022-09-04T15:26:05.373-0500 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.34.0/darwin_amd64/terraform-provider-google_v4.34.0_x5 2022-09-04T15:26:05.400-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: configuring server automatic mTLS: timestamp=2022-09-04T15:26:05.400-0500 2022-09-04T15:26:05.429-0500 [DEBUG] provider: using plugin: version=5 2022-09-04T15:26:05.429-0500 [DEBUG] provider.terraform-provider-google_v4.34.0_x5: plugin address: address=/var/folders/ny/rqpnlzz932q19z7l0shj6knw0000gp/T/plugin1999672421 network=unix timestamp=2022-09-04T15:26:05.429-0500 2022-09-04T15:26:05.540-0500 [WARN] ValidateProviderConfig from "provider[\"registry.terraform.io/hashicorp/google\"]" changed the config value, but that value is unused 2022-09-04T15:26:05.542-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:05 [INFO] Authenticating using DefaultClient...: timestamp=2022-09-04T15:26:05.542-0500 2022-09-04T15:26:05.542-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:05 [INFO] -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: timestamp=2022-09-04T15:26:05.542-0500 2022-09-04T15:26:05.542-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:05 [INFO] Authenticating using DefaultClient...: timestamp=2022-09-04T15:26:05.542-0500 2022-09-04T15:26:05.542-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:05 [INFO] -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: timestamp=2022-09-04T15:26:05.542-0500 2022-09-04T15:26:05.542-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:05 [DEBUG] Waiting for state to become: [success]: timestamp=2022-09-04T15:26:05.542-0500 2022-09-04T15:26:06.341-0500 [INFO] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:06 [INFO] Terraform is using this identity: user@domain: timestamp=2022-09-04T15:26:06.341-0500 2022-09-04T15:26:06.355-0500 [INFO] Starting apply for google_pubsub_lite_topic.example-topic 2022-09-04T15:26:06.355-0500 [DEBUG] google_pubsub_lite_topic.example-topic: applying the planned Create change 2022-09-04T15:26:06.358-0500 [ERROR] provider.terraform-provider-google_v4.34.0_x5: Response contains error diagnostic: @module=sdk.proto diagnostic_severity=ERROR diagnostic_summary="Cannot determine zone: set in this resource, or set provider-level zone." tf_proto_version=5.2 tf_provider_addr=provider tf_resource_type=google_pubsub_lite_topic tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/diag/diagnostics.go:56 diagnostic_detail= tf_req_id=7a0932c2-2bfd-b835-5e06-950719b4ad9f timestamp=2022-09-04T15:26:06.358-0500 2022-09-04T15:26:06.386-0500 [ERROR] vertex "google_pubsub_lite_topic.example-topic" error: Cannot determine zone: set in this resource, or set provider-level zone. 2022-09-04T15:26:06.415-0500 [DEBUG] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:06 [DEBUG] [core] [Server #1 ListenSocket #2] ListenSocket deleted 2022-09-04T15:26:06.415-0500 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2022-09-04T15:26:06.415-0500 [DEBUG] provider.terraform-provider-google_v4.34.0_x5: 2022/09/04 15:26:06 [DEBUG] [transport] transport: http2Server.HandleStreams failed to read frame: read unix /var/folders/ny/rqpnlzz932q19z7l0shj6knw0000gp/T/plugin1999672421->: use of closed network connection 2022-09-04T15:26:06.423-0500 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/4.34.0/darwin_amd64/terraform-provider-google_v4.34.0_x5 pid=3923 2022-09-04T15:26:06.423-0500 [DEBUG] provider: plugin exited

Let me know if you need more information. Thank you for your help.

edwardmedia commented 2 years ago

@flpablo I see what is going on. Do you have below environment variables available? Please set them if you don't have. Those become required if you don't have them setup.

https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#zone

flpablo commented 2 years ago

@edwardmedia Yes, I've tried setting the environment variables, but the problem is that it creates (by default) the topic to be "Zonal lite topic". I can't figure out how to "overcome" this behavior and create a "Regional lite topic". In the example code you provided a few days ago, did it created a "Regional" topic?

jessee-kung commented 1 year ago

Same question here. It seems that there's no option to configure the Pub/Sub Lite topic as Regional or Zonal?

Providing region and left the zone argument empty leads the same 400 error:

resource "google_pubsub_lite_topic" "topic" {
  name    = "my_topic"
  project = data.google_project.project.number
  region  = "us-west1"
  ...
}

Result:

Error: Cannot determine zone: set in this resource, or set provider-level zone.
│ 
│   with module.infra.module.pubsublite["my_topic"].google_pubsub_lite_topic.topic,
│   on ../../modules/infra/pubsublite/deployment.tf line 10, in resource "google_pubsub_lite_topic" "topic":
│   10: resource "google_pubsub_lite_topic" "topic" {
│

Provding both region and zone argument as the value of region argument (e.g., all filled in us-west1, which looks weird), it creates regional topic:

resource "google_pubsub_lite_topic" "topic" {
  name    = "my_topic"
  project = data.google_project.project.number
  region  = "us-west1"
  zone = "us-west1"
  ...
}

Result:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

It should be unexpected behavior.

edwardmedia commented 1 year ago

I see where the problem is. To fix it, 1) Need to update all urls like below to dynamically assign {{location}} with either region or zone whichever is available. If region is picked, the regional resource is created. When zone is used, the resource becomes zonal.

{{PubsubLiteBasePath}}projects/{{project}}/locations/{{location}}/topics?topicId={{name}

2) ExactlyOneOf needs to be added to both region and zone as only one of them is used. But this is breaking-change.

edwardmedia commented 1 year ago

It might be better to add the new field location, and remove both region and zone. It is a feature request and add the label with enhancement accordingly

hashicorp / terraform-provider-google

PubSub Lite Topic Location #12464

Community Note

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

References

google_pubsub_lite_topic.example will be created