google_logging_project_sink always requires unique_writer_identity argument

steenblik commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

$ terraform -v                                                                                                                                                                                                                                                                                                   
Terraform v1.2.7
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.31.0

Affected Resource(s)

google_logging_project_sink

Terraform Configuration Files

resource "google_logging_project_sink" "this" {
  name        = "test-sink"
  project     = var.project
  destination = "logging.googleapis.com/projects/${var.project}/locations/global/buckets/test-bucket"
}

Debug Output

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_logging_project_sink.this will be created
  + resource "google_logging_project_sink" "this" {
      + destination            = "logging.googleapis.com/projects/redacted-project/locations/global/buckets/test-bucket"
      + id                     = (known after apply)
      + name                   = "test-sink"
      + project                = "redacted-project"
      + unique_writer_identity = false
      + writer_identity        = (known after apply)

      + bigquery_options {
          + use_partitioned_tables = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.
google_logging_project_sink.this: Creating...
╷
│ Error: googleapi: Error 400: Advanced sink options require using per sink service accounts. Use uniqueWriterIdentity=true to create a unique service account for this sink, badRequest
│ 
│   with google_logging_project_sink.this,
│   on main.tf line 5, in resource "google_logging_project_sink" "this":
│    5: resource "google_logging_project_sink" "this" {

Expected Behavior

Creating a google_logging_project_sink resource pointing at a cloud logging bucket in the same project should not require unique_writer_identity to be true. The documentation for the resource says that unique_writer_identity is only required when setting up a cross project sink or when using bigquery_options.

results in a failure unless unique_writer_identity

Actual Behavior

When creating a simple google_logging_project_sink without bigquery_options the apply fails unless unique_writer_identity is set to true (See above Debug output).

b/300616737

edwardmedia commented 2 years ago

@steenblik can you try with the destination like below format?

resource "google_logging_project_sink" "this" {
  name        = "test-sink"
  project     = var.project
  destination = "storage.googleapis.com/test-bucket"
}

edwardmedia commented 2 years ago

@steenblik is this still an issue?

steenblik commented 2 years ago

Hi @edwardmedia - I am not able to get this to work for Cloud Logging buckets, which is what I am interested in. I believe this is an upstream issue or simply a misunderstanding on my part about when a new writer id is required.

edwardmedia commented 2 years ago

b/242557342

edwardmedia commented 2 years ago

@steenblik where did you see below document?

The documentation for the resource says that unique_writer_identity is only required when setting up a cross project sink or when using bigquery_options

steenblik commented 2 years ago

Hi @edwardmedia - I got this impression for the terraform documentation. I've just re-read it and my paraphrase was not entirely accurate. However, for me the doc leaves the impression that unique_writer_identity is required only when for cross project logging or with bigquery_options. Otherwise why not just say it's always required or when it's not required.

Whether or not to create a unique identity associated with this sink. If false (the default), then the writer_identity used is serviceAccount:cloud-logs@system.gserviceaccount.com. If true, then a unique service account is created and used for this sink. If you wish to publish logs across projects or utilize bigquery_options, you must set unique_writer_identity to true.

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink#unique_writer_identity

leoromanovsky commented 1 year ago

I also ran into this issue with a sink to a bucket destination:

resource "google_logging_project_bucket_config" "mybucket" {
  project        = "my-project
  location       = "global"
  retention_days = 120
  bucket_id      = "MyBucket"
}

resource "google_logging_project_sink" "my_sink" {
  name                   = "my_sink"
  destination            = "logging.googleapis.com/${google_logging_project_bucket_config.mybucket.id}"
  filter                 = "some filter"

  depends_on = [
    google_logging_project_bucket_config.mybucket
  ]
}

with error:

Advanced sink options require using per sink service accounts. Use uniqueWriterIdentity=true to create a unique service account for this sink

milesarmstrong commented 1 year ago

We've hit a similar issue with this resource as well.

We initially created a bucket and log sink without unique_writer_identity set, because from the documentation it seems like false should be the default:

resource "google_logging_project_bucket_config" "bucket" {
  project = "project_id"

  location         = "europe-west2"
  retention_days   = 400
  enable_analytics = true
  locked           = false
  bucket_id        = "bucket_name"
  description      = "..."
}

resource "google_logging_project_sink" "sink" {
  name                   = "sink_name"
  destination            = "logging.googleapis.com/${google_logging_project_bucket_config.bucket.id}"
  project                = "project_id"
  ...
  (6 exclusion filters)
}

A plan and apply successfully creates the bucket and sink.

Subsequent plans then show:

  # module.environment.google_logging_project_sink.sink must be replaced
-/+ resource "google_logging_project_sink" "sink" {
      - disabled               = false -> null
      ~ id                     = "projects/project_name/sinks/sink_name" -> (known after apply)
        name                   = "sink_name"
      ~ unique_writer_identity = true -> false # forces replacement
      + writer_identity        = (known after apply)
        # (2 unchanged attributes hidden)

        # (6 unchanged blocks hidden)
    }

Updating the terraform to explicitly set unique_writer_identity = false and applying does not make the diff go away.

pengq-google commented 1 year ago

We changed the default value of the unique_writer_identity value from false to true about 3 weeks ago: https://github.com/GoogleCloudPlatform/magic-modules/pull/8875

and the customer_writer_identity will be added in https://github.com/GoogleCloudPlatform/magic-modules/pull/8995

hashicorp / terraform-provider-google