`google_logging_project_bucket_config` import prepends `project/` to 'project' argument

wwilfinger commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

% terraform -v
Terraform v1.5.6
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.81.0

Affected Resource(s)

google_logging_project_bucket_config

Terraform Configuration Files

Manually create a log bucket that matches the following terraform

resource "google_logging_project_bucket_config" "default" {
  project          = "example-project-id"
  location         = "global"
  retention_days   = 30
  bucket_id        = "example-bucket-id"
}

Run a terraform import into the resource with

% terraform import google_logging_project_bucket_config.default projects/example-project-id/locations/global/buckets/example-bucket-id

Expected Behavior

A subsequent terraform plan should show no changes needed

Actual Behavior

A subsequent terraform plan shows that the project argument is in terraform state with a prefix of projects/example-project-id. Terraform wishes to change it back to example-project-id with a recreate.

Terraform will perform the following actions:

  # google_logging_project_bucket_config.default must be replaced
-/+ resource "google_logging_project_bucket_config" "default" {
      + description      = (known after apply)
      - enable_analytics = false -> null
      ~ id               = "projects/example-project-id/locations/global/buckets/example-bucket-id" -> (known after apply)
      ~ lifecycle_state  = "ACTIVE" -> (known after apply)
      - locked           = false -> null
      ~ name             = "projects/example-project-id/locations/global/buckets/example-bucket-id" -> (known after apply)
      ~ project          = "projects/example-project-id" -> "example-project-id" # forces replacement
        # (3 unchanged attributes hidden)
    }

Steps to Reproduce

See above

Important Factoids

The destroy and recreate won't work because a deleted Log Bucket is marked for deletion for 7 days before it is actually deleted.

Changing the terraform to have project = "projects/example-project-id" will silence the terraform plan diff. However, if you try to create a brand new Log Bucket, without having to terraform state import, with project set to project = "projects/example-project-id", that fails with a 404 Not Found error from the GCP API.

References

https://github.com/hashicorp/terraform-provider-google/issues/14645 somewhat related perhaps

wwilfinger commented 1 year ago

https://github.com/hashicorp/terraform-provider-google/blob/main/google/services/logging/resource_logging_bucket_config.go#L138-L140

Perhaps this works for other parent types like "folder" and "organization", but doesn't work as well with "project"? I don't have folder or orgs set up for GCP so I cannot easily test that theory.

edwardmedia commented 1 year ago

b/299177196

sdykhner-twc commented 1 year ago

@wwilfinger

I have the same issue using Terraform 1.4.6 and hashicorp/google provider 4.65.2

I was able to "fix" it via following steps (my infrastructure uses remote state):

Run terraform state pull > example.tfstate to retrieve the current state of your infrastructure.
Open the pulled file. Increase "serial" number of the state. This is needed to be able to push the state back. Then find your google_logging_project_bucket_config.default resource in the file (you may be able to find it by searching for "type": "google_logging_project_bucket_config", "google_logging_project_bucket_config" or resource name) and edit its "project" field by removing project/ from the value. Save the changes.
Run terraform state push example.tfstate to push new state to the backend.

If you do not store your backend remotely, I assume you just have to edit local state file.

Sure, it's just a workaround, but I hope it helps if you need to solve the issue "now" and proceed with your work.

pengq-google commented 1 year ago

Thanks for the comments. We are looking into it and will update as soon as possible. Comment here to keep the issue open.

pengq-google commented 1 year ago

I tried to reproduce this issue today but my import works at intended by using hashicorp/google provider 4.75.0

I used the commands to try to reproduce the issue:

terraform init -upgrade
rm terraform.tfstate.backup
rm terraform.tfstate
terraform import google_logging_project_bucket_config.{my_bucket} projects/{my_project_id}/locations/global/buckets/{my_bucket_id}

If this issue re-occurs, do you mind share the log here with TF_LOG=DEBUG?

TF_LOG=DEBUG terraform import google_logging_project_bucket_config.{your_bucket} projects/{project_id}/locations/{location_id}/buckets/{bucket_id}

Thanks!

wwilfinger commented 1 year ago

My original post was opened against the provider v4.81.0. I reproduced this morning with both the latest v4.84.0 and the v4.75.0 you used.

To be clear, the import is successful but on subsequent plan there is a destroy and replace diff because of a diff in the "project" argument of the resource.

Relevant bit of debug log with the project id and bucket name replaced to be generic

2023-10-03T09:50:28.059-0500 [WARN]  Provider "registry.terraform.io/hashicorp/google" produced an unexpected new value for google_logging_project_bucket_config.workload during refresh.
      - .enable_analytics: was null, but now cty.False
      - .lifecycle_state: was null, but now cty.StringVal("ACTIVE")
      - .locked: was null, but now cty.False
      - .name: was null, but now cty.StringVal("projects/example-project-id/locations/global/buckets/example-bucket-name")
      - .retention_days: was null, but now cty.NumberIntVal(30)
      - .description: was null, but now cty.StringVal("")

I'm happy to provide the full debug logs, but I do not want to try to obfuscate them and post them on the internet forever. You can email me at my github handle at google mail and I can send them over. Appreciate it.

pengq-google commented 1 year ago

Thank you for your feedback! I just sent you an email and re-open the ticket in our system. I'll keep working on it.

pdecat commented 1 year ago

Hi @wwilfinger , are you actually using a google_project datasource to feed the project field of the google_logging_project_bucket_config resource?

I've just discovered that it puts the projects/ prefix in its id attribute.

wwilfinger commented 1 year ago

are you actually using a google_project datasource to feed the project field of the google_logging_project_bucket_config resource?

The project id as a string literal right on the resource, like project = "example-project-id", reproduces.

pdecat commented 1 year ago

The Create function of the google_logging_project_bucket_config resource is supposed to automatically import the resource if it exists, and create it if it doesn't: https://github.com/terraform-providers/terraform-provider-google/blob/3eeed812d3bbeffb050002db14ffaefb8db3ea17/google/services/logging/resource_logging_project_bucket_config.go#L141

If you remove the google_logging_project_bucket_config.default resource from your state, what does an apply do?

Regarding the manual import behavior, I believe it is coming from this generic function: https://github.com/terraform-providers/terraform-provider-google/blob/3e00cfbafa1ef13ebfd7e039bcec77407b824bfc/google/services/logging/resource_logging_bucket_config.go#L142

And the 404 error happening when passing the projects/ prefix in the project argument is coming from this part where the resourceLoggingBucketConfigCreate() function also adds it a second time: https://github.com/terraform-providers/terraform-provider-google/blob/3e00cfbafa1ef13ebfd7e039bcec77407b824bfc/google/services/logging/resource_logging_bucket_config.go#L213

pengq-google commented 1 year ago

I reproduced this error on my laptop:

  # google_logging_project_bucket_config.tf_bucket_beta must be replaced
-/+ resource "google_logging_project_bucket_config" "tf_bucket_beta" {
        bucket_id        = "tf_bucket_beta"
        description      = "tf_bucket_beta"
      - enable_analytics = false -> null
      ~ id               = "projects/<my-project>/locations/global/buckets/tf_bucket_beta" -> (known after apply)
      ~ lifecycle_state  = "ACTIVE" -> (known after apply)
        location         = "global"
      - locked           = false -> null
      ~ name             = "projects/<my-project>/locations/global/buckets/tf_bucket_beta" -> (known after apply)
      ~ project          = "projects/<my-project>" -> "<my-project>" # forces replacement
        retention_days   = 30
    }

with config

resource "google_logging_project_bucket_config" "tf_bucket_beta" {
  provider    = google-beta
  project     = google_project.project0.project_id
  location    = "global"
  description = "tf_bucket_beta"
  bucket_id   = "tf_bucket_beta"

pengq-google commented 1 year ago

changed my config from

resource "google_logging_project_bucket_config" "tf_bucket_beta" {
  provider    = google-beta
  project     = google_project.project0.project_id
  location    = "global"
  description = "tf_bucket_beta"
  bucket_id   = "tf_bucket_beta"
}

to

resource "google_logging_project_bucket_config" "tf_bucket_beta" {
  provider    = google-beta
  project     = "projects/${google_project.project0.project_id}"
  location    = "global"
  description = "tf_bucket_beta"
  bucket_id   = "tf_bucket_beta"
}

solved the issue.

I am not sure why this happened or how long it has been like this. Thank you for raise the issue. Now I suspect it is because only resourceLoggingProjectBucketConfigAcquireOrCreate takes in projectBucketConfigID, while resourceLoggingProjectBucketConfigUpdate doesn't. https://github.com/hashicorp/terraform-provider-google/blob/0a8915bf4b03100573aec7caa8b3491567efa028/google/services/logging/resource_logging_project_bucket_config.go#L113C6-L113C27

Will try to modify the code to see if this issue is fixed. Before I confirming the root cause and fixing it, please using project = "projects/${google_project.project0.project_id}" to work around.

pengq-google commented 1 year ago

pdecat

Thanks for digging into this!:)

pengq-google commented 1 year ago

@pdecat Sorry I saw your confused emoji. I am not sure whether you are confused by me @ you, or confused by what I said above. I didn't have time to look into this issue (but it's always on my mind!) until last night, so I just spent some time on reading the code base, and left some comments. Then I noticed that you've already looked at the code and gave us advice. I just want to say thank you for your help! ^_^

pdecat commented 1 year ago

@pdecat Sorry I saw your confused emoji.

Oops, it was a mis-click, I meant :heart:

manzomanze commented 4 months ago

Do you have any updates on this issue? we encountered it today

MaStFAU commented 3 months ago

also important to mention that the docs on https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_bucket_config are misleading because it states These buckets cannot be removed so deleting this resource will remove the bucket config from your terraform state but will leave the logging bucket unchanged. which I find misleading. I guess it is only talking about the _Required and _Default buckets but custom buckets will be deleted. And this is problematic with the given issue as TF requires a replacement but deleting the bucket is not done instantly as mentioned above.

hashicorp / terraform-provider-google