omelnyk1
commented
4 years ago JFYI, update operation works good, if log bucket already exists.
Closed omelnyk1 closed 1 year ago
omelnyk1
commented
4 years ago JFYI, update operation works good, if log bucket already exists.
edwardmedia
commented
4 years ago @omelnyk1 I think the error message is clear, and you have found the solution. Bucket is a different resource, google_logging_project_bucket_config is not intended to manage the Bucket. If you want Terraform to manage the Bucket, you may include google_storage_bucket and other resources in your plan. If this helps resolve your issue, please go ahead to close the issue.
"message": "Bucket newdbalogsbucket does not exist",
omelnyk1
commented
4 years ago Hi @edwardmedia
Thanks for your reply. To be fair, it's not clear how to create logging bucket (not "_Default" and "_Required") from the scratch with Terraform. Resource google_storage_bucket doesn't suite.
edwardmedia
commented
4 years ago @omelnyk1 I agreed the error message is confusing, which led me to think of a regular bucket. That is wrong. I did test your code and it works fine. I am able to see the Logs buckets being created. Did you delete the newdbalogsbucket through other method? To repro the error you have, I have tried to manually delete it from Console and its status turned to "pending" and never end. Can you provide the detail steps to repro your issue?
omelnyk1
commented
4 years ago @edwardmedia A deleted logging bucket stays in pending state for 7 days. I removed it manually via GCP Console as well. For new tests you'll be able to specify new bucket ids. For instance:
resource "google_logging_project_bucket_config" "basic" {
project = "my-project"
location = "global"
retention_days = 180
bucket_id = "newexample"
description = "Apache logs"
}
Please find full debug log below: https://gist.github.com/omelnyk1/2c2911617791e90afddf8b945c5158f6
edwardmedia
commented
4 years ago @omelnyk1 there is an improvement for this resource in v3.44.0 Can you try with this version instead?
omelnyk1
commented
4 years ago @edwardmedia
Create operation works good:
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# google_logging_project_bucket_config.basic will be created
+ resource "google_logging_project_bucket_config" "basic" {
+ bucket_id = "apache_logs"
+ description = "Apache logs"
+ id = (known after apply)
+ lifecycle_state = (known after apply)
+ location = "global"
+ name = (known after apply)
+ project = "my-project"
+ retention_days = 60
}
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
google_logging_project_bucket_config.basic: Creating...
google_logging_project_bucket_config.basic: Creation complete after 2s [id=projects/my-project/locations/global/buckets/apache_logs]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.Update operation is fine as well:
$ terraform apply
google_logging_project_bucket_config.basic: Refreshing state... [id=projects/my-project/locations/global/buckets/apache_logs]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
~ update in-place
Terraform will perform the following actions:
# google_logging_project_bucket_config.basic will be updated in-place
~ resource "google_logging_project_bucket_config" "basic" {
bucket_id = "apache_logs"
~ description = "Apache logs" -> "Apache logs 65 days"
id = "projects/my-project/locations/global/buckets/apache_logs"
lifecycle_state = "ACTIVE"
location = "global"
name = "projects/my-project/locations/global/buckets/apache_logs"
project = "my-project"
~ retention_days = 60 -> 65
}
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
google_logging_project_bucket_config.basic: Modifying... [id=projects/my-project/locations/global/buckets/apache_logs]
google_logging_project_bucket_config.basic: Modifications complete after 1s [id=projects/my-project/locations/global/buckets/apache_logs]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
Delete operation didn't make effect on resource:
$ terraform destroy
google_logging_project_bucket_config.basic: Refreshing state... [id=projects/everon-test12/locations/global/buckets/apache_logs]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# google_logging_project_bucket_config.basic will be destroyed
- resource "google_logging_project_bucket_config" "basic" {
- bucket_id = "apache_logs" -> null
- description = "Apache logs 65 days" -> null
- id = "projects/my-project/locations/global/buckets/apache_logs" -> null
- lifecycle_state = "ACTIVE" -> null
- location = "global" -> null
- name = "projects/my-project/locations/global/buckets/apache_logs" -> null
- project = "everon-test12" -> null
- retention_days = 65 -> null
}
Plan: 0 to add, 0 to change, 1 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
google_logging_project_bucket_config.basic: Destroying... [id=projects/my-project/locations/global/buckets/apache_logs]
google_logging_project_bucket_config.basic: Destruction complete after 0s
Destroy complete! Resources: 1 destroyed.
but with gcloud tool I see that apahe_logs bucket is in ACTIVE state. Should be DELETE_REQUESTED:
$ gcloud alpha logging buckets describe apache_logs --location=global --project=my-project
createTime: '2020-10-23T09:08:00.037656026Z'
description: Apache logs 65 days
lifecycleState: ACTIVE
name: projects/my-project/locations/global/buckets/apache_logs
retentionDays: 65
updateTime: '2020-10-23T09:13:24.121270345Z'
Could you please have a look? Thank you.
edwardmedia
commented
4 years ago @omelnyk1 I agreed it should have been set to DELETE_REQUESTED instead of keeping ACTIVE. I saw that too. When to update the bucket with a new bucket_id, it actually successfully creates the new bucket but leaves the old bucket untouched.
omelnyk1
commented
4 years ago @edwardmedia thank you. Will you be able consider issue https://github.com/hashicorp/terraform-provider-google/issues/7590 too? We'd like to configure 2 resources in a pair: log sink -> logging bucket
nat-henderson
commented
4 years ago This is intentional - if you take a look at the logs you'll see
[WARN] Logging bucket configs cannot be deleted. Removing logging bucket config from state.This was implemented by @chrisst who has since left our team - @rileykarson do you know why this is the way it is? The API does have a DELETE: https://cloud.google.com/logging/docs/reference/v2/rest/v2/projects.locations.buckets/delete.
rileykarson
commented
4 years ago Hmm- I've got no context and couldn't find any in the original PR. Maybe because it's a gradual deletion? we've supported those before, though.
We could consider moving this to an enhancement, to add delete support. If it's not on the resource page, we should document this behaviour before removing the bug label.
nat-henderson
commented
4 years ago Okay, will do. It says in the docs page that
Logging buckets are automatically created for a given folder, project, organization, billingAccount and cannot be deleted. Creating a resource of this type will acquire and update the resource that already exists at the desired location. These buckets cannot be removed so deleting this resource will remove the bucket config from your terraform state but will leave the logging bucket unchanged. The buckets that are currently automatically created are "_Default" and "_Required".
This doesn't seem obviously right to me anymore but I assume it was right when we added the resource and is correct about how the resource works today. I'll swap this to an enhancement.
chrisst
commented
4 years ago If I remember correctly there were only 2 logging buckets that could be configured, Default and Audit, and they were pre-created for every project. There was no way to create or to delete them and thus this resource was more of a configuration of a singleton resource. The relevant excerpt from https://cloud.google.com/logging/docs/buckets is:
For each Google Cloud project, Logging automatically creates two Logs Buckets: _Required and _Default. All logs generated in the project are stored in the _Required and _Default Logs Buckets, which live in the project that the logs are generated in. The following describes the role and purpose of the _Required and _Default buckets:
_Required: This bucket holds Admin Activity audit logs, System Event audit logs, and Access Transparency logs, and retains them for 400 days. You aren't charged for the logs stored in _Required, and the retention period of the logs stored here cannot be modified. You cannot delete this bucket.
_Default: This bucket holds all other ingested logs in a Google Cloud project except for the logs held in the _Required bucket. Standard Cloud Logging pricing applies to these logs. Log entries held in the _Default bucket are retained for 30 days, unless you apply custom retention rules. You can't delete this bucket, but you can disable the _Default log sink that routes logs to this bucket.
At the time there wasn't a unified way for handling singleton type resources and it was a compromise to hide away the reality that Create wasn't actually creating but importing the resource. For these same buckets there was a delete API but it didn't allow you to delete those buckets so I decided to remove from state instead of failing to delete and requiring that the consumer did the state removal explicitly.
It looks like there are now custom named logging buckets (or I missed this on the first pass) that can be created and do not behave like the singleton buckets of _Default and _Required, in which case it makes sense that delete should be supported for those.
omelnyk1
commented
4 years ago @chrisst you're right. It seems custom named logging buckets were released only this summer. I think so, cause article https://cloud.google.com/blog/products/management-tools/cloud-logging-adds-log-buckets-feature was dated on August 21, 2020.
matthewblain
commented
4 years ago It appears that google_logging_project_bucket_config and google_logging_project_sink have different philosophies as to how they manage this sort of 'singleton' resources, as I found out and filed in #7811 .
Issue #1561 (and followups) about App Engine is a similar philosophical question. And I've seen other issues for other resources which are broadly similar.
Is there a document describing how the terraform-provider-google interacts with the lifecycle of resources in these various categories? I'm new to Terraform, but found writing docs like this helped when working on similar issues inside the App Engine and Google Cloud team years ago.
nat-henderson
commented
3 years ago There's no such document, because we handle those things on a case-by-case basis. One approach we've had success with is "create" == "acquire" - if it exists, begin to manage it, if it does not, create it. Is there something in particular you're interested in?
omelnyk1
commented
3 years ago Hi guys, Any updates on this?
nat-henderson
commented
3 years ago This is in our "Goals" milestone, which means that we agree it should be done, but don't currently have the resources to do it and also don't think we will have the resources soon (by contrast with "Near Term Goals").
We would love to accept a community PR that fixes this issue - otherwise it may be quite some time. We re-prioritize based on issue thumbs-ups - if many people give the issue a thumbs-up, we'll get around to it sooner. "Many people" here depends on how hard the issue is - 30+ for sure, but could be as low as 10+ if the issue is an easy one. This is getting close to the 10+ threshold - if it reaches that we'll be due for another discussion on priority.
rm-paoloventriglia
commented
2 years ago I appreciate this has been opened for some time but I find it a bit confusing that we are given the ability to create a new custom logging bucket but not the ability to destroy it. Also the warning on the google_logging_project_bucket_config page is not correct as it states "Creating a resource of this type will acquire and update the resource that already exists at the desired location." This is not correct as you can create custom buckets since v3.44.0
Could you please either update the documentation or add ability to delete the resource.
c2thorn
commented
1 year ago Deletion support was added with https://github.com/GoogleCloudPlatform/magic-modules/pull/5949
github-actions[bot]
commented
1 year ago I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Community Note
modular-magicianuser, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned tohashibot, a community member has claimed the issue already.Terraform Version
Terraform v0.12.26 Providers:
Affected Resource(s)
Terraform Configuration Files
Debug Output
The core error was:
Panic Output
Expected Behavior
I followed manual https://cloud.google.com/logging/docs/buckets and expected to create new log bucket. gcloud works like a charm:
Actual Behavior
Judging by output log, terraform provider trying to look up by bucket_id and update that resource.
Steps to Reproduce
terraform applyImportant Factoids
References
https://cloud.google.com/logging/docs/buckets#creating_a_logs_bucket
0000