search

hashicorp / terraform-provider-google

Terraform Provider for Google Cloud Platform
https://registry.terraform.io/providers/hashicorp/google/latest/docs
Mozilla Public License 2.0
2.33k stars 1.74k forks source link

Failing test(s): TestAccLoggingBucketConfigOrganization_basic #16696

Open SarahFrench opened 11 months ago

SarahFrench commented 11 months ago

Failure rates

Screenshot 2023-12-06 at 18 55 04

Impacted tests

Affected Resource(s)

Nightly build test history

Message(s)


------- Stdout: -------
=== RUN   TestAccLoggingBucketConfigOrganization_basic
=== PAUSE TestAccLoggingBucketConfigOrganization_basic
=== CONT  TestAccLoggingBucketConfigOrganization_basic
    vcr_utils.go:152: Step 1/4 error: After applying this test step and performing a `terraform refresh`, the plan was not empty.
        stdout
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        Terraform will perform the following actions:
          # google_logging_organization_bucket_config.basic will be updated in-place
          ~ resource "google_logging_organization_bucket_config" "basic" {
                id              = "organizations/*******/locations/global/buckets/_Default"
                name            = "organizations/*******/locations/global/buckets/_Default"
                # (6 unchanged attributes hidden)
              - index_configs {
                  - field_path = "jsonPayload.request.url" -> null
                  - type       = "INDEX_TYPE_STRING" -> null
                }
              - index_configs {
                  - field_path = "jsonPayload.response.status" -> null
                  - type       = "INDEX_TYPE_INTEGER" -> null
                }
            }
        Plan: 0 to add, 1 to change, 0 to destroy.
--- FAIL: TestAccLoggingBucketConfigOrganization_basic (10.95s)
FAIL

b/315294638

SarahFrench commented 11 months ago

I don't think this PR is a cause of the test failure, but the timing is very close after being merged on Nov 3rd : https://github.com/GoogleCloudPlatform/magic-modules/pull/9373

pengq-google commented 10 months ago 
❯ make testacc TEST=./google-beta/services/pubsub TESTARGS='-run=TestAccLoggingBucketConfigOrganization_basic_'
==> Checking that code complies with gofmt requirements...
go vet
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta/services/pubsub -v -run=TestAccLoggingBucketConfigOrganization_basic_ -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
testing: warning: no tests to run
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/google-beta/services/pubsub 0.450s [no tests to run]
❯ make testacc TEST=./google/services/logging TESTARGS='-run=TestAccLoggingBucketConfigOrganization_basic_'
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google/services/logging -v -run=TestAccLoggingBucketConfigOrganization_basic_ -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google/version.ProviderVersion=acc"
testing: warning: no tests to run
PASS
ok      github.com/hashicorp/terraform-provider-google/google/services/logging  0.359s [no tests to run]

the test passed on my laptop. is it already fixed? or running it local is not the correct way to re-produce the error? I can't access to the Nightly build test history

SarahFrench commented 10 months ago

Hi @pengq-google I can't tell from the information you posted, but the tests may being skipped and not actually passing due to the issue I described here. Please let me know if the tests pass for you after checking all required ENVs are set.

I just checked a debug log from the test failing and I believe the cause of the test failures is that the test creates the resource with no index_config fields defined in the Terraform config, but the API returns some index config information after the create action:

 ---[ REQUEST ]---------------------------------------
PATCH /v2/organizations/123456789/locations/global/buckets/_Default?alt=json&updateMask=retentionDays%2Cdescription HTTP/1.1
Host: logging.googleapis.com
User-Agent: Terraform/1.2.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev
Content-Length: 98
Content-Type: application/json
Accept-Encoding: gzip

{
 "cmekSettings": null,
 "description": "retention test 30 days",
 "indexConfigs": [],
 "retentionDays": 30
}

 -----------------------------------------------------
2024/01/02 04:22:41 [DEBUG] Google API Response Details:
 ---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Tue, 02 Jan 2024 04:22:41 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "organizations/1234567889/locations/global/buckets/_Default",
  "description": "retention test 30 days",
  "updateTime": "2024-01-02T04:22:41.066690578Z",
  "retentionDays": 30,
  "lifecycleState": "ACTIVE",
  "indexConfigs": [
+    {
+      "fieldPath": "jsonPayload.request.url",
+      "type": "INDEX_TYPE_STRING",
+      "createTime": "2023-11-11T05:13:00.950893533Z"
+    },
+    {
+      "fieldPath": "jsonPayload.response.status",
+      "type": "INDEX_TYPE_INTEGER",
+      "createTime": "2024-01-02T04:08:30.909281305Z"
+    }
  ]
}

 -----------------------------------------------------

I believe what's causing the test to fail is that Terraform identifies that the new index configs information in the state doesn't match the configuration and will plan to remove the extra information.

melinath commented 10 months ago

@pengq-google it looks like you might have an extra _ at the end of the test name, which is probably why it's not being found. It should just be TestAccLoggingBucketConfigOrganization_basic:

~make testacc TEST=./google-beta/services/pubsub TESTARGS='-run=TestAccLoggingBucketConfigOrganization_basic'~

melinath commented 10 months ago

Actually it also needs to be adjusted to be targeting the correct service - currently it's only running tests for pubsub. Sorry for the double ping!

make testacc TEST=./google-beta/services/logging TESTARGS='-run=TestAccLoggingBucketConfigOrganization_basic_'
pengq-google commented 10 months ago

Thanks for all the advice!!!

pengq-google commented 10 months ago

With proper setting of Environment variables, 

❯ make testacc TEST=./google/services/logging TESTARGS='-run=TestAccLoggingBucketConfigOrganization_basic'
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google/services/logging -v -run=TestAccLoggingBucketConfigOrganization_basic -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccLoggingBucketConfigOrganization_basic
=== PAUSE TestAccLoggingBucketConfigOrganization_basic
=== CONT  TestAccLoggingBucketConfigOrganization_basic
--- PASS: TestAccLoggingBucketConfigOrganization_basic (38.10s)
PASS
ok      github.com/hashicorp/terraform-provider-google/google/services/logging  38.500s
❯ make testacc TEST=./google-beta/services/logging TESTARGS='-run=TestAccLoggingBucketConfigOrganization_basic'
==> Checking that code complies with gofmt requirements...
go vet
TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google-beta/services/logging -v -run=TestAccLoggingBucketConfigOrganization_basic -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google-beta/version.ProviderVersion=acc"
=== RUN   TestAccLoggingBucketConfigOrganization_basic
=== PAUSE TestAccLoggingBucketConfigOrganization_basic
=== CONT  TestAccLoggingBucketConfigOrganization_basic
--- PASS: TestAccLoggingBucketConfigOrganization_basic (40.43s)
PASS
ok      github.com/hashicorp/terraform-provider-google-beta/google-beta/services/logging        40.846s

I also tried with TF_LOG=DEBUG and TF_LOG=TRACE and can't produce this

what I got is:

---[ REQUEST ]---------------------------------------
PATCH /v2/organizations/[my-org- number]/locations/global/buckets/_Default?alt=json&updateMask=retentionDays%2Cdescription HTTP/1.1
Host: logging.googleapis.com
User-Agent: Terraform/1.6.0-dev (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/acc
Content-Length: 98
Content-Type: application/json
Accept-Encoding: gzip

{
 "cmekSettings": null,
 "description": "retention test 30 days",
 "indexConfigs": [],
 "retentionDays": 30
}

-----------------------------------------------------
2024/01/07 06:57:09 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Sun, 07 Jan 2024 06:57:09 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "name": "organizations/[my-org- number]/locations/global/buckets/_Default",
  "description": "retention test 30 days",
  "updateTime": "2024-01-07T06:57:09.176681119Z",
  "retentionDays": 30,
  "lifecycleState": "ACTIVE"
}

Is it because I am using my own project and provide a real org number? should I use fake org number "123456789"?

Thanks!

SarahFrench commented 10 months ago

It may be because the failure is due to multiple acceptance tests interacting with the same resource at the same time. From a quick search of the provider's codebase I found TestAccLoggingBucketConfigOrganization_indexConfigs, which adds the index configs that are present in the error in the issue description.

If that's the cause (I haven't looked into it extensively so may have missed something) the solution to the tests failing is likely to be changing the tests to not run in parallel anymore; the _Default bucket here is at the organisation level and our tests cannot provision new orgs to keep their actions separate from each other.