Failing test(s): TestAccComposer1Environment_withNodeConfig

[SarahFrench]

Failure rates

• 100% since 2023-06-29
• 75% since 2023-08-23 (TestAccComposer1Environment_withNodeConfig)

Impacted tests

• TestAccComposer1Environment_withNodeConfig
• TestAccComposerEnvironment_ComposerV2
• TestAccComposerEnvironment_ComposerV2HighResilience
• TestAccComposerEnvironment_UpdateComposerV2
• TestAccComposerEnvironment_UpdateComposerV2WithTriggerer

Affected Resource(s)

• google_composer_environment

Nightly build test history

• See links above

Message(s)

------- Stdout: -------
=== RUN   TestAccComposerEnvironment_ComposerV2
=== PAUSE TestAccComposerEnvironment_ComposerV2
=== CONT  TestAccComposerEnvironment_ComposerV2
    vcr_utils.go:150: Step 1/3 error: Error running apply: exit status 1
        Error: Error waiting to create Environment: Error waiting for Creating Environment: Error code 9, message: Failed to create environment, but no error was surfaced. This can be caused by a lack of proper permissions. Check if this environment's service account 653407317329-compute@developer.gserviceaccount.com has the 'roles/composer.worker' role and there is no firewall inhibiting internal communications set. In case, it is a Compute default service account it should have Editor permissions. For the required permissions information, please, refer to https://cloud.google.com/composer/docs/how-to/access-control  https://cloud.google.com/composer/docs/troubleshooting-environment-creation page contains more troubleshooting instructions
          with google_composer_environment.test,
          on terraform_plugin_test.tf line 2, in resource "google_composer_environment" "test":
           2: resource "google_composer_environment" "test" {
--- FAIL: TestAccComposerEnvironment_ComposerV2 (3220.26s)
FAIL

Note (2023-12-18): TestAccComposer1Environment_withNodeConfig is failing due to the same error as above

b/326649868

[roaks3]

The error message indicates a permission issue, which we've come across for these before. Unclear if this should be forwarded to the service team, since we may need to revisit how the permissions are being configured within our test environment.

[SarahFrench]

I found some config from the Composer tests didn't include dependencies between google_composer_environment and the related google_project_iam_member resources, so tests may attempt to create environments before the necessary permissions are in place. See https://github.com/GoogleCloudPlatform/magic-modules/pull/9610

I've made it so the PR won't close this issue.

[SarahFrench]

I've updated this issue as it was a bit stale. The permissions issue is consistently seen for TestAccComposer1Environment_withNodeConfig and other tests pass without permissions issues.

[melinath]

Possibly this is a race condition? A time_sleep might help if the permissions just need to propagate.

[melinath]

Note: https://github.com/hashicorp/terraform-provider-google/issues/14375 also impacts this test.

[melinath]

Marking for forwarding since this is related to a service account created by the test - not part of the test environment.

[roaks3]

There was an attempted fix with https://github.com/GoogleCloudPlatform/magic-modules/pull/11926 which adds a 3m wait after the permission is granted, but it is still failing at a very high rate. It occasionally does pass, but it seems like either the 3 minutes is not long enough, or we need another approach.

[melinath]

This is still failing at a very high rate.