search

pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
459 stars 155 forks source link

TestRandomTagsCombinations sporadic test failures #3403

Closed t0yv0 closed 5 months ago

t0yv0 commented 8 months ago

What happened?

Several failures showed up in TestRandomTagsCombinations. I suspect this is less of a logical error and more of a problem with concurrency here with the individual runs not being sufficiently isolated on stomping on each other's bucket name. This is really a problem though as it blocks CI.

one

      program.go:1869: Performing extra runtime validation.
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"s"},"resourceTags":{"x":"","y":""}}
      examples_go_test.go:201: key=legacy-bucket tags are as expected: {"x":"","y":""}
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"s"},"resourceTags":{"x":"","y":""}}
      examples_go_test.go:201: key=appconfig-app tags are as expected: {"x":"","y":""}
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"s"},"resourceTags":{"x":"","y":""}}
      examples_go_test.go:201: key=bucket tags are as expected: {"x":"","y":""}
      examples_go_test.go:204: 
            Error Trace:    /home/runner/work/pulumi-aws/pulumi-aws/examples/examples_go_test.go:204
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1870
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1792
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1682
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1559
            Error:          Not equal: 
                            expected: map[string]string{"x":"", "y":""}
                            actual  : map[string]string{"x":""}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,4 +1,3 @@
                            -(map[string]string) (len=2) {
                            - (string) (len=1) "x": (string) "",
                            - (string) (len=1) "y": (string) ""
                            +(map[string]string) (len=1) {
                            + (string) (len=1) "x": (string) ""
                             }
            Test:           TestRandomTagsCombinationsGo/test-92
            Messages:       bad bucket tags

two

  program.go:1869: Performing extra runtime validation.
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"","y":"s"},"resourceTags":{"x":"s","y":"s"}}
      examples_go_test.go:201: key=appconfig-app tags are as expected: {"x":"s","y":"s"}
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"","y":"s"},"resourceTags":{"x":"s","y":"s"}}
      examples_go_test.go:201: key=bucket tags are as expected: {"x":"s","y":"s"}
      examples_go_test.go:204: 
            Error Trace:    /home/runner/work/pulumi-aws/pulumi-aws/examples/examples_go_test.go:204
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1870
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1792
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1682
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1559
            Error:          Not equal: 
                            expected: map[string]string{"x":"s", "y":"s"}
                            actual  : map[string]string{"x":"", "y":""}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,4 +1,4 @@
                             (map[string]string) (len=2) {
                            - (string) (len=1) "x": (string) (len=1) "s",
                            - (string) (len=1) "y": (string) (len=1) "s"
                            + (string) (len=1) "x": (string) "",
                            + (string) (len=1) "y": (string) ""
                             }
            Test:           TestRandomTagsCombinationsGo/test-75
            Messages:       bad bucket tags

three


    aws:s3:BucketV2 (bucketv2):
      error: 1 error occurred:
        * creating S3 Bucket (bucketv2-fa45cff): operation error S3: CreateBucket, https response error StatusCode: 409, RequestID: 7EJB2W9G0KM64VF5, HostID: n3aGqCDIF0ox3FY2tu79uwAGLSARlwd9gJVk/wtTDjptnT95wmJxN1bsCjOy3dchPKAlROck3NE=, BucketAlreadyOwnedByYou:

  Resources:
      + 4 created

four

     program.go:1869: Performing extra runtime validation.
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":""},"resourceTags":{"x":"s"}}
      examples_go_test.go:201: key=appconfig-app tags are as expected: {"x":"s"}
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":""},"resourceTags":{"x":"s"}}
      examples_go_test.go:201: key=bucket tags are as expected: {"x":"s"}
      examples_go_test.go:204: 
            Error Trace:    /home/runner/work/pulumi-aws/pulumi-aws/examples/examples_go_test.go:204
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1870
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1792
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1682
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1559
            Error:          Not equal: 
                            expected: map[string]string{"x":"s"}
                            actual  : map[string]string{"x":"s", "y":""}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,3 +1,4 @@
                            -(map[string]string) (len=1) {
                            - (string) (len=1) "x": (string) (len=1) "s"
                            +(map[string]string) (len=2) {
                            + (string) (len=1) "x": (string) (len=1) "s",
                            + (string) (len=1) "y": (string) ""
                             }
            Test:           TestRandomTagsCombinationsGo/test-59
            Messages:       bad bucket tags
      program.go:1451: Destroying stack

five

      command.go:106: Command completed without output
      program.go:1869: Performing extra runtime validation.
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"","y":""},"resourceTags":{}}
      examples_go_test.go:201: key=bucket tags are as expected: {"x":"","y":""}
      examples_go_test.go:198: phase: 2
      examples_go_test.go:199: state: {"defaultTags":{"x":"","y":""},"resourceTags":{}}
      examples_go_test.go:201: key=legacy-bucket tags are as expected: {"x":"","y":""}
      examples_go_test.go:207: 
            Error Trace:    /home/runner/work/pulumi-aws/pulumi-aws/examples/examples_go_test.go:207
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1870
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1792
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1682
                                        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/pkg/v3@v3.104.2/testing/integration/program.go:1559
            Error:          Not equal: 
                            expected: map[string]string{"x":"", "y":""}
                            actual  : map[string]string{"x":"s", "y":"s"}

                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,4 +1,4 @@
                             (map[string]string) (len=2) {
                            - (string) (len=1) "x": (string) "",
                            - (string) (len=1) "y": (string) ""
                            + (string) (len=1) "x": (string) (len=1) "s",
                            + (string) (len=1) "y": (string) (len=1) "s"
                             }
            Test:           TestRandomTagsCombinationsGo/test-56
            Messages:       bad legacy bucket tags
      program.go:1451: Destroying stack
      command.go:34: **** Invoke '/usr/local/bin/pulumi destroy --non-interactive --yes --skip-preview' in '/home/runner/go/src/stackName-1707352404396370602-665013'

Example

N/A

Output of pulumi about

N/A

Additional context

N/A

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

t0yv0 commented 8 months ago

Caused release failure see https://github.com/pulumi/ci-mgmt/issues/817

VenelinMartinov commented 7 months ago

Here are some more examples of the failure: https://github.com/pulumi/pulumi-aws/actions/runs/7916718574/job/21611796602

t0yv0 commented 6 months ago

Failed again in https://github.com/pulumi/pulumi-aws/pull/3767 with latest additions.

t0yv0 commented 5 months ago

This has been fixed now.