search

GoogleCloudPlatform / dotnet-docs-samples

.NET code samples used on https://cloud.google.com
https://cloud.google.com/dotnet
Apache License 2.0
732 stars 1.19k forks source link

[IAM]: Tests are flaky #1622

Open amanda-tarafa opened 2 years ago

amanda-tarafa commented 2 years ago

In particular if run concurrently: https://source.cloud.google.com/results/invocations/fb9a32a6-b22a-4fe7-b1fd-fe7c137c462b/log

amanda-tarafa commented 1 year ago

Another one here: https://source.cloud.google.com/results/invocations/c8b62f40-423e-487d-a70b-46cded4c3f3b/log

amanda-tarafa commented 1 year ago

And another one here: https://fusion2.corp.google.com/invocations/4adb6d71-fe0e-4cab-80f5-952b6526b951/log

amanda-tarafa commented 1 year ago

And more here: https://fusion2.corp.google.com/invocations/7c0070b7-a921-4ba6-ada9-bd10f4fa6654/log, for service accounts. The IDs for the resources seem to be generated based on sequential numbers and that of course causes a lot of overlap.

amanda-tarafa commented 1 year ago

Again, service account tests are flaky, or, service account creation takes a while to propagate, not sure which.

https://source.cloud.google.com/results/invocations/23208802-bd8f-4d8c-b4e3-feed40c83849/log

[xUnit.net 00:00:01.89]     GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts [FAIL]
  Failed GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts [1 s]
  Error Message:
   Google.GoogleApiException : The service iam has thrown an exception. HttpStatusCode is NotFound. Unknown service account
  Stack Trace:
     at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response)
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at ServiceAccounts.RenameServiceAccount(String email, String newDisplayName) in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\ServiceAccounts\ServiceAccountsRename.cs:line 36
   at GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts() in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\ServiceAccountTests\ServiceAccountTests.cs:line 20
amanda-tarafa commented 1 year ago

This is definetely because of tests overstepping each other, see here and here which executed roughly at the same time as part of #2281 checks:

Failed GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts [933 ms]
  Error Message:
   Google.GoogleApiException : The service iam has thrown an exception. HttpStatusCode is NotFound. Unknown service account
  Stack Trace:
     at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response)
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at ServiceAccounts.RenameServiceAccount(String email, String newDisplayName) in /tmpfs/src/github/dotnet-docs-samples/iam/api/ServiceAccounts/ServiceAccountsRename.cs:line 36
   at GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts() in /tmpfs/src/github/dotnet-docs-samples/iam/api/ServiceAccountTests/ServiceAccountTests.cs:line 20
Failed GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts [484 ms]
  Error Message:
   Google.GoogleApiException : The service iam has thrown an exception. HttpStatusCode is Conflict. Service account dotnet-test-8 already exists within project projects/dotnet-docs-samples-tests.
  Stack Trace:
     at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response)
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at ServiceAccounts.CreateServiceAccount(String projectId, String name, String displayName) in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\ServiceAccounts\ServiceAccountsCreate.cs:line 42
   at GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts() in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\ServiceAccountTests\ServiceAccountTests.cs:line 18
amanda-tarafa commented 8 months ago

Another flake #2438 

[xUnit.net 00:00:04.24]     GoogleCloudSamples.AccessTest.TestAccess [FAIL]
  Failed GoogleCloudSamples.AccessTest.TestAccess [2 s]
  Error Message:
   Google.GoogleApiException : The service cloudresourcemanager has thrown an exception. HttpStatusCode is Conflict. There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff. The request's ETag '\007\006\016\221\005\363y\335' did not match the current policy's ETag '\007\006\016\221\006\"\031\347'.
  Stack Trace:
     at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response)
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at AccessManager.SetPolicy(String projectId, Policy policy) in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\Access\SetPolicy.cs:line 33
   at GoogleCloudSamples.AccessTest.TestAccess() in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\AccessTest\AccessTest.cs:line 121
amanda-tarafa commented 5 months ago

Again in #2596 

[xUnit.net 00:00:01.81]     GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts [FAIL]
  Failed GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts [1 s]
  Error Message:
   Google.GoogleApiException : The service iam has thrown an exception. HttpStatusCode is Conflict. Service account dotnet-test-594 already exists within project projects/dotnet-docs-samples-tests.
  Stack Trace:
     at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response)
   at Google.Apis.Requests.ClientServiceRequest`1.Execute()
   at ServiceAccounts.CreateServiceAccount(String projectId, String name, String displayName) in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\ServiceAccounts\ServiceAccountsCreate.cs:line 42
   at GoogleCloudSamples.ServiceAccountTests.TestServiceAccounts() in C:\tmpfs\src\github\dotnet-docs-samples\iam\api\ServiceAccountTests\ServiceAccountTests.cs:line 18