Build failed: dotnet-helix-machines-daily/main #2024070201

dotnet-eng-status[bot] commented 2 months ago

Build #2024070201 failed

:x: : internal / dotnet-helix-machines-daily failed

Summary

Finished - Tue, 02 Jul 2024 12:10:46 GMT Duration - 2 minutes Requested for - Microsoft.VisualStudio.Services.TFS Reason - schedule

Details

SynchronizeSecrets

:x: - [Log] - Unhandled Exception: Microsoft.Rest.Azure.CloudException: The access token is from the wrong issuer 'https://sts.windows.net/975f013f-7f24-47e8-a7d3-abc4752bf346/'. It must match the tenant 'https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/' associated with this subscription. Please use the authority (URL) 'https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later.
:warning: - [Log] - Extra secret 'AADTenantId' consider deleting it.
:warning: - [Log] - Extra secret 'AnalysisNotificationConnectionString' consider deleting it.
:warning: - [Log] - Extra secret 'AnalysisQueueName' consider deleting it.
:warning: - [Log] - Extra secret 'AutoScaleApplicationId' consider deleting it.
:warning: - [Log] - Extra secret 'AutoScaleApplicationInsightsKey' consider deleting it.
:warning: - [Log] - Extra secret 'AutoScaleWebJobStorageConnectionString' consider deleting it.
:warning: - [Log] - Extra secret 'AzureDevOpsExtensionSecret' consider deleting it.
:warning: - [Log] - Extra secret 'AzureSubscriptionId' consider deleting it.
:warning: - [Log] - Extra secret 'BuildAssetRegistryWriteConnectionString' consider deleting it.
:warning: - [Log] - Extra secret 'CloudServicesAppInsightsKey' consider deleting it.

Changes

5afd2aaa - Na Wen - Update KV name to use the new one cert with auto rotation (!40636)
de4ab7d9 - Doug Bunting - Unmonitor queues w/ deprecated bases (!40865)
65eed44a - DotNet Bot - [main] Update dependencies from dotnet/arcade (!40692)

dougbu commented 2 months ago

secret-manager problem started w/ build #2024063001. A bit more info might point in the right direction:

Synchronizing secret helixagentlogs-connection-string, type azure-storage-connection-string
Secret helixagentlogs-connection-string scheduled for rotation on 6/30/2024 12:10:53 PM +00:00, will rotate.
Generating new value(s) for secret helixagentlogs-connection-string...
##[error]Unhandled Exception: Microsoft.Rest.Azure.CloudException: The access token is from the wrong issuer 'https://sts.windows.net/975f013f-7f24-47e8-a7d3-abc4752bf346/'. It must match the tenant 'https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/' associated with this subscription. Please use the authority (URL) 'https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47' to get the token. Note, if the subscription is transferred to another tenant there is no impact to the services, but information about new tenant could take time to propagate (up to an hour). If you just transferred your subscription and see this error message, please try back later.
   at Microsoft.Azure.Management.Storage.StorageAccountsOperations.ListWithHttpMessagesAsync(Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.Storage.StorageAccountsOperationsExtensions.ListAsync(IStorageAccountsOperations operations, CancellationToken cancellationToken)
   at Microsoft.DncEng.SecretManager.StorageUtils.FindAccount(String accountName, StorageManagementClient client, CancellationToken cancellationToken) in /_/src/SecretManager/Microsoft.DncEng.SecretManager/StorageUtils.cs:line 139
   at Microsoft.DncEng.SecretManager.StorageUtils.RotateStorageAccountKey(String subscriptionId, String accountName, RotationContext context, TokenCredentialProvider tokenCredentialProvider, CancellationToken cancellationToken) in /_/src/SecretManager/Microsoft.DncEng.SecretManager/StorageUtils.cs:line 22
   at Microsoft.DncEng.SecretManager.SecretTypes.AzureStorageConnectionString.RotateValue(Parameters parameters, RotationContext context, CancellationToken cancellationToken) in /_/src/SecretManager/Microsoft.DncEng.SecretManager/SecretTypes/AzureStorageConnectionString.cs:line 45
   at Microsoft.DncEng.SecretManager.SecretType`1.RotateValues(TParameters parameters, RotationContext context, CancellationToken cancellationToken) in /_/src/SecretManager/Microsoft.DncEng.SecretManager/SecretType.cs:line 111
   at Microsoft.DncEng.SecretManager.Commands.SynchronizeCommand.RunAsync(CancellationToken cancellationToken) in /_/src/SecretManager/Microsoft.DncEng.SecretManager/Commands/SynchronizeCommand.cs:line 219

@chcostam, @AlitzelMendez, @missymessa, @epananth did you make any manual changes to that secret❔

Otherwise, it's probably the new reference from the #3090 fix to the .NET Release Infrastructure subscription. we already have to fixup things in that area b/c the name is too long (see #3228 but I don't get the connection to this failure — if there is a connection. Fix would be to move pool into the Helix sub.

dougbu commented 2 months ago

Assigning to @garath b/c he has a suspicion the problem is w/in secret-manager itself and he is working on upgrading that tool

garath commented 2 months ago

Fix is in dotnet/dnceng#3310. That change needs to flow to helix-machines once it's published.

dougbu commented 2 months ago

Fix is in #3310. That change needs to flow to helix-machines once it's published.

dotnet-helix-machines only subscribes to dotnet/arcade changes. will dotnet tool update microsoft.dnceng.secretmanager --add-source {dotnet-eng feed URI} --prerelease do the trick❔

garath commented 2 months ago

Yep, once it's published.

garath commented 1 month ago

Helix-machines got the update in 40963.

dotnet / dnceng