Anecdotally, workloads which are eligible for tagging cannot run for more than 24 hours and fail with the following error.
error
```
An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.
Unhandled exception. Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.)
---> System.AggregateException: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.)
---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.
---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
---> System.IO.IOException: The read operation failed, see inner exception.
---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Security.SslStream'.
at System.Net.Security.SslStream.g__ThrowExceptional|123_0(ExceptionDispatchInfo e)
at System.Net.Security.SslStream.DecryptData(Int32 frameSize)
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Http.HttpConnection.InitialFillAsync(Boolean async)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
at Azure.Core.HttpPipelineMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, Boolean doNotThrow, Boolean retry, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.SendPostAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.SendPostAsync(Uri endpoint, IDictionary`2 headers, IDictionary`2 bodyParameters, ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders, Func`2 onBeforePostRequestData)
at Microsoft.Identity.Client.OAuth2.OAuth2Client.GetTokenAsync(Uri endPoint, RequestContext requestContext, Boolean addCommonHeaders, Func`2 onBeforePostRequestHandler)
at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ILoggerAdapter logger)
at Microsoft.Identity.Client.OAuth2.TokenClient.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, String scopeOverride, String tokenEndpointOverride, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.SilentRequestHelper.RefreshAccessTokenAsync(MsalRefreshTokenCacheItem msalRefreshTokenItem, RequestBase request, AuthenticationRequestParameters authenticationRequestParameters, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.Silent.CacheSilentStrategy.TryGetTokenUsingFociAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.Silent.CacheSilentStrategy.RefreshRtOrFailAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.Silent.CacheSilentStrategy.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.Silent.SilentRequest.ExecuteAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
at Microsoft.Identity.Client.ApiConfig.Executors.ClientApplicationBaseExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenSilentParameters silentParameters, CancellationToken cancellationToken)
at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
at Azure.Identity.MsalPublicClient.AcquireTokenSilentCoreAsync(String[] scopes, String claims, IAccount account, String tenantId, Boolean async, CancellationToken cancellationToken)
at Azure.Identity.MsalPublicClient.AcquireTokenSilentAsync(String[] scopes, String claims, IAccount account, String tenantId, Boolean async, CancellationToken cancellationToken)
at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
---> (Inner Exception #1) System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at Azure.Core.CancellationHelper.ThrowOperationCanceledException(Exception innerException, CancellationToken cancellationToken)
at Azure.Core.CancellationHelper.ThrowIfCancellationRequested(CancellationToken cancellationToken)
at Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken, Exception inner, TimeSpan timeout)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
--- End of inner exception stack trace ---
at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage)
at Azure.Identity.SharedTokenCacheCredential.GetTokenImplAsync(Boolean async, TokenRequestContext requestContext, CancellationToken cancellationToken)
at Azure.Identity.SharedTokenCacheCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
at Microsoft.Azure.PowerShell.Authenticators.MsalAccessToken.Renew()
at Microsoft.Azure.PowerShell.Authenticators.MsalAccessToken.AuthorizeRequest(Action`2 authTokenSetter)
at Microsoft.WindowsAzure.Commands.Storage.Common.Cmdlet.NewAzureStorageContext.GetTokenStrFromAccessToken(IAccessToken accessToken)
at Microsoft.WindowsAzure.Commands.Storage.Common.Cmdlet.NewAzureStorageContext.<>c__DisplayClass99_0.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.Azure.Storage.Auth.TokenCredential.RenewTokenAsync(Object state)
at System.Threading.Tasks.Task.<>c.b__128_1(Object state)
at System.Threading.QueueUserWorkItemCallback.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
VERBOSE: 2023-09-09T19:54:46.4242261-04:00: Awaiting Operation: '9fe5c7a4-f30c-412f-8c37-ee6fb03dd83e', Start: '09/08/2021 00:00:00 +00:00', End: '09/08/2021 06:00:00 +00:00'. Current wait time: 00:06:05.6266995
2023-09-09T19:54:46.4246362-04:00: Completed Operation: '32a7ebc2-9ebb-4a60-a1a0-c22a13b75ade', Start: '09/07/2021 06:00:00 +00:00', End: '09/07/2021 12:00:00 +00:00'.
VERBOSE: 2023-09-09T19:54:46.4300062-04:00: Awaiting Operation: '735dd511-af3a-4019-9e84-26505c865209', Start: '09/08/2021 06:00:00 +00:00', End: '09/08/2021 12:00:00 +00:00'. Current wait time: 00:06:05.3766995
2023-09-09T19:54:46.4406450-04:00: Completed Operation: 'dbbfda0f-5588-49b8-b872-bcf648481e71', Start: '09/07/2021 18:00:00 +00:00', End: '09/08/2021 00:00:00 +00:00'.
2023-09-09T19:54:46.4416992-04:00: Completed Operation: 'd3c58a7e-17ff-4802-9285-083faad86dba', Start: '09/07/2021 00:00:00 +00:00', End: '09/07/2021 06:00:00 +00:00'.
2023-09-09T19:54:46.7713522-04:00: Completed Operation: '2afeced1-e4fa-448f-a326-5de2c878211f', Start: '09/08/2021 12:00:00 +00:00', End: '09/08/2021 18:00:00 +00:00'.
VERBOSE: 2023-09-09T19:54:46.7713336-04:00: Awaiting Operation: 'c933d373-ff88-4906-9b86-0a5dd8ea4b48', Start: '09/08/2021 18:00:00 +00:00', End: '09/09/2021 00:00:00 +00:00'. Current wait time: 00:06:05.0485740
VERBOSE: 2023-09-09T19:54:46.8777916-04:00: Awaiting Operation: '82987175-b3db-4ecb-b975-9a562f55711a', Start: '09/09/2021 06:00:00 +00:00', End: '09/09/2021 12:00:00 +00:00'. Current wait time: 00:06:04.9235739
2023-09-09T19:54:46.8777916-04:00: Completed Operation: 'dc80bb7c-1bc8-47b1-a9b7-5ee1f6b9c096', Start: '09/09/2021 00:00:00 +00:00', End: '09/09/2021 06:00:00 +00:00'.
[process exited with code 3762504530 (0xe0434352)]
You can now close this terminal with Ctrl+D, or press Enter to restart.
```
I suspect this is due to Azure auth token expiry. Spike & debug.
Anecdotally, workloads which are eligible for tagging cannot run for more than 24 hours and fail with the following error.
error
``` An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit. Unhandled exception. Azure.Identity.AuthenticationFailedException: SharedTokenCacheCredential authentication failed: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.) ---> System.AggregateException: Retry failed after 2 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.) ---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'. at System.Net.Security.SslStream.I suspect this is due to Azure auth token expiry. Spike & debug.