search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.57k stars 4.82k forks source link

[QUERY] What causes a "System.Net.WebException: The operation has timed out" while listing resources? #16877

Closed daveoshinsky closed 2 years ago

daveoshinsky commented 4 years ago

Using Assembly Microsoft.Azure.ResourceManager, Version=2.0.0.0, we are seeing a web timeout exception shown below. Is this because of a problem on the client side, a network problem, or an Azure outage of some kind? Prior to this error, the same code has been working reliably.

61113 eeb9 11/11 21:22:37 1842 Exception- System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: The operation has timed out. at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in :0 at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in :0 at System.Threading.Tasks.TaskFactory1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func2[T,TResult] endFunction, System.Action1[T] endAction, System.Threading.Tasks.Task1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in :0 --- End of stack trace from previous location where exception was thrown --- at System.Net.Http.MonoWebRequestHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0041b] in <8c16ab9f87dc459985afab8723556d8d>:0 --- End of inner exception stack trace --- at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x00042] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess (System.Threading.Tasks.Task task) [0x0001c] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00009] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Hyak.Common.RetryHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00163] in <e5563ee354f14017a3914d2e39dc3606>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x00042] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess (System.Threading.Tasks.Task task) [0x0001c] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00009] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter1[TResult].GetResult () [0x00000] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Azure.Management.Resources.ResourceOperationsExtensions.List (Microsoft.Azure.Management.Resources.IResourceOperations operations, Microsoft.Azure.Management.Resources.Models.ResourceListParameters parameters) [0x0003a] in :0 at CloudSDKWrapper.AzureResourceManagement.GetResourcesList (System.String resourceGroup, CloudSDKWrapper.EResourceType type) [0x0004f] in <55ffac162b8c4e729ba42164d34c6b4d>:0 61113 eeb9 11/11 21:22:37 1842 Failed to get resourcegroup of storage account [sgn-cvlt] 61113 eeb9 11/11 21:22:37 1842 FetchCosmosDBAccountCredentials() - Failed to obtain Cosmos DB credentials for account [sgn-cvlt] with message ''

ghost commented 4 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Wmengmsft, @MehaKaushik, @shurd

jsquire commented 4 years ago

Thank you for your feedback. Tagging and routing to the team member best able to assist.

markcowl commented 4 years ago

@daveoshinsky Are you using the Microsoft.Azure.Management.ResourceManager nuget package, or the Microsoft.Azure.Management.Resources nuget package? And can you tell us which version of the package you are using?

daveoshinsky commented 4 years ago

We are directly using Microsoft.Azure.ResourceManager.dll version 2.18.2.0, dated 7/31/2015. This was not referenced using NuGet. Following are the DLL properties.

image

YalinLi0312 commented 4 years ago

Can you update to the latest Microsoft.Azure.Management.ResourceManager nuget package and try it again?

daveoshinsky commented 4 years ago

The software interfaces in the new DLL you suggest are very different from those in the old DLL we are using. It looks like I cannot switch to it quickly, without some development work. Would you happen to know where the old DLL (Microsoft.Azure.ResourceManager.dll version 2.18.2.0, dated 7/31/2015) came from, and how the new package Microsoft.Azure.Management.ResourceManager relates to it?

YalinLi0312 commented 4 years ago

The package you currently use is pretty old, I'm not clear where it is. And don't find it in Nuget.

markcowl commented 4 years ago

@daveoshinsky The big issue is with the implementation of retries and long-running operations, which is in the ClientRuntime package. Unfortunately, the package you are using is using a Client runtime package that is more than 4 years old. There are significant breaking changes in the API since this package was produced, unfortunately.

If it is impossible to update, there are some settings on the client itself, and some general http settings that cna help witht his issue, particularly, ConnectionPooling and KeepAlive settings may impact the frequency of timeouts.

daveoshinsky commented 4 years ago

This timeout occurred in retrieving a list of Azure resource groups, after 100 seconds. I see there's a way to provide an HttpClient with increased HttpClient.Timeout to the ResourceManagementClient constructor, in the old SDK. Is a 100 second timeout because of a recoverable error? Or is it non-recoverable, like a complete network outage?

namespace Microsoft.Azure.Management.Resources { public class ResourceManagementClient : ServiceClient, IResourceManagementClient, IDisposable { ....... public ResourceManagementClient(SubscriptionCloudCredentials credentials); ....... public ResourceManagementClient(SubscriptionCloudCredentials credentials, HttpClient httpClient); ....... }

ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.

Issue Details
Using Assembly Microsoft.Azure.ResourceManager, Version=2.0.0.0, we are seeing a web timeout exception shown below. Is this because of a problem on the client side, a network problem, or an Azure outage of some kind? Prior to this error, the same code has been working reliably. 61113 eeb9 11/11 21:22:37 1842 Exception- System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: The operation has timed out. at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in :0 at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in :0 at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in :0 --- End of stack trace from previous location where exception was thrown --- at System.Net.Http.MonoWebRequestHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x0041b] in <8c16ab9f87dc459985afab8723556d8d>:0 --- End of inner exception stack trace --- at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x00042] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess (System.Threading.Tasks.Task task) [0x0001c] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00009] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Hyak.Common.RetryHandler.SendAsync (System.Net.Http.HttpRequestMessage request, System.Threading.CancellationToken cancellationToken) [0x00163] in :0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x00042] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess (System.Threading.Tasks.Task task) [0x0001c] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00009] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Runtime.CompilerServices.TaskAwaiter`1[TResult].GetResult () [0x00000] in <8301cb0e88d04fd59e2245fc8907971f>:0 at Microsoft.Azure.Management.Resources.ResourceOperationsExtensions.List (Microsoft.Azure.Management.Resources.IResourceOperations operations, Microsoft.Azure.Management.Resources.Models.ResourceListParameters parameters) [0x0003a] in :0 at CloudSDKWrapper.AzureResourceManagement.GetResourcesList (System.String resourceGroup, CloudSDKWrapper.EResourceType type) [0x0004f] in <55ffac162b8c4e729ba42164d34c6b4d>:0 61113 eeb9 11/11 21:22:37 1842 Failed to get resourcegroup of storage account [sgn-cvlt] 61113 eeb9 11/11 21:22:37 1842 FetchCosmosDBAccountCredentials() - Failed to obtain Cosmos DB credentials for account [sgn-cvlt] with message ''
Author: daveoshinsky
Assignees: YalinLi0312
Labels: `ARM`, `Mgmt`, `Service Attention`, `customer-reported`, `needs-team-attention`, `question`
Milestone: -
KranthiPakala-MSFT commented 3 years ago

@armleads-azure - Following up to see if there is any update on this issue? - Thank you

navba-MSFT commented 2 years ago

@daveoshinsky Apologies for the late reply. Could you please try the sample mentioned here which uses the Microsoft.Azure.Management.ResourceManager dotnet SDK to list the Azure resources and let us know if you still face this issue ?

Namespace:Microsoft.Azure.Management.ResourceManager.Fluent Assembly: Microsoft.Azure.Management.ResourceManager.Fluent.dll Package: Microsoft.Azure.Management.ResourceManager.Fluent v1.38.0

Awaiting your reply.

ghost commented 2 years ago

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!