microsoft / PowerPlatform-DataverseServiceClient

Code Replica for Microsoft.PowerPlatform.Dataverse.Client and supporting nuget packages.
MIT License
277 stars 50 forks source link

During performance test got "The request channel timed out attempting to send after 00:04:00" #280

Closed HectorMeneses333 closed 2 years ago

HectorMeneses333 commented 2 years ago

During performance test at 50 concurrent users we got the following error:

The request channel timed out attempting to send after 00:04:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. The HTTP request to 'https://irisoncology-uat.crm.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.46.4784' has exceeded the allotted timeout of 00:04:00. The time allotted to this operation may have been a portion of a longer timeout.

The related operation is creating an appointment emr entity in Dataverse. Most of the performance test calls succeeded and had response times of around 12 secs ( slow ). But one of them reported the above error.

I found the thread below that suggests setting appropriate values for Max Concurrent Calls, Max Concurrent Instances and Max Concurrent Sessions ( look for 1) Adjust Service Throttling Settings ): https://social.msdn.microsoft.com/Forums/vstudio/en-US/692f5631-c40c-40d7-9a98-8d0d7eb0994b/systemtimeoutexception-the-request-channel-timed-out-while-waiting-for-a-reply-after?forum=wcf

How can we modify those values in our system? Thanks!

MattB-msft commented 2 years ago

Take a look here : maximize throughput

You will see mention of the crmsvcclient, you can direct transpose dataverse serviceclient there. All the processes and properties are the same .

The full page is a very good read and worth the time of your dealing with mass operations.

HectorMeneses333 commented 2 years ago

Adding changes for //Change max connections from .NET to a remote service default: 2 System.Net.ServicePointManager.DefaultConnectionLimit = 65000; //Bump up the min threads reserved for this app to ramp connections faster - minWorkerThreads defaults to 4, minIOCP defaults to 4
System.Threading.ThreadPool.SetMinThreads(100,100);

did the trick...I'm not getting that error above anymore.

However, the docs page for SetMinThreads has this Caution:

_"Caution

By default, the minimum number of threads is set to the number of processors on a system. You can use the SetMinThreads method to increase the minimum number of threads. However, unnecessarily increasing these values can cause performance problems. If too many tasks start at the same time, all of them might appear to be slow. In most cases, the thread pool will perform better with its own algorithm for allocating threads. Reducing the minimum to less than the number of processors can also hurt performance."_

What is the recommended value for azure functions running in premium layer?

MattB-msft commented 2 years ago

Take a read though this: Azure Functions Best Practices

There is a good deal of guidance there on how to deal with those sorts of items.

HectorMeneses333 commented 2 years ago

Thanks Matt! I'm wondering how the following is related to SetMinThreads

new property "RecommendedDegreesOfParallelism". This property will report the recommended number of threads for communicating with Dataverse.

I can see in my system that I get a RecommendedDegreesOfParallelism =32

is it fair to say that setMinThreads(32, 32)

would ensure that when communicating with Dataverse that the 32 threads would be created on demand.

How is the RecommendedDegreesOfParallelism calculated in your side?

MattB-msft commented 2 years ago

Generally, that number is used to set the max concurrency number for any parallel operation. See : (Parallel programming in .NET)[https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/]

Insofar as how the number is calculated, that is a combination of a number of factors on the platform side ranging from infrastructure to number of nodes in the supporting cluster and size of host's