cyanfish
commented
10 months ago A few things. First, I would verify there's not an actual deadlock issue.
Second, long-running tasks should in general be avoided. For something like SQL, you should be using async/await so that it yields the thread. If you're using EF, that supports this. Though I see you mention it's an old codebase so maybe that's not feasible, but it is the "correct" solution.
Failing all that, you can use a custom ThreadPool/TaskScheduler to dispatch the work. Or perhaps you could create multiple server processes and load-balance between them.
zachrybaker
Ben, great work here, appreciate the investment of your time.
I've seeing what appears to be client-server serviceability issues when stress testing using this package as a way to process separate some data services that use an old synchronous codebase of mainframe code replatformed to C#. Trying not to literally run the mainframe inside of .net web apps :)
I set up a scenario where i have 3 clients utilizing 40 threads each, where each thread is keeping one request outstanding for data at time with the server. So at any one point, there may be 120 requests outstanding against the data service, calling one of about 350 different methods that are codegenerated.
The client side reuses a single static NamedPipeChannel and a static client service. The server's NamedPipeServer is also singular and reused, as is the server services bound to it, and the server does not explicitly create a thread for fulfilling the request, it just utilizes the threadpool. Over time it will grow to have north of a hundred threads involved.
I have set the timeout to infinite, because the data services unfortunately are pretty unpredictable in terms of their run times when executed parallel due to blocking and locking on the sql server side.
What I'm seeing is that over time these static services stop completing and even servicing requests.
Your note here on the long-running tasks and the .net task scheduler is interesting. In my case I also was really hoping that the thread pool servicing the pipe could be used to run the work of the implementation of the service, but I see you suggesting he should probably use newed threads for long running tasks.
In my case creating threads is a sizeable overhead in typical loads as many of the requests are going to run to completion in less than 50ms. But there are some that take seconds. I'm going to test generating code with background threads to run the data load, just to see if anything behaves differently, and perhaps create a 2nd threadpool for the work itself and ConcurrentQueue my way out of the mess. But I was curious if you've seen a scenario like I'm describing solved in a way I have not described here?