Use BeginAsync/EndAsync methods and remove the use of ThreadPool

[CheloXL]

A good enhancement could be to use the BeginAsync/EndAsync methods of the stream class, and remove all the uses of the ThreadPool. That would make the system faster as it will not have to create threads to make synchronous operations asynchronous. It's a waste of resources.

[sta]

Hmm?

At least now websocket-sharp needs a thread to receive the message.

If uses methods such as the BeginAsync/EndAsync style methods, it needs to use the callback method to loop BeginAsync method. It means that create another thread.

Do you have another idea using the BeginAsync/EndAsync style methods ?

[CheloXL]

What I meant is that instead of simply reading the information synchronically, you can use BeginReceive/EndReceive to not "lock" the thread while reading information (well, the thread will actually be locked, but not all the other threads).

Same applies to sending data. You can leave the Send method (that it is synchronous right now), but change the SendAsync to use BeginSend/EndSend.

Of course that you are right, you need at least one Thread per service. But the beauty of Begin[Receive|Send] are that they do not spawn a new thread on the threadpool.

Another optimization could be to replace the locks with SemaphoreSlim(s) with a reentrancy of 1, since you actually don't need a full memory lock, but just a semaphore so one thread won't try to send/receive data while other is already doing that. But I'm not sure that's so critical...

[sta]

Indeed, i think that SendAsync method of WebSocketService class is required improvement, it's necessary to determine whether modify or delete this method with improving broadcast data method, and methods like BeginSend/EndSend for asynchronous data transfer may be required to WebSocket class.

But, do you think that any BeginOperationName style methods don't create another thread using threadpool (or other function) inside these methods ?

[CheloXL]

See the following links on why BeginAsync/EndAsync (when available) are better alternatives to the same sync method running on the threadpool.

http://stackoverflow.com/questions/9453560/why-use-async-requests-instead-of-using-a-larger-threadpool and http://stackoverflow.com/questions/145304/when-to-use-thread-pool-in-c

Don't pay attention to some specifics to ASP.NET. The threadpool is the same for all the processes running under the same application.

[sta]

Hmm?

Why use async requests instead of using a larger threadpool?

This Q&A of stackoverflow mentions about new async/await syntax of new feature of C# 5.0. This new async/await syntax is not same as classic BeginOperationName/EndOperationName style methods.

When to use thread pool in C#?

This Q&A of stackoverflow mentions about selectively creating of thread according to the purpose (Using threadpool is almost better ?).

Using prepared BeginOperationName/EndOperationName style methods for async access may be safer than using threadpool directly. But it means that these methods don't allow to create a thread more than necessary, does not mean that don't use another thread or threadpool inside these methods.

In addition, it depends on the implementation of each BeginOperationName/EndOperationName methods whether safer or not.

I found no difference only about whether using threadpool (or other thread function) direct or indirect. Of course, it will be required more attention to use threadpool directly because of the limited number of thread resources.

I think new async/await syntax is great, but the result of these syntax may use threadpool at the bottom ?

Do you know some site to reference about inside of new async/await syntax ?

[CheloXL]

BeginOperation/EndOperation do not really create a new thread on the threadpool (in this specific case, where the stream has an IOCP). So, basically, those operations create a wrapper around an IOCP.

Regarding the inners of async/await, here you have some information: http://channel9.msdn.com/Shows/Going+Deep/Mads-Torgersen-Inside-C-Async http://blogs.msdn.com/b/pfxteam/archive/2012/04/12/async-await-faq.aspx http://www.reflector.net/2012/05/understanding-async-code-net-reflector/

Threads are not necessary for asynchronous programming. "Asynchronous" means that the API doesn't block the calling thread. It does not mean that there is another thread that is blocking. I know it is hard to get at the beginning if you never worked with systems that works in an async fashion but do not have threads...

Also, you can check http://msdn.microsoft.com/en-us/library/hh191443.aspx regarding async/await. I'm quoting here a specific paragraph regarding threading:

The async and await keywords don't cause additional threads to be created. Async methods don't require multithreading because an async method doesn't run on its own thread. The method runs on the current synchronization context and uses time on the thread only when the method is active. You can use Task.Run to move CPU-bound work to a background thread, but a background thread doesn't help with a process that's just waiting for results to become available.

[sta]

Well, thanks for your post about async/await syntax. I was able to better understand.

Indeed, the method using async/await syntax don't create a new thread by itself because the async part of the method is posted more selective to the Task class function.

Quoted from Asynchronous Programming with Async and Await (C# and Visual Basic) - Threads

The async-based approach to asynchronous programming is preferable to existing approaches in almost every case. In particular, this approach is better than BackgroundWorker for IO-bound operations because the code is simpler and you don't have to guard against race conditions. In combination with Task.Run, async programming is better than BackgroundWorker for CPU-bound operations because async programming separates the coordination details of running your code from the work that Task.Run transfers to the threadpool.

Q. How does the Task class do exec the async part ? A. Hmm,,, it seems that the Task class use threadpool to exec the async part.

I know that the Task class uses threadpool at inside. This case is just to use threadpool indirectly.

To get back about BeginAsync/EndAsync.

The BeginAsync/EndAsync methods that you say are classic BeginOperationName/EndOperationName style methods for async access since C# 2.0 (or 1.0) ?

If it's so, why do you bring up mention about new async/await at this issue thread about using classic feature ? It's because you think that both classic and new have a similar mechanism ?

If it's so, i can mention same things about BeginAsync/EndAsync methods too.

Indeed, the BeginAsync/EndAsync methods don't create a new thread by itself because the async work of the methods is delegated to thread function of another resource.

I think that it's because the thread mechanism are hidden very well if you think that async API don't need any other thread, at least on C#/.NET.

I have a simple question for you. Q. For what does AsyncWaitHandle property exist in IAsyncResult interface that is used by BeginOperationName/EndOperationName methods ?

Type of AsyncWaitHandle property is WaitHandle class. This object is used to block the thread until the async work on another thread to be completed. What is reason for its existence if BeginOperationName/EndOperationName methods don't use any other thread ?

[sta]

Do you talk about asynchronous coding style in C#, by any chance ?

[CheloXL]

I believe we are mixing two things here. One is asynchronous programming, and the other is threading/tasks.

Let's forget for a moment about threading and go back in time, to the first x86 era (the first 8086... not 286, 386 or 486). By that time (and even before that... I found async programming while working with a z80) you do not have threads, tasks or anything else. You don't even know that a thing called thread were about to exists. All the code was always executed in a single process.

When you had to read something from disk, or play a sample through the soundcard, you (and I'm simplifying a lot here) basically told the device controller (the disk controller, or the sound controller... things that were not bound to CPU execution): "Hey, here you have this command: Read this sector(s) and put the result into this memory address".

After that command, your program continues to do whatever it was doing. No freezing, no waiting for data. When the (disk/soundcard/networkcard) controller had the data, it fires an "interrupt", where of course you had to previously attach some code. At that moment, the execution of your program was halted and the code attached to the interrupt was executed. Once the code in the interrupt was executed, your program returned to its previous state and continue working.

Now, back to today: Technology advanced a lot, but the underlining mechanism for IO devices still follows that pattern: You send a command to the device's controller, and once it is done, it notifies you. There are no threads involved at all simply because that concept doesn't exists.

BeginAsync/EndAsync (on IOCP) do that. They work by handling the operation on IO devices following that pattern. That's why no threads are created. You don't need one.

So, to go back to my original suggestion: Taking advantage of the BeginAsync/EndAsync methods will improve the overall functionality of the application because you do not need any threading for handling the async request.

[CheloXL]

So, basically, this is how BeginAsync/EndAsync works (again, when working with IOCP devices):

1. Your program starts (usually in a thread)
2. Your program calls a BeginAsync method. The BeginAsync methods tells the relevant IO controller to do whatever action you commanded.
3. Your program continues on the same thread. No new threads were created, but the controller is processing the command.
4. The controller finishes the job. It dispatches an event that in the end will call the EndAsync delegate.
5. Your program is interrupted. The EndAsync delegate executes.
6. When the EndAsync delegate exists, your program continues from where it left.

So, as you can see, no thread switching nor new threads are created on the process. If you use the sync method in a new thread (so, to not block the current thread) you are wasting resources because what the sync method does is to spin/block the thread until the end event is dispatched. So, you have context switching, new threads created on the threadpool that can cause thread starvation, etc...

[sta]

Would not be a need to extend the story so far ?

I talk about the current C# (C# 3.5 limited to websocket-sharp). It's important for me how the function is implemented in C#.

So, i have some questions for you.

Q. The BeginAsync/EndAsync methods that you say are classic BeginOperationName/EndOperationName style methods for async access since C# 2.0 (or 1.0) ?

Q. If it's not so, what specifically is implementaion of the BeginAsync/EndAsync style methods in C# ? Is there your implementaion of BeginAsync/EndAsync style methods sample ?

Q. What specifically is the IOCP in standard C#/.NET without unmanaged dll ?

Q. Why did you bring up mention about new async/await syntax while you said it's not necessary to use threadpool though the result of new syntax use threadpool at the bottom.

Q. If async I/O don't use the thread, it use callback worker after accepting I/O result, is it same as before ?

I'm feeling a shift to talk with you.

Abstract, conceptual stories are not really matter to me. Because I can not trust them unless i see the Implementation or something close to it.

So, could you answer to me about above questions ?

[CheloXL]

Q. The BeginAsync/EndAsync methods that you say are classic BeginOperationName/EndOperationName style methods for async access since C# 2.0 (or 1.0) ?

Yes, they are the same. The only difference is that BeginOperationName/EndOperationName only worked on IOCP devices. BeginAsync/EndAsync now work in that way on IOCP devices, and by creating new threads on other systems. So basically, it is an unified way to work with async methods by abstracting the thread creation where you need them.

Q. If it's not so, what specifically is implementaion of the BeginAsync/EndAsync style methods in C# ? Is there your implementaion of BeginAsync/EndAsync style methods sample?

As I mentioned, they are the same for IOCP ports (so, all the streams but memory stream work by no creating new threads to handle the async process).

Q. What specifically is the IOCP in standard C#/.NET without unmanaged dll?

All the C# code that interact with the system, at the end, call unmanaged DLLs. There is no other way to handle, lets say, a disk read without calling an unmanaged DLL. Check the .net source if you want and you will see that all the filesystem operations are external calls to the winapi. Same is for accessing any other device.

Q. Why did you bring up mention about new async/await syntax while you said it's not necessary to use threadpool though the result of new syntax use threadpool at the bottom.

I mentioned the new syntax because I use .net 4. Nothing else. Just because I'm used to. There are no differences with the "old" BeginOperationName/EndOperationName on how they work. And probably because working with tasks is easier, but I didn't know that the project was targeted to .net 3.5. My fault here if I messed things up and confused things up.

Q. If async I/O don't use the thread, it use callback worker after accepting I/O result, is it same as before?

It is not the same because as I mentioned, the callback worker will interrupt the current thread action as it will execute in the same thread and when it finished, the thread will continue from where it was interrupted. If you want the main thread to continue working without interruption, you can of course create a new thread on the EndOperationName callback so the user callback executes on its own thread.

That really depends on how much processing you have to do on the callback. The first option (no new thread) is useful when the callback doesn't has a lot to process (so, the main thread is not interrupted that much). With the second option, the callback can execute whatever you want to do, at the expense of creating a new thread and context switching.

In any case and for the second option, that's the optimal solution if you want to provide the user with the ability to execute a long action on the callback, as while the network card is receiving data and until the data is completed, there is no thread creation/switching and the created thread is not spinning just waiting for the network card to read all the data.

And please, don't take me wrong, but those are not abstract stories. I've been working with computers for 26 years and I really know what I'm talking about :)

[sta]

Thank you so much for your answers.

You wrote:

And please, don't take me wrong, but those are not abstract stories. I've been working with computers for 26 years and I really know what I'm talking about :)

So sorry. I pay honor to your 26 years career.

Let me continue to further questions on your answers.

Q. What specifically is the IOCP in standard C#/.NET without unmanaged dll ?

Your A.

All the C# code that interact with the system, at the end, call unmanaged DLLs. There is no other way to handle, lets say, a disk read without calling an unmanaged DLL. Check the .net source if you want and you will see that all the filesystem operations are external calls to the winapi. Same is for accessing any other device.

This question was not a correct question. I correct this as follows.

Q. What specifically is the class that supports the IOCP in standard C#/.NET without thirdparty's managed and unmanaged library ?

I think that it's the System.Net.Sockets.Socket class, off course, maybe some others. If it's so, does the IOCP family that you say means that the Socket class and the class that uses the Socket class with own network I/O ?

If it's so, the System.Ner.HttpListener class will be included in the IOCP family ?

[CheloXL]

The Socket class is one of the classes that supports IOCP. But as I mentioned, they do that by working with unmanaged libraries.

There is an internal class, UnsafeNclNativeMethods, that has all the calls to the windows native libraries that deals with sockets, crypto, filesystem, ole32, etc.

And yes, you are correct. The HttpListener class also uses IOCP. In fact, if you disassemble that class, you will see a lot of calls to the above class to manage IO.

[sta]

Well, HttpListener.BeginGetContext/HttpListener.EndGetContext method pair is one pair of BeginAsync/EndAsync family that you say.

And, HttpListener.BeginGetContext method syntax is as follows.

public IAsyncResult BeginGetContext(AsyncCallback callback, Object state)

Q. If async I/O don't use the thread, it use callback worker after accepting I/O result, is it same as before? Your A.

It is not the same because as I mentioned, the callback worker will interrupt the current thread action as it will execute in the same thread and when it finished, the thread will continue from where it was interrupted. If you want the main thread to continue working without interruption, you can of course create a new thread on the EndOperationName callback so the user callback executes on its own thread.

Does your answer means that threadpool (or another thread) is not used to execute callback of BeginGetContext ?

[CheloXL]

Thread pool maintains 2 types of threads – Worker & I/O. As name implies Worker threads are computational threads while I/O are used for wait (block) of long duration (e.g. when invoking a remote service). A good rule to follow is to ensure all your waits are on I/O thread.

Ex:

int wt, iot;
ThreadPool.GetAvailableThreads(out wt, out iot);
Console.WriteLine("Worker = " + wt);
Console.WriteLine("I/O = " + iot);

One can only do a request for a worker thread by ThreadPool.QueueUserWorkItem. If all worker threads are occupied your request will be blocked till time a worker thread is available.

int wt, iot;
for (int i = 0; i < 10; i++) {
    ThreadPool.QueueUserWorkItem(Dummy);
}

Console.ReadLine();
ThreadPool.GetAvailableThreads(out wt, out iot);
Console.WriteLine("Worker = " + wt);
Console.WriteLine("I/O = " + iot);

static void Dummy(object o) {
    //any sync operation...
    Thread.Sleep(50000000);
}

Unlike worker threads you don’t have any direct API access to request I/O threads. But .NET leverages I/O threads automatically when you use asynchronous programming.

for (int i = 0; i < 10; i++) {
    /* Invoke a WCF Service via a proxy, asynchronously*/
    client.BeginGetData(10, null, null); /*Put some Thread.Sleep in server side code*/
}
ThreadPool.GetAvailableThreads(out wt, out iot);
Console.WriteLine("Worker = " + wt);
Console.WriteLine("I/O = " + iot);

As you probably didn't expect, the output of the above program shows that only one I/O thread is in use instead of 10. This optimization makes I/O so important. You want to ensure that minimal threads are blocked and I/O threads just give you that. Also note that when the call returns another I/O thread is picked up from pool and callback method gets executed.

Here you have some nice explanation by Microsoft: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365683(v=vs.85).aspx

[sta]

I think maybe such your answer makes me feel a misalignment of the topics. I've asked about only HttpListener.BeginGetContext method.

Well, i continue to ask questions.

The before that i say means as follows.

Do sync GetContext method -> Post the work that uses the result of GetContext to the threadpool

And the after as follows.

Do async BeginGetContext method -> Exec the `callback` of BeginGetContext that uses the result of BeginGetContext.

Q. How many times does it decrease to use the threadpool from the before to the after ?

And,

Quoted from HttpListener Class (System.Net).

In the more complex asynchronous model, your application does not block while waiting for requests and each request is processed in its own execution thread. Use the BeginGetContext method to specify an application-defined method to be called for each incoming request. Within that method, call the EndGetContext method to obtain the request, process it, and respond.

Q. What does the above quote means ? It looks like a contradiction with what you say.

[CheloXL]

Sorry if you feel that I'm misaligned with the topic. But I started the topic by saying that you should use the BeginNameAsync methods of the stream class because that would make the system faster.

Asynchrony is essential for activities that are potentially blocking. If such an activity is blocked within a synchronous process, the entire application must wait. In an asynchronous process, the application can continue with other work that doesn't depend on the web resource until the potentially blocking task finishes.

In typical server-side scenarios, multiple threads are allocated to process multiple overlapping client requests.

Whenever one of these threads performs a blocking operation (i.e. an operation that accesses the network or some other form of I/O) the thread will remain idle waiting for the operation to complete. If during the interval new requests arrive, each of them is assigned its own thread.

Depending mainly on the rate at which new client requests arrive, and the time it takes for each request to be served, the number of threads executing (or blocked) on the server might grow rapidly.

Threads happen to have a significant memory footprint (about 1 MB of virtual memory space); therefore too many simultaneous threads on a server can easily max out the available memory long before processor utilization comes near a 100%, becoming the main bottleneck for the server’s throughput.

In cases like these it is possible to improve scalability without changing hardware by using non-blocking calls when communicating to external resources: threads don’t need to waste time waiting for those calls to complete, and instead can be returned to the thread pool so that they can be reused to service other incoming requests.

Non-blocking I/O calls help in keeping the thread count low, removing the memory bottleneck and making it possible for the application to scale better on the same hardware.

I shown you, on my last comment, an example about that. I used a WCF service as example just for simplicity, but it is the same for any other class that uses IOCP internally (WCF uses HTTP that uses TCP that uses Sockets). Even with files on disk that would work. Only one thread is allocated to handle all the 10 requests, while on the other example (the way you are actually allocating threads on the server), 10 threads are created.

Regarding the HttpListener: The HttpListener, specifically, uses IOCP internally to handle the async and then gets a thread from the threadpool to execute the callback. That it is done in that way because the HttpListener may need to process authentication/authorization and other stuff that can take a lot of time and if it blocks the IO thread to do that, it will not be able to handle other requests in the meantime. On the other side, the TcpListener does work with the IO thread and doesn't touch the ThreadPool.

In either way (the HttpListener way or the TcpListener way), you are still gaining performance because, on the first case, the HttpListener will get a thread from the threadpool only when the request is completed and that thread will be eventually freed and put back into the pool (so, the limit of threads taken from the pool will depend on the amount of work that thread has to do with the request, and not by the number of requests), and on the second case, the number of threads taken from the threadpool will always be 1, and will be an IO thread. And of course, for the second case, you can always get a thread from the pool on the EndNameAsync method of the TcpListener to continue working and do not block the IO thread, just like what the HttpListener does.

[sta]

Hey, what you mean is the following ?

Asynchronous Stream Processing

This includes also what i want to say.

Quoted from Asynchronous Stream Processing

The benefit of this, however, depends on how the streams themselves implement the asynchronous BeginRead and BeginWrite methods. The implementation of these methods on the base Stream class simply executes Read and Write from the ThreadPool using an asynchronous delegate invocation. So while each read and write will be asynchronous, from a performance perspective this will likely be slower overall than just doing the reads and writes synchronously.

I've already mentioned as follows.

Indeed, the BeginAsync/EndAsync methods don't create a new thread by itself because the async work of the methods is delegated to thread function of another resource.

Or, is it better as follows ?

The BeginAsync/EndAsync methods don't create a new thread by itself because the async work of the methods depends on the other I/O process.

So, i'm already full for your IOCP story, i want you to stop it.

The contradiction that i've mentioned is about the following. You wrote:

Threads are not necessary for asynchronous programming. "Asynchronous" means that the API doesn't block the calling thread. It does not mean that there is another thread that is blocking. I know it is hard to get at the beginning if you never worked with systems that works in an async fashion but do not have threads...

So, i've wanted to ask what does "Threads are not necessary for asynchronous programming." means. Could you answer this question if it means without using even I/O thread on C# ?

And, You wrote:

Only one thread is allocated to handle all the 10 requests, while on the other example (the way you are actually allocating threads on the server), 10 threads are created.

I've already mentioned about this as follows.

Indeed, i think that SendAsync method of WebSocketService class is required improvement, it's necessary to determine whether modify or delete this method with improving broadcast data method, and methods like BeginSend/EndSend for asynchronous data transfer may be required to WebSocket class.

What do you want to say ?

For now, i plan to modify the method of receiving message. But I'm still pending about the cases of HttpListener/TcpListener.

I don't trust the IOCP that you say, so much. Do you have some experiences on the actual use ?

For example, like the following. libtorrent blog: asynchronous disk I/O

Oops! This is a question about the IOCP lol

[CheloXL]

When I said "Threads are not necessary for asynchronous programming" I meant exactly that, on a generic context. I were not talking specifically about c#. When I explained how things works a long time ago, I explained how async programming worked without using threads (at that time there were no threads at all). Now, of course async programming uses threading, simply because it makes a lot of sense.

The TcpListener uses only 1 thread from the IO thread pool. You don't need to trust me, you can just browse the code on the TcpListener class. And as I already mentioned, the HttpListener uses internally one thread from the IO pool to handle requests/response, and on the EndNameAsync of the internal thread is called, the HttpListener executes its own EndNameAsync delegate on a new thread from the threadpool.

And yes, I have experience on using IOCP on the TcpListener. Some time ago I created a propietary RPC mechanism that worked using the Tcp stack, and found myself migrating all the sync code I created at that time to async code to overcome the issues I early mentioned. I found a reduction in memory usage of about 70% on high loads, and a speed increase of about 60% (most of the speed increase was related to the threading issue I mentioned about context switching and thread creation).

Regarding your last example... it seems that the issue is not with IOCP, but with how other (non-windows) systems implement that. Not sure how that applies here, but if a system is not well designed... sorry for the users of that system :) I never average for the lowest common denominator, unless there are very, very good reasons to do so.

[sta]

You wrote:

And yes, I have experience on using IOCP on the TcpListener...

Hmm,,,i've wanted to ask about your experience of actual use the IOCP (or other async I/O functions) on native (unmanaged) environment. So i've mentioned about above blog article... Well, i don't need your answer about this any more, thx.