houseofcat
commented
3 years ago So a couple of things, Task.Run is basically sugar syntax for StartNew, with default parameters. It does have an option to have a custom TaskScheduler on one StartNew overload which will achieve what you are saying what you want.
https://devblogs.microsoft.com/pfxteam/task-run-vs-task-factory-startnew/
Long running doesn't work the way I think lots of people think it does. It's not a dedicated thread as far as I can recall. Long running means "hey task scheduler, don't expect me back for a while". This affects hysteresis, the threshold for generating new threads (by default new threads - up to SetMaxThreads() - is only performed once every 0.5 seconds). So that would be all you use lose, is if these were the last 2 threads in the pool, you would have to wait 1.0 seconds of CPU time, to get replacement threads assuming they were returned to the optimum levels (this is a whole hard to explain thing sorry). Even then if you use async and await that isn't fully commited threads (i.e. I took two threads). The thread used will eventually yield because of the code. So for a very hot calculation loop, that's synch, like is prime, long running is chosen because the thread really won't be back for a while. For a properly async await method it's virtually unnecessary, just in my opinion because the mechanism returns that very thread. In instances where you have a bit of hot looping before an await occurs, you would avoid a "lock in" by having called await Task.Yield(); to force background processing immediately. So also you have to understand that Async/Await is allowing the shuffling of threads inside and outside of that Task for the life time of that application. Also, with lower overhead due to SynchronizationContext not in use. The only time a thread is "assigned" is when its not "awaiting" so essentially between await steps, make sense? So creating an Infinitely running task/loop, doesn't mean I stole a thread, its only using a thread when its invoking code, otherwise the ThreadPool gets it back for other things.
I think 77 pipeline consumers, for the most part, sounds ridiculous for a single application... But I then remember we have 40 in a single kestrel app ourselves. The CPU utilization on idle is maybe 0-1%. The problem is curation of active consumers.
- How much work is the whole system (including non-consumer code) global system performing.
- Which queue is bursting traffic, etc.
- How many 100% full throttled queues can we handle per our hardware.
The top 3 queues - our ingestion queues - pegged at 100% siphon off all performance from the application (i.e. Threadpool starvation). The CPU usage at this point is 80-95% in one application.
There's nothing to actually fix though. It is a knowable quantity through observation, metrics, and benchmarking etc. I had back and forths with this developer that implemented his design with the library and while it did not alleviate his frustration, I kept explaining its doing what you are telling it to do, if you let everything burst simultaneously, in a highly threaded application, it will shit its pants because we don't have 1,000,000 green threads as goroutines like golang. It's that simple I am afraid.
I was still able to improve the design with some hotfixes and observing how the developer used the application. I made some quality of life improvements -> Global Consumer Options is one of those features. This allowed him to mass tweak the entire performance in high periods of traffic to self-throttle vs.... going back to the drawing board and rethink his "monolithic" one application does everything design - which he still should do.
This software scales up to the hardware, just like RabbitMQ. It won't hold your hand in this regard, which sounds like what you are attempting to do and got a little bit by. Splitting traffic off by horizontal scaling is one way of solving this, partition queues between applications is the best way, because you are dedicated a box to your hottest queues etc.
Now, for performance reasons and working with Golang I had a few ideas I researched as a kind of a version 2 to pipelines. Naturally, I was quiet pleased with Pipelines but did see an inherent weakness in ThreadPool contention on very large apps that couldn't readily be solved without custom TaskSchedulers/ThreadPools - which is why that code never existed too till recently, I don't like that as a solve.
I spent some time in GCP dataflow documentation, tensorflow documentation, and reading OLD .NET documents around Dataflows. I lacked some familiarity to design out a viable example, so I spent a bit learning each and every dataflow block and how it could be molded into something performant. It turns out, raw parallelism is the key to allowing some insulation to Thread Contention, the problem with that is, you need parallel hardware to do it. You won't be able to run 77 Dataflows with degrees of parallelism greater than 1, but they would run happily at 1. So it scaled to the hardware a lot better than Pipelines, I definitely encourage you to consider using the Dataflow RabbitMQ structure/pattern if you really want to melt your CPU.
Eventually - any hyper thread app with no upper limits - will still starve the ThreadPool, which sounds exactly like what you ran into.
Regarding the BufferBlock, while Channels GC has improved in Net5.0, I don't know know exactly what Stephen is showing. Really shouldn't use unbounded anythings in my opinion. It is possible BufferBlock is making a copy. BufferBlock is chained to the next block, in isolation it may not make sense, but in an entire flow its vastly more efficient and necessary for predictable and smooth performance.
tiggerite
This is more of a discussion point than an issue really, as the functionality is ok as it is.
However, because ConsumerPipeline uses Task.Run to create FeedPipelineWithDataTasks, this blocks one of the threadpool threads, and even using SetMinThreads doesn't seem to really alleviate the impact this has when running in Kestrel - it starts to struggle badly when there are more than around 70 of them.
https://github.com/houseofcat/Tesseract/blob/master/src/HouseofCat.RabbitMQ.Pipelines/ConsumerPipeline.cs#L73 https://github.com/houseofcat/Tesseract/blob/master/src/HouseofCat.RabbitMQ.Pipelines/ConsumerPipeline.cs#L82
Should it instead use Task.Factory.StartNew with the option of TaskCreationOptions.LongRunning? As far as I understand it, this would then force the creation of a new thread and then SetMinThreads would have more of an impact, also the threadpool wouldn't lose a thread (unexpectedly).
To get around this issue at the moment I simply switch to using Dataflow instead when the number is sufficiently large, which is okay, but does use a bit more memory; I guess to be expected as it uses BufferBlock as per https://devblogs.microsoft.com/dotnet/an-introduction-to-system-threading-channels/#relationship-to-the-rest-of-net-core and not a huge problem for now.