search

Heliosearch / heliosearch

The next generation of open source search
http://heliosearch.org
91 stars 19 forks source link

Configurable Threadpools #22

Open joel-bernstein opened 10 years ago

joel-bernstein commented 10 years ago

It would be great if there was a way to configure named threadpools that were managed by Heliosearch. These threadpools could then shared by features that could benefit from parallel execution.

yonik commented 10 years ago

Good idea... One general thing to watch out for... any bounded threadpool should be thought of like a lock/semaphore. Two different modules, each with their own bounded threadpool, calling each other is a deadlock recipe. Even code that calls itself recursively (as faceting does) could deadlock with itself using a bounded threadpool.

Another related thing that would be useful: a completion service that only used N threads from a given thread pool (i.e. you have a bigger thread pool, but you don't want one request to dominate it). See PerSegmentSingleValuedFaceting

Another thought is that the size of a threadpool would often be related to maximizing throughput (i.e. for a CPU bound task, it would be related to the number of CPU cores on a box). Given that, it almost makes most sense for these threadpools to be global (on the CoreContainer, shared across all cores). I'm not sure what that means as far as configuration though.... solrconfig.xml wouldn't be the right spot for anything on CoreContainer.