Run nightly benchmarks with a single thread executor?

[jpountz]

We have some code that only gets executed when running with an executor, e.g. the logic that top docs collectors have to share information about minimum competitve scores across threads. But we don't exercise it at all in nightly benchmarks today. Worse we have collector managers that don't try to share information at all across threads today, leading to lots of duplicate work when searching with threads, but nobody knows since nightly benchmarks never run with an executor.

Using threads on nightly benchmarks is probably not a good idea as it would increase noise.

So what about running nightly benchmarks with a single-thread executor? This would still exercise the logic that tries to share information across collectors that run in different threads without adding more noise?

[mikemccand]

I really like this idea @jpountz! It would exercise an important code path.

Maybe the noise for more than one thread would be acceptable? If we limit the threadpool to well below the core count on the box so there's no heavy contention, maybe?