The way to benchmark multithreaded actions.

[kayanme]

Consider some object, which should hold a lot of concurrent operations at the same time. There is the need to benchmark one operation from one thread (still having others working) to verify productivity during parallel load. The problem is how to correctly hold that background threads still measuring one needed.

I see some ways, but none of them are completely possible now.

1. An ability to run benchmark methods in parallel, so every thread runs its own measuring still providing the needed concurrency.
2. Establish background in setup method. Unfortunatly, there is a need of "unsetup" method, where threads should be stopped.

[AndreyAkinshin]

@kayanme, this is a very great idea. We definitely should add this feature. Unfortunately, it is hard to implement right way. There are a lot of multi-threading side effects like false sharing which can easily spoil a benchmark. We will add it sometimes, but not in the nearest future.

[jmbryan22]

danielpalme/IocPerformance#77

[roji]

Just a big +1 to adding support for properly benchmarking in a concurrent environment - it's super important to be able to see how code behaves under concurrent load (lock contention, CPU caches...). Some knobs would be necessary - number of threads etc.

[spati2]

+1

[nxrighthere]

This should be one of the top priorities, please.

[lydonchandra]

+1

[osjoberg]

+1

[im-aIex]

While not an exact replacement, with the deprecation of the Visual Studio load tests, it would be great to be able to migrate some the loads tests to benchmarks and get some similar data out of it (customize # of 'users'/threads, time per run, etc.).

[kirsan31]

Any news/thoughts on this?

[ashishnegi]

I have simpler scenario where i want to benchmark operations like Add on ConcurrentDictionary like collections, parameterizing the number of threads. i.e. how does this collection's a particular operation scale with the concurrent requests ? For inspiration, google's benchmark library support this.

[lupengamzn]

+1

[mikehung]

+1

[sakno]

+1

[timcassell]

Would it make sense to have a helper class to do this, rather than attributes and code-gen?

class Benchmark
{
    private ThreadHelper threadHelper;

    [GlobalSetup]
    public void Setup()
    {
        threadHelper = new ThreadHelper(
            () => { ... },
            maxConcurrency
        );
    }

    [Benchmark]
    public void Multithread()
    {
        ...
        threadHelper.Execute();
    }
}

I've done something like this for concurrency tests during unit testing, so I could do it again for this if it makes sense, but we'd need to settle on a good API.

[Edit] This would only solve measuring concurrent actions. Running benchmarks in parallel is a separate beast.

[timcassell]

I wrote a helper to make it simpler and more efficient to benchmark multi-threaded actions than manually setting up threads. https://github.com/timcassell/ProtoBenchmarkHelpers This can be used until BDN adds multi-threaded benchmarking officially.

[Stabzs]

@timcassell does your helper work for async actions? The generation of async state machines seems to wreak havoc on all BDN benchmarking and while your helper is great for running multiple threads in parallel, it doesn't seem to be sufficient for async methods. Either way, thanks for your work, it is fantastic!

[timcassell]

@Stabzs I did not have async in mind when I wrote that. I'm not sure adding async support would be of much use, though. If your async actions hop to different threads while they're running, all the benefits of the custom thread management goes out the window, and you might as well just use Task.Run and Task.WhenAll. Do you have a specific use-case in mind?

[Stabzs]

@timcassell thanks for the quick response!

There's a handful of scenarios that I'd like to benchmark where I need to run multiple concurrent tasks to simulate contention across threads, such as the memory cost of asynchronously waiting on a potentially contended SemaphoreSlim instance. Some optimizations (pooling, ValueTask, etc etc), can have significant allocation reductions. However, the overhead from arrays of tasks and Task.Run skew the allocations of the asynchronous workers themselves, making it difficult to measure performance.

In addition, both allocations and elapsed time tend to be very sporadic when benchmarking async methods.

Overall, it doesn't seem that there is a great way to benchmark async methods. I've used your helper before, but in hindsight, it probably is the wrong approach.

[timcassell]

In addition, both allocations and elapsed time tend to be very sporadic when benchmarking async methods.

Well that's not surprising if they're hopping to different threads.

Some optimizations (pooling, ValueTask, etc etc), can have significant allocation reductions. However, the overhead from arrays of tasks and Task.Run skew the allocations of the asynchronous workers themselves, making it difficult to measure performance.

That's a good point. I have an idea how to support this. I'll open a new issue on that repo for this.

[timcassell]

@Stabzs In the meantime, you could try synchronously waiting on the tasks inside the actions. task.GetAwaiter().GetResult(). That's how BDN currently consumes tasks anyway.

[Stabzs]

@timcassell I appreciate the suggestion! I'll give that a try and see if it changes the results.

[timcassell]

No-one has suggested what the API for this should look like yet, so here's my idea.

public class Benchmarks
{
    [ParamsAllValues]
    public bool MyFlag { get; set; }

    [Benchmark]
    public void SingleThread() { }

    [ThreadedBenchmark(MaxConcurrency = 2)] // Default MaxConcurrency is Environment.ProcessorCount
    public class MultiThread : Benchmarks // Inherit from the parent type to have access to the same params as regular benchmarks
    {
        [ThreadInvoke] // Invoke once on a thread
        public void Action() { }

        [ThreadInvoke(3)] // Invoke 3 times, each on a different thread
        public int Func() => 42;

        // Support the same return types as regular benchmark methods
        [ThreadInvoke]
        public async Task AsyncAction() { }
    }
}

That would produce a table like this:

Method	MyFlag	Mean
SingleThread	False	0.00 ns
MultiThread	False	0.00 ns
SingleThread	True	0.00 ns
MultiThread	True	0.00 ns

This encapsulates the threaded actions within a child class (even though the result table labels it as a Method, I think it's quite readable).

[CDboyOne]

+1 We also need multi-threading benchmark very much