Added initial benchmarks for creation and diffing

twop commented 2 years ago

closes #21

Done

very simple recursive creation and processing messages benchmarks. It is roughly based on the same view code

let rec view (depth: int) =
    Stack() {
        Label($"label1:{depth}")
            .textColor("red")
            .automationId($"label1:{depth}")

        Label($"label2:{depth}")
            .textColor("green")
            .automationId($"label2:{depth}")

        Button($"btn: {depth}", Depth depth)

        if (depth > 0) then view(depth - 1)
    }

Initial results

Create just widget tree

Method	depth	Mean	Error	StdDev
CreateWidgets	100	127.1 μs	0.20 μs	0.16 μs
CreateWidgets	1000	2,845.5 μs	56.45 μs	116.58 μs

Processing 100 messages + mount

Method	depth	Mean	Error	StdDev
ProcessMessages	100	61.72 ms	0.560 ms	0.468 ms
ProcessMessages	1000	1,438.17 ms	23.754 ms	21.057 ms

Note

Results are machine dependent and I will take regular snapshots and make before/after comparison when I optimize things
It Is based on a very very simple UI runtime created just for tests. Thus the goal is to benchmark the core itself as much as possible.
Just these 2 benchmarks initially, we will add more over time. For now I'm particularly interested in creation and diffing.
I fixed tests as a part of this change, yes I know, I should have done it in a separate PR ^_^
I'm benchmarking with .net 6, we should probably make Mono work as well, but I was unable to to figure out how yet (m1 issues probably)

TimLariviere commented 2 years ago

That looks very good!

Just a few remarks:

Any chance we can benchmark allocations as well?
A depth of 1000, not sure if it's really useful to benchmark this. In real world, I think the max depth will be 50. So maybe we can change the test cases to be 10, 50, 100?
Any idea why going from 100 to 1000, we have a times 23 increase in process time?

TimLariviere commented 2 years ago

I'm benchmarking with .net 6, we should probably make Mono work as well, but I was unable to to figure out how yet (m1 issues probably)

I'll run the benchmarks on macOS Intel and Windows, and post the results here

twop commented 2 years ago

With new code

Method	depth	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
ProcessMessages	10	175.2 ms	2.00 ms	1.78 ms	68333.3333	17666.6667	3333.3333	222 MB
ProcessMessages	15	3,250.6 ms	59.98 ms	53.17 ms	770000.0000	178000.0000	78000.0000	2,470 MB

Method	depth	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
CreateWidgets	10	356.9 us	7.08 us	6.63 us	193.3594	75.1953	-	551 KB
CreateWidgets	20	82,410.7 us	875.48 us	818.92 us	19285.7143	3428.5714	857.1429	68,073 KB

twop commented 2 years ago

@TimLariviere I modified benchmark to be more "realistic", when I profiled it the perf was dominated by string builders and attributes creation. I don't have access to visual studio profiler on win, thus I'm wondering what will you see.

Note at the end of Benchmarks.fs I left commented code that I use to manually run through the profiler (I use dotTrace)

TimLariviere commented 2 years ago

Results for my machines:

Windows 11

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
Intel Core i5-9600KF CPU 3.70GHz (Coffee Lake), 1 CPU, 6 logical and 6 physical cores
.NET SDK=6.0.101
  [Host]   : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT DEBUG
  .NET 6.0 : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT

Job=.NET 6.0  Runtime=.NET 6.0

Method	depth	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
CreateWidgets	10	323.7 μs	1.85 μs	1.54 μs	106.4453	38.5742	-	551 KB
CreateWidgets	20	101,939.1 μs	1,878.27 μs	1,756.94 μs	11600.0000	2800.0000	800.0000	68,072 KB
ProcessMessages	10	173.7 ms	2.13 ms	1.89 ms	40000.0000	5000.0000	2000.0000	222 MB
ProcessMessages	15	4,920.9 ms	63.42 ms	59.32 ms	510000.0000	196000.0000	96000.0000	2,470 MB

macOS Monterey (Intel)


BenchmarkDotNet=v0.13.1, OS=macOS Monterey 12.1 (21C52) [Darwin 21.2.0]
Intel Core i7-6820HQ CPU 2.70GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.101
  [Host]   : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT DEBUG
  .NET 6.0 : .NET 6.0.1 (6.0.121.56705), X64 RyuJIT

Job=.NET 6.0  Runtime=.NET 6.0

Method	depth	Mean	Error	StdDev	Gen 0	Gen 1	Gen 2	Allocated
CreateWidgets	10	439.6 μs	3.39 μs	3.00 μs	120.6055	45.4102	-	551 KB
CreateWidgets	20	159,309.5 μs	3,076.23 μs	3,159.06 μs	14000.0000	3000.0000	750.0000	68,072 KB
ProcessMessages	10	257.5 ms	3.79 ms	3.36 ms	46000.0000	14000.0000	2000.0000	222 MB
ProcessMessages	15	7,457.8 ms	33.21 ms	31.06 ms	605000.0000	196000.0000	96000.0000	2,470 MB

TimLariviere / Fabulous-new