Closed ohbarye closed 5 months ago
This pull request adds a benchmark script to measure which concurrency method is the best for each scenario.
Based on the benchmark's result, I changed the default worker to :none since it's the happiest way for users.
:none
The following benchmarks are the results of running the benchmark suite.
This runs a script that does not do any IO or CPU bound work.
ruby benchmark/success_simple.rb ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22] Warming up -------------------------------------- ractor 20.000 i/100ms process 3.000 i/100ms thread 126.000 i/100ms none 668.000 i/100ms Calculating ------------------------------------- ractor 173.918 (±11.5%) i/s - 880.000 in 5.129007s process 28.861 (± 3.5%) i/s - 147.000 in 5.100393s thread 1.130k (± 5.5%) i/s - 5.670k in 5.031552s none 6.534k (± 2.3%) i/s - 32.732k in 5.011885s
This runs a script that does CPU bound work.
ruby benchmark/success_cpu_bound.rb Call tarai function with(9, 4, 0) ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22] Warming up -------------------------------------- ractor 3.000 i/100ms process 2.000 i/100ms thread 1.000 i/100ms none 1.000 i/100ms Calculating ------------------------------------- ractor 32.788 (± 6.1%) i/s - 165.000 in 5.057492s process 22.098 (± 4.5%) i/s - 112.000 in 5.080410s thread 7.439 (± 0.0%) i/s - 38.000 in 5.108195s none 7.494 (± 0.0%) i/s - 38.000 in 5.070547s
This runs a script that does IO bound work.
ruby benchmark/success_io_bound.rb ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22] Warming up -------------------------------------- ractor 11.000 i/100ms process 3.000 i/100ms thread 17.000 i/100ms none 22.000 i/100ms Calculating ------------------------------------- ractor 82.488 (±14.5%) i/s - 407.000 in 5.054559s process 35.403 (± 5.6%) i/s - 177.000 in 5.013818s thread 143.022 (± 7.7%) i/s - 714.000 in 5.021129s none 223.252 (± 9.0%) i/s - 1.122k in 5.071176s
This runs a script that fails and shrink happens.
ruby benchmark/failure_simple.rb ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22] Warming up -------------------------------------- ractor 6.000 i/100ms process 1.000 i/100ms thread 9.000 i/100ms none 815.000 i/100ms Calculating ------------------------------------- ractor 62.770 (±15.9%) i/s - 306.000 in 5.009858s process 1.783 (± 0.0%) i/s - 9.000 in 5.049606s thread 85.218 (± 9.4%) i/s - 423.000 in 5.007178s none 5.387k (± 3.3%) i/s - 27.710k in 5.149867s
Change
This pull request adds a benchmark script to measure which concurrency method is the best for each scenario.
Based on the benchmark's result, I changed the default worker to
:none
since it's the happiest way for users.Benchmarks
The following benchmarks are the results of running the benchmark suite.
Benchmark success:simple
This runs a script that does not do any IO or CPU bound work.
Benchmark success:cpu_bound
This runs a script that does CPU bound work.
Benchmark success:io_bound
This runs a script that does IO bound work.
Benchmark failure:simple
This runs a script that fails and shrink happens.