Open ItsDoot opened 6 months ago
Decided to compare the benchmark results from https://github.com/JoJoJet/bevy-trait-query/commit/c757a179a5cbff5a85f4504643aa95e423755c0c (when benchmark results were last posted) until https://github.com/JoJoJet/bevy-trait-query/commit/2869f9d369e731c50b4abfd2b44c4e02f38323b8 (latest commit at time of writing).
I noticed some significant performance differences, and am curious why that could be:
Running benches/all.rs (target/release/deps/all-164e82f8f06f0a40) Gnuplot not found, using plotters backend All<> - 1 match time: [86.178 µs 86.584 µs 87.044 µs] change: [+63.099% +63.800% +64.533%] (p = 0.00 < 0.05) Performance has regressed. Found 6 outliers among 100 measurements (6.00%) 6 (6.00%) high severe 10000 All<> - 2 matches time: [115.23 µs 115.41 µs 115.63 µs] change: [+33.806% +34.788% +35.940%] (p = 0.00 < 0.05) Performance has regressed. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe 20000 All<> - 1-2 matches time: [99.420 µs 99.742 µs 100.10 µs] change: [+41.814% +43.307% +44.795%] (p = 0.00 < 0.05) Performance has regressed. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high severe 15000 Running benches/concrete.rs (target/release/deps/concrete-fdfa9e097d2b391d) Gnuplot not found, using plotters backend concrete - 1 match time: [10.453 µs 10.460 µs 10.468 µs] change: [+30.343% +31.453% +32.302%] (p = 0.00 < 0.05) Performance has regressed. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) high mild 1 (1.00%) high severe 10000 concrete - 2 matches time: [10.536 µs 10.571 µs 10.609 µs] change: [+21.170% +22.323% +23.419%] (p = 0.00 < 0.05) Performance has regressed. Found 13 outliers among 100 measurements (13.00%) 1 (1.00%) low severe 1 (1.00%) low mild 5 (5.00%) high mild 6 (6.00%) high severe 10000 Running benches/fragmented.rs (target/release/deps/fragmented-599b12242e573cfa) Gnuplot not found, using plotters backend concrete - fragmented time: [1.0395 µs 1.0597 µs 1.0862 µs] change: [-3.1553% -1.5826% -0.1409%] (p = 0.03 < 0.05) Change within noise threshold. Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe 4497169560 One<> - fragmented time: [3.1337 ns 3.1424 ns 3.1598 ns] change: [-40.444% -40.207% -39.959%] (p = 0.00 < 0.05) Performance has improved. Found 15 outliers among 100 measurements (15.00%) 7 (7.00%) high mild 8 (8.00%) high severe 0 All<> - fragmented time: [5.5079 µs 5.5820 µs 5.6784 µs] change: [-0.0460% +0.7549% +1.6465%] (p = 0.08 > 0.05) No change in performance detected. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high severe 2014870000 Running benches/one.rs (target/release/deps/one-6682cb865896f1c3) Gnuplot not found, using plotters backend One<> time: [27.594 µs 27.610 µs 27.630 µs] change: [+0.1968% +1.0798% +2.2258%] (p = 0.03 < 0.05) Change within noise threshold. Found 5 outliers among 100 measurements (5.00%) 1 (1.00%) high mild 4 (4.00%) high severe 10000 One<> - filtering time: [14.427 µs 14.456 µs 14.496 µs] change: [-0.8238% -0.6224% -0.3342%] (p = 0.00 < 0.05) Change within noise threshold. Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) high mild 2 (2.00%) high severe 5000
Here's 3 more runs, on Windows 11 this time (Linux Mint prior): https://gist.github.com/ItsDoot/571107bd776bdc4a4178c520d77e612a
Decided to compare the benchmark results from https://github.com/JoJoJet/bevy-trait-query/commit/c757a179a5cbff5a85f4504643aa95e423755c0c (when benchmark results were last posted) until https://github.com/JoJoJet/bevy-trait-query/commit/2869f9d369e731c50b4abfd2b44c4e02f38323b8 (latest commit at time of writing).
I noticed some significant performance differences, and am curious why that could be: