JoJoJet / bevy-trait-query

adds trait queries to the bevy game engine
Apache License 2.0
65 stars 11 forks source link

Significant performance differences since benchmarks were last posted #51

Open ItsDoot opened 6 months ago

ItsDoot commented 6 months ago

Decided to compare the benchmark results from https://github.com/JoJoJet/bevy-trait-query/commit/c757a179a5cbff5a85f4504643aa95e423755c0c (when benchmark results were last posted) until https://github.com/JoJoJet/bevy-trait-query/commit/2869f9d369e731c50b4abfd2b44c4e02f38323b8 (latest commit at time of writing).

I noticed some significant performance differences, and am curious why that could be:

     Running benches/all.rs (target/release/deps/all-164e82f8f06f0a40)
Gnuplot not found, using plotters backend
All<> - 1 match         time:   [86.178 µs 86.584 µs 87.044 µs]
                        change: [+63.099% +63.800% +64.533%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high severe

10000
All<> - 2 matches       time:   [115.23 µs 115.41 µs 115.63 µs]
                        change: [+33.806% +34.788% +35.940%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe

20000
All<> - 1-2 matches     time:   [99.420 µs 99.742 µs 100.10 µs]
                        change: [+41.814% +43.307% +44.795%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe

15000
     Running benches/concrete.rs (target/release/deps/concrete-fdfa9e097d2b391d)
Gnuplot not found, using plotters backend
concrete - 1 match      time:   [10.453 µs 10.460 µs 10.468 µs]
                        change: [+30.343% +31.453% +32.302%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

10000
concrete - 2 matches    time:   [10.536 µs 10.571 µs 10.609 µs]
                        change: [+21.170% +22.323% +23.419%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  5 (5.00%) high mild
  6 (6.00%) high severe

10000
     Running benches/fragmented.rs (target/release/deps/fragmented-599b12242e573cfa)
Gnuplot not found, using plotters backend
concrete - fragmented   time:   [1.0395 µs 1.0597 µs 1.0862 µs]
                        change: [-3.1553% -1.5826% -0.1409%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

4497169560
One<> - fragmented      time:   [3.1337 ns 3.1424 ns 3.1598 ns]
                        change: [-40.444% -40.207% -39.959%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  7 (7.00%) high mild
  8 (8.00%) high severe

0
All<> - fragmented      time:   [5.5079 µs 5.5820 µs 5.6784 µs]
                        change: [-0.0460% +0.7549% +1.6465%] (p = 0.08 > 0.05)
                        No change in performance detected.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high severe

2014870000
     Running benches/one.rs (target/release/deps/one-6682cb865896f1c3)
Gnuplot not found, using plotters backend
One<>                   time:   [27.594 µs 27.610 µs 27.630 µs]
                        change: [+0.1968% +1.0798% +2.2258%] (p = 0.03 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe

10000
One<> - filtering       time:   [14.427 µs 14.456 µs 14.496 µs]
                        change: [-0.8238% -0.6224% -0.3342%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

5000
ItsDoot commented 6 months ago

Here's 3 more runs, on Windows 11 this time (Linux Mint prior): https://gist.github.com/ItsDoot/571107bd776bdc4a4178c520d77e612a