Closed byroot closed 10 hours ago
There is a large number of outstanding performance PRs that I want to merge, but we need a decent benchmark to judge if they are effective.
I went to borrow rapidjson's benchmark suite, which is a good start.
I only kept the comparison with Oj and RapidJSON, because YAJL is slower on most benchmarks, so little point comparing to it.
FYI @jhawthorn
== Encoding small nested array (121 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 88.225k i/100ms oj 209.862k i/100ms rapidjson 128.978k i/100ms Calculating ------------------------------------- json 914.611k (± 0.4%) i/s (1.09 μs/i) - 4.588M in 5.016099s oj 2.163M (± 0.2%) i/s (462.39 ns/i) - 10.913M in 5.045964s rapidjson 1.392M (± 1.3%) i/s (718.55 ns/i) - 6.965M in 5.005438s Comparison: json: 914610.6 i/s oj: 2162693.5 i/s - 2.36x faster rapidjson: 1391682.6 i/s - 1.52x faster == Encoding small hash (65 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 142.093k i/100ms oj 651.412k i/100ms rapidjson 237.706k i/100ms Calculating ------------------------------------- json 1.478M (± 0.7%) i/s (676.78 ns/i) - 7.389M in 5.000866s oj 7.150M (± 0.7%) i/s (139.85 ns/i) - 35.828M in 5.010756s rapidjson 2.250M (± 1.6%) i/s (444.46 ns/i) - 11.410M in 5.072451s Comparison: json: 1477595.1 i/s oj: 7150472.0 i/s - 4.84x faster rapidjson: 2249926.7 i/s - 1.52x faster == Encoding twitter.json (466906 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 101.000 i/100ms oj 223.000 i/100ms rapidjson 105.000 i/100ms Calculating ------------------------------------- json 1.017k (± 0.7%) i/s (982.83 μs/i) - 5.151k in 5.062786s oj 2.244k (± 0.7%) i/s (445.72 μs/i) - 11.373k in 5.069428s rapidjson 1.069k (± 4.6%) i/s (935.20 μs/i) - 5.355k in 5.016652s Comparison: json: 1017.5 i/s oj: 2243.6 i/s - 2.21x faster rapidjson: 1069.3 i/s - same-ish: difference falls within error == Encoding citm_catalog.json (500299 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 77.000 i/100ms oj 129.000 i/100ms rapidjson 96.000 i/100ms Calculating ------------------------------------- json 767.217 (± 2.5%) i/s (1.30 ms/i) - 3.850k in 5.021957s oj 1.291k (± 1.5%) i/s (774.45 μs/i) - 6.579k in 5.096439s rapidjson 959.527 (± 1.1%) i/s (1.04 ms/i) - 4.800k in 5.003052s Comparison: json: 767.2 i/s oj: 1291.2 i/s - 1.68x faster rapidjson: 959.5 i/s - 1.25x faster == Encoding canada.json (2090234 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 1.000 i/100ms oj 3.000 i/100ms rapidjson 1.000 i/100ms Calculating ------------------------------------- json 19.748 (± 0.0%) i/s (50.64 ms/i) - 99.000 in 5.013336s oj 31.016 (± 0.0%) i/s (32.24 ms/i) - 156.000 in 5.029732s rapidjson 19.419 (± 0.0%) i/s (51.50 ms/i) - 98.000 in 5.050382s Comparison: json: 19.7 i/s oj: 31.0 i/s - 1.57x faster rapidjson: 19.4 i/s - 1.02x slower == Encoding many #to_json calls (2661 bytes) oj does not match expected output. Skipping rapidjson unsupported (Invalid object key type: Object) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 2.129k i/100ms Calculating ------------------------------------- json 21.599k (± 0.6%) i/s (46.30 μs/i) - 108.579k in 5.027198s
== Parsing small nested array (121 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 47.497k i/100ms oj 54.115k i/100ms oj strict 53.854k i/100ms Oj::Parser 150.904k i/100ms rapidjson 80.775k i/100ms Calculating ------------------------------------- json 481.096k (± 1.1%) i/s (2.08 μs/i) - 2.422M in 5.035657s oj 554.878k (± 0.6%) i/s (1.80 μs/i) - 2.814M in 5.071521s oj strict 547.888k (± 0.7%) i/s (1.83 μs/i) - 2.747M in 5.013212s Oj::Parser 1.545M (± 0.4%) i/s (647.16 ns/i) - 7.847M in 5.078302s rapidjson 822.422k (± 0.6%) i/s (1.22 μs/i) - 4.120M in 5.009178s Comparison: json: 481096.4 i/s Oj::Parser: 1545223.5 i/s - 3.21x faster rapidjson: 822422.4 i/s - 1.71x faster oj: 554877.7 i/s - 1.15x faster oj strict: 547887.7 i/s - 1.14x faster == Parsing small hash (65 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 154.479k i/100ms oj 220.283k i/100ms oj strict 249.928k i/100ms Oj::Parser 445.062k i/100ms rapidjson 289.615k i/100ms Calculating ------------------------------------- json 1.581M (± 3.0%) i/s (632.55 ns/i) - 8.033M in 5.086476s oj 2.202M (± 3.5%) i/s (454.08 ns/i) - 11.014M in 5.008146s oj strict 2.498M (± 3.5%) i/s (400.25 ns/i) - 12.496M in 5.008245s Oj::Parser 4.640M (± 0.4%) i/s (215.50 ns/i) - 23.588M in 5.083443s rapidjson 3.111M (± 0.3%) i/s (321.44 ns/i) - 15.639M in 5.027097s Comparison: json: 1580898.5 i/s Oj::Parser: 4640298.1 i/s - 2.94x faster rapidjson: 3111005.2 i/s - 1.97x faster oj strict: 2498421.4 i/s - 1.58x faster oj: 2202276.6 i/s - 1.39x faster == Parsing test from oj (256 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 37.580k i/100ms oj 41.899k i/100ms oj strict 50.731k i/100ms Oj::Parser 74.589k i/100ms rapidjson 50.954k i/100ms Calculating ------------------------------------- json 382.150k (± 1.0%) i/s (2.62 μs/i) - 1.917M in 5.015737s oj 420.282k (± 0.2%) i/s (2.38 μs/i) - 2.137M in 5.084338s oj strict 511.758k (± 0.5%) i/s (1.95 μs/i) - 2.587M in 5.055821s Oj::Parser 759.087k (± 0.3%) i/s (1.32 μs/i) - 3.804M in 5.011388s rapidjson 518.273k (± 1.8%) i/s (1.93 μs/i) - 2.599M in 5.015867s Comparison: json: 382149.6 i/s Oj::Parser: 759087.1 i/s - 1.99x faster rapidjson: 518272.8 i/s - 1.36x faster oj strict: 511758.4 i/s - 1.34x faster oj: 420282.5 i/s - 1.10x faster == Parsing twitter.json (567916 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 52.000 i/100ms oj 63.000 i/100ms oj strict 74.000 i/100ms Oj::Parser 79.000 i/100ms rapidjson 56.000 i/100ms Calculating ------------------------------------- json 522.896 (± 0.4%) i/s (1.91 ms/i) - 2.652k in 5.071809s oj 624.849 (± 0.6%) i/s (1.60 ms/i) - 3.150k in 5.041398s oj strict 737.779 (± 0.4%) i/s (1.36 ms/i) - 3.700k in 5.015117s Oj::Parser 789.254 (± 0.3%) i/s (1.27 ms/i) - 3.950k in 5.004764s rapidjson 565.663 (± 0.4%) i/s (1.77 ms/i) - 2.856k in 5.049015s Comparison: json: 522.9 i/s Oj::Parser: 789.3 i/s - 1.51x faster oj strict: 737.8 i/s - 1.41x faster oj: 624.8 i/s - 1.19x faster rapidjson: 565.7 i/s - 1.08x faster == Parsing citm_catalog.json (1727030 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 27.000 i/100ms oj 31.000 i/100ms oj strict 36.000 i/100ms Oj::Parser 42.000 i/100ms rapidjson 38.000 i/100ms Calculating ------------------------------------- json 305.248 (± 0.3%) i/s (3.28 ms/i) - 1.539k in 5.041813s oj 320.265 (± 3.4%) i/s (3.12 ms/i) - 1.612k in 5.039715s oj strict 373.701 (± 1.6%) i/s (2.68 ms/i) - 1.872k in 5.010633s Oj::Parser 457.792 (± 0.4%) i/s (2.18 ms/i) - 2.310k in 5.046049s rapidjson 350.933 (± 8.8%) i/s (2.85 ms/i) - 1.748k in 5.052491s Comparison: json: 305.2 i/s Oj::Parser: 457.8 i/s - 1.50x faster oj strict: 373.7 i/s - 1.22x faster rapidjson: 350.9 i/s - 1.15x faster oj: 320.3 i/s - 1.05x faster == Parsing canada.json (2251051 bytes) ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 2.000 i/100ms oj 2.000 i/100ms oj strict 2.000 i/100ms Oj::Parser 2.000 i/100ms rapidjson 28.000 i/100ms Calculating ------------------------------------- json 29.216 (± 6.8%) i/s (34.23 ms/i) - 146.000 in 5.053753s oj 24.899 (± 0.0%) i/s (40.16 ms/i) - 126.000 in 5.061915s oj strict 24.828 (± 4.0%) i/s (40.28 ms/i) - 124.000 in 5.003067s Oj::Parser 30.867 (± 3.2%) i/s (32.40 ms/i) - 156.000 in 5.057104s rapidjson 285.761 (± 1.0%) i/s (3.50 ms/i) - 1.456k in 5.095715s Comparison: json: 29.2 i/s rapidjson: 285.8 i/s - 9.78x faster Oj::Parser: 30.9 i/s - same-ish: difference falls within error oj: 24.9 i/s - 1.17x slower oj strict: 24.8 i/s - 1.18x slower
There is a large number of outstanding performance PRs that I want to merge, but we need a decent benchmark to judge if they are effective.
I went to borrow rapidjson's benchmark suite, which is a good start.
I only kept the comparison with Oj and RapidJSON, because YAJL is slower on most benchmarks, so little point comparing to it.
FYI @jhawthorn
Encoding
Parsing