search

sile / jsone

Erlang JSON library
MIT License
291 stars 71 forks source link

Updated OTP 24 JIT benchmark? #57

Closed eproxus closed 3 years ago

eproxus commented 3 years ago

Would love to see an updated benchmark with the new JIT in OTP 24!

sile commented 3 years ago

Sound good! I'll try running a benchmark this weekend.

sile commented 3 years ago

Sorry for the too delayed response.

I updated the benchmark in #61.

The following is the summary.

Encoding (Unit: IPS=inputs per second)

Input data \ Library Jason jiffy JSON* jsone JSX Poison Tiny
Blockchain 2.77 K 4.55 K 0.45 K 1.44 K (3) 0.60 K 1.30 K 0.99 K
Giphy 230.65 487.67 47.73 114.57 (4) 44.97 114.57 113.59
GitHub 880.03 1566.67 139.79 300.26 (5) 99.68 424.75 455.07
GovTrack 6.57 24.92 2.33 5.35 (5) 2.65 7.06 7.86
Issue 90 22.80 21.92 0.77 14.30 (3) 5.33 12.60 12.95
JSON Generateor 200.40 606.81 42.45 147.12 (4) 68.73 187.95 123.93
Pokedex 209.51 776.67 62.60 161.45 (4) 69.87 190.93 125.16
UTF-8 unescaped 626.25 6644.53 1167.89 582.41 (4) 273.48 401.44 220.14

* Only JSON didn't escape non-ASCII unicode characters on the encoding

Decoding (Unit: IPS=inputs per second)

Input data \ Library Jason jiffy JSON jsone JSX Poison Tiny
Blockchain 2.75 K 2.62 K 0.35 K 2.21 K (3) 0.89 K 1.32 K 1.49 K
Giphy 212.18 243.45 35.67 109.11 (5) 64.32 110.76 114.54
GitHub 973.41 1052.94 137.02 662.39 (3) 271.97 438.79 542.67
GovTrack 10.77 8.32 0.80 5.08 (3) 2.81 3.58 3.65
Issue 90 17.85 41.16 0.88 10.79 (5) 6.02 13.63 14.03
JSON Generateor 320.79 243.93 25.16 184.23 (3) 111.24 135.47 139.78
JSON Generateor (Pretty) 273.57 205.09 25.04 158.82 (3) 97.93 123.31 136.65
Pokedex 527.63 285.43 33.70 245.36 (3) 140.90 172.45 152.59
UTF-8 escaped 1224.48 7923.08 326.43 573.70 (4) 550.36 918.21 520.31
UTF-8 unescaped 5.56 K 12.54 K 1.35 K 5.09 K (3) 3.30 K 4.39 K 1.46 K
eproxus commented 3 years ago

Great! Thanks! Would you mind running the same benchmark on the same hardware with Erlang 23 and post the results here? It would be interesting to compare them.

sile commented 3 years ago

Would you mind running the same benchmark on the same hardware with Erlang 23 and post the results here?

Sound goods! I'll do that when I have time.

sile commented 3 years ago

Added a comparison between OTP-24 w/ JIT and OTP-24 w/o JIT on the same machine.

NOTE:

Encoding

Jason (w/o JIT) jiffy JSON jsone JSX Poison
Blockchain 2.44 K (1.45 K) 6.46 K (6.78 K) 0.54 K (0.47 K) 1.25 K (0.88 K) 0.59 K (0.57 K) 1.35 K (1.14 K)
Giphy 180.38 (144.91) 705.90 (715.32) 54.30 (49.26) 120.16 (109.21) 44.47 (41.73) 137.99 (116.60)
GitHub 0.80 K (0.50 K) 2.13 K (2.11 K) 0.15 K (0.13 K) 0.38 K (0.31 K) 0.09 K (0.01 K) 0.44 K (0.33 K)
GovTrack 8.34 (5.15) 27.52 (29.98) 3.04 (2.18) 5.66 (4.73) 2.12 (2.18) 8.51 (5.45)
Issue 90 19.31 (7.30) 23.97 (23.50) 0.82 (0.82) 12.42 (7.09) 4.58 (5.10) 12.52 (3.99)
JSON Gen 198.29 (153.43) 738.16 (711.54) 42.61 (48.47) 163.48 (130.89) 65.54 (52.48) 182.33 (123.18)
Pokedex 0.20 K (0.16 K) 1.02 K (0.86 K) 0.06 K (0.07 K) 0.19 K (0.15 K) 0.06 K (0.05 K) 0.18 K (0.13 K)
UTF-8 unes 0.69 K (0.68 K) 6.48 K (6.49 K) 1.24 K (1.16 K) 0.81 K (0.55 K) 0.28 K (0.21 K) 0.41 K (0.35 K)

Decoding

Jason (w/o JIT) jiffy JSON jsone JSX Poison
Blockchai 3.62 K (3.17 K) 3.26 K (3.17 K) 0.48 K (0.31 K) 2.22 K (1.44 K) 1.22 K (0.87 K) 1.58 K (0.98 K)
Giphy 272.91 (190.01) 323.06 (313.38) 43.37 (36.32) 126.36 (89.69) 98.52 (80.79) 111.63 (90.92)
GitHub 1.16 K (0.65 K) 1.15 K (1.00 K) 0.16 K (0.12 K) 0.72 K (0.42 K) 0.29 K (0.24 K) 0.49 K (0.33 K)
GovTrack 11.73 (6.99) 8.71 (8.64) 1.01 (0.84) 5.21 (4.10) 2.62 (2.42) 3.50 (2.74)
Issue 90 22.86 (7.36) 28.92 (29.13) 1.19 (0.81) 11.57 (5.90) 6.08 (5.59) 14.80 (5.76)
JSON G 426.36 (252.23) 308.76 (295.74) 36.10 (24.16) 226.20 (166.45) 115.25 (92.19) 142.91 (103.69)
JSON G (P) 383.44 (227.40) 245.91 (240.35) 37.83 (28.44) 216.12 (154.64) 111.01 (74.18) 133.09 (101.27)
Pokedex 677.03 (434.25) 356.64 (324.02) 55.46 (38.38) 289.15 (244.69) 153.85 (120.73) 184.15 (139.49)
UTF-8 esc 1.23 K (0.86 K) 6.70 K (6.82 K) 0.54 K (0.40 K) 0.66 K (0.54 K) 0.77 K (0.54 K) 1.20 K (0.89 K)
UTF-8 une 7.94 K (4.65 K) 10.8 K (10.4 K) 2.40 K (1.80 K) 5.01 K (2.88 K) 4.45 K (2.72 K) 6.26 K (3.47 K)
eproxus commented 3 years ago

@sile Thanks for doing this. Wow, some benchmarks are almost 2x faster. (And interesting enough, some jiffy benchmarks are slower [Issue 90])

eproxus commented 3 years ago

Any idea how Jason can be faster than jiffy (which is a C driver)?

pichi commented 3 years ago

@eproxus There is overhead calling NIF functions making or deconstructing erlang terms and necessary checks which can't be omitted in NIFs and could and are omitted inside BEAM instructions. Inside BEAM, there are erlang term handling functions and even macros that allow the compiler to make some optimizations that are not possible with NIF counterparts, especially due to dynamic linking boundaries.