Closed eproxus closed 3 years ago
Sound good! I'll try running a benchmark this weekend.
Sorry for the too delayed response.
I updated the benchmark in #61.
The following is the summary.
Input data \ Library | Jason | jiffy | JSON* | jsone | JSX | Poison | Tiny |
---|---|---|---|---|---|---|---|
Blockchain | 2.77 K | 4.55 K | 0.45 K | 1.44 K (3) | 0.60 K | 1.30 K | 0.99 K |
Giphy | 230.65 | 487.67 | 47.73 | 114.57 (4) | 44.97 | 114.57 | 113.59 |
GitHub | 880.03 | 1566.67 | 139.79 | 300.26 (5) | 99.68 | 424.75 | 455.07 |
GovTrack | 6.57 | 24.92 | 2.33 | 5.35 (5) | 2.65 | 7.06 | 7.86 |
Issue 90 | 22.80 | 21.92 | 0.77 | 14.30 (3) | 5.33 | 12.60 | 12.95 |
JSON Generateor | 200.40 | 606.81 | 42.45 | 147.12 (4) | 68.73 | 187.95 | 123.93 |
Pokedex | 209.51 | 776.67 | 62.60 | 161.45 (4) | 69.87 | 190.93 | 125.16 |
UTF-8 unescaped | 626.25 | 6644.53 | 1167.89 | 582.41 (4) | 273.48 | 401.44 | 220.14 |
* Only JSON
didn't escape non-ASCII unicode characters on the encoding
Input data \ Library | Jason | jiffy | JSON | jsone | JSX | Poison | Tiny |
---|---|---|---|---|---|---|---|
Blockchain | 2.75 K | 2.62 K | 0.35 K | 2.21 K (3) | 0.89 K | 1.32 K | 1.49 K |
Giphy | 212.18 | 243.45 | 35.67 | 109.11 (5) | 64.32 | 110.76 | 114.54 |
GitHub | 973.41 | 1052.94 | 137.02 | 662.39 (3) | 271.97 | 438.79 | 542.67 |
GovTrack | 10.77 | 8.32 | 0.80 | 5.08 (3) | 2.81 | 3.58 | 3.65 |
Issue 90 | 17.85 | 41.16 | 0.88 | 10.79 (5) | 6.02 | 13.63 | 14.03 |
JSON Generateor | 320.79 | 243.93 | 25.16 | 184.23 (3) | 111.24 | 135.47 | 139.78 |
JSON Generateor (Pretty) | 273.57 | 205.09 | 25.04 | 158.82 (3) | 97.93 | 123.31 | 136.65 |
Pokedex | 527.63 | 285.43 | 33.70 | 245.36 (3) | 140.90 | 172.45 | 152.59 |
UTF-8 escaped | 1224.48 | 7923.08 | 326.43 | 573.70 (4) | 550.36 | 918.21 | 520.31 |
UTF-8 unescaped | 5.56 K | 12.54 K | 1.35 K | 5.09 K (3) | 3.30 K | 4.39 K | 1.46 K |
Great! Thanks! Would you mind running the same benchmark on the same hardware with Erlang 23 and post the results here? It would be interesting to compare them.
Would you mind running the same benchmark on the same hardware with Erlang 23 and post the results here?
Sound goods! I'll do that when I have time.
Added a comparison between OTP-24 w/ JIT and OTP-24 w/o JIT on the same machine.
NOTE:
Tiny
are removed from the tables belowJason (w/o JIT) | jiffy | JSON | jsone | JSX | Poison | |
---|---|---|---|---|---|---|
Blockchain | 2.44 K (1.45 K) | 6.46 K (6.78 K) | 0.54 K (0.47 K) | 1.25 K (0.88 K) | 0.59 K (0.57 K) | 1.35 K (1.14 K) |
Giphy | 180.38 (144.91) | 705.90 (715.32) | 54.30 (49.26) | 120.16 (109.21) | 44.47 (41.73) | 137.99 (116.60) |
GitHub | 0.80 K (0.50 K) | 2.13 K (2.11 K) | 0.15 K (0.13 K) | 0.38 K (0.31 K) | 0.09 K (0.01 K) | 0.44 K (0.33 K) |
GovTrack | 8.34 (5.15) | 27.52 (29.98) | 3.04 (2.18) | 5.66 (4.73) | 2.12 (2.18) | 8.51 (5.45) |
Issue 90 | 19.31 (7.30) | 23.97 (23.50) | 0.82 (0.82) | 12.42 (7.09) | 4.58 (5.10) | 12.52 (3.99) |
JSON Gen | 198.29 (153.43) | 738.16 (711.54) | 42.61 (48.47) | 163.48 (130.89) | 65.54 (52.48) | 182.33 (123.18) |
Pokedex | 0.20 K (0.16 K) | 1.02 K (0.86 K) | 0.06 K (0.07 K) | 0.19 K (0.15 K) | 0.06 K (0.05 K) | 0.18 K (0.13 K) |
UTF-8 unes | 0.69 K (0.68 K) | 6.48 K (6.49 K) | 1.24 K (1.16 K) | 0.81 K (0.55 K) | 0.28 K (0.21 K) | 0.41 K (0.35 K) |
Jason (w/o JIT) | jiffy | JSON | jsone | JSX | Poison | |
---|---|---|---|---|---|---|
Blockchai | 3.62 K (3.17 K) | 3.26 K (3.17 K) | 0.48 K (0.31 K) | 2.22 K (1.44 K) | 1.22 K (0.87 K) | 1.58 K (0.98 K) |
Giphy | 272.91 (190.01) | 323.06 (313.38) | 43.37 (36.32) | 126.36 (89.69) | 98.52 (80.79) | 111.63 (90.92) |
GitHub | 1.16 K (0.65 K) | 1.15 K (1.00 K) | 0.16 K (0.12 K) | 0.72 K (0.42 K) | 0.29 K (0.24 K) | 0.49 K (0.33 K) |
GovTrack | 11.73 (6.99) | 8.71 (8.64) | 1.01 (0.84) | 5.21 (4.10) | 2.62 (2.42) | 3.50 (2.74) |
Issue 90 | 22.86 (7.36) | 28.92 (29.13) | 1.19 (0.81) | 11.57 (5.90) | 6.08 (5.59) | 14.80 (5.76) |
JSON G | 426.36 (252.23) | 308.76 (295.74) | 36.10 (24.16) | 226.20 (166.45) | 115.25 (92.19) | 142.91 (103.69) |
JSON G (P) | 383.44 (227.40) | 245.91 (240.35) | 37.83 (28.44) | 216.12 (154.64) | 111.01 (74.18) | 133.09 (101.27) |
Pokedex | 677.03 (434.25) | 356.64 (324.02) | 55.46 (38.38) | 289.15 (244.69) | 153.85 (120.73) | 184.15 (139.49) |
UTF-8 esc | 1.23 K (0.86 K) | 6.70 K (6.82 K) | 0.54 K (0.40 K) | 0.66 K (0.54 K) | 0.77 K (0.54 K) | 1.20 K (0.89 K) |
UTF-8 une | 7.94 K (4.65 K) | 10.8 K (10.4 K) | 2.40 K (1.80 K) | 5.01 K (2.88 K) | 4.45 K (2.72 K) | 6.26 K (3.47 K) |
@sile Thanks for doing this. Wow, some benchmarks are almost 2x faster. (And interesting enough, some jiffy benchmarks are slower [Issue 90])
Any idea how Jason can be faster than jiffy (which is a C driver)?
@eproxus There is overhead calling NIF functions making or deconstructing erlang terms and necessary checks which can't be omitted in NIFs and could and are omitted inside BEAM instructions. Inside BEAM, there are erlang term handling functions and even macros that allow the compiler to make some optimizations that are not possible with NIF counterparts, especially due to dynamic linking boundaries.
Would love to see an updated benchmark with the new JIT in OTP 24!