The benchmarks follow the criteria:
They are written as the average software developer would write them, i.e.
The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.
My other benchmarks: jit-benchmarks, crystal-benchmarks-game
The measured values are:
base
+ increase
, where base
is the RSS before the benchmark and increase
is the peak increase of the RSS during the benchmark;All values are presented as: median
±median absolute deviation
.
UPDATE: 2024-06-06
Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
Scala (Staged) | 0.390±0.015 | 222.77±03.17 + 15.42±01.67 | 24.89±00.74 |
Racket (Staged) | 0.893±0.001 | 101.38±00.97 + 0.00±00.00 | 34.61±00.07 |
Rust | 1.011±0.000 | 1.05±00.01 + 0.00±00.00 | 43.04±00.28 |
V/gcc | 1.060±0.000 | 1.92±00.03 + 0.00±00.00 | 43.67±00.15 |
C/gcc | 1.111±0.001 | 1.01±00.01 + 0.00±00.00 | 47.01±00.65 |
C++/g++ | 1.113±0.001 | 1.98±00.04 + 0.00±00.00 | 46.04±00.34 |
D/gdc | 1.120±0.000 | 6.39±00.05 + 0.00±00.00 | 49.26±00.57 |
C++/clang++ | 1.125±0.001 | 1.74±00.02 + 0.00±00.00 | 46.90±00.51 |
C/clang | 1.140±0.001 | 1.02±00.01 + 0.00±00.00 | 48.57±00.37 |
D/ldc2 | 1.163±0.003 | 2.32±00.80 + 0.00±00.00 | 49.08±00.30 |
Nim/gcc | 1.168±0.001 | 1.01±00.02 + 0.00±00.00 | 49.66±00.25 |
Java | 1.193±0.000 | 39.79±00.09 + 1.55±00.06 | 49.18±00.26 |
Vala/gcc | 1.210±0.001 | 4.50±00.02 + 0.00±00.00 | 50.69±00.11 |
Kotlin/JVM | 1.222±0.002 | 43.53±00.14 + 0.52±00.26 | 50.93±00.10 |
Vala/clang | 1.238±0.006 | 4.52±00.03 + 0.00±00.00 | 52.25±01.41 |
Go | 1.247±0.000 | 3.55±00.03 + 0.00±00.00 | 51.47±00.33 |
Zig | 1.265±0.000 | 1.03±00.02 + 0.00±00.00 | 52.54±00.44 |
C#/.NET Core | 1.367±0.001 | 32.66±00.13 + 0.15±00.02 | 58.23±00.27 |
Go/gccgo | 1.481±0.002 | 23.56±00.01 + 0.00±00.00 | 62.86±00.67 |
Nim/clang | 1.565±0.000 | 1.29±00.01 + 0.00±00.00 | 64.41±00.46 |
F#/.NET Core | 1.574±0.004 | 37.41±00.18 + 0.30±00.00 | 67.12±00.40 |
Crystal | 1.597±0.008 | 3.00±00.05 + 0.00±00.00 | 67.57±00.71 |
OCaml | 1.638±0.005 | 3.01±00.05 + 2.81±00.06 | 76.01±00.28 |
Julia | 1.638±0.003 | 256.65±00.11 + 0.40±00.03 | 69.57±00.34 |
Chez Scheme | 1.714±0.004 | 24.77±00.02 + 4.46±00.12 | 72.48±00.24 |
Racket | 1.756±0.028 | 113.87±00.09 + 0.00±00.00 | 71.30±01.18 |
V/clang | 1.973±0.018 | 1.96±00.03 + 0.00±00.00 | 85.72±01.29 |
C#/Mono | 2.054±0.011 | 25.60±00.08 + 0.00±00.00 | 87.72±01.03 |
MLton | 2.088±0.027 | 1.77±00.04 + 0.25±00.00 | 86.67±01.18 |
Scala | 2.779±0.003 | 72.03±00.11 + 249.04±00.15 | 121.64±00.71 |
Node.js | 2.920±0.028 | 44.53±00.02 + 4.37±00.00 | 123.76±01.03 |
Haskell (MArray) | 3.095±0.007 | 3.17±00.04 + 4.92±00.04 | 130.42±01.60 |
D/dmd | 3.326±0.001 | 3.39±00.03 + 0.00±00.00 | 124.72±00.18 |
Haskell (FP) | 3.624±0.002 | 3.27±00.05 + 4.89±00.04 | 154.06±00.80 |
Ruby/truffleruby (JVM) | 4.996±0.396 | 374.17±10.09 + 496.92±17.09 | 243.84±22.97 |
Ruby/truffleruby | 5.193±0.358 | 203.03±01.33 + 550.75±59.29 | 253.87±16.16 |
Swift | 5.656±0.005 | 16.69±00.04 + 0.00±00.00 | 211.92±00.58 |
Lua/luajit | 5.881±0.025 | 2.54±00.02 + 0.00±00.00 | 241.87±05.05 |
Python/pypy | 9.544±0.044 | 58.79±00.12 + 29.96±00.01 | 422.77±04.07 |
Idris | 15.129±0.051 | 20.71±00.03 + 8.83±00.04 | 661.53±10.66 |
Elixir | 20.395±0.043 | 70.50±00.79 + 0.00±00.00 | 807.25±05.02 |
Ruby (--jit) | 31.817±0.055 | 21.59±00.04 + 4.76±00.01 | 1311.22±06.23 |
PHP | 34.492±0.160 | 17.78±00.18 + 0.00±00.00 | 1476.88±31.23 |
Lua | 36.833±0.101 | 2.28±00.03 + 0.00±00.00 | 1512.87±19.03 |
Ruby | 68.430±0.445 | 11.43±00.04 + 0.00±00.00 | 3008.91±26.07 |
Python | 69.075±0.780 | 11.17±00.08 + 0.00±00.00 | 3039.83±50.68 |
Ruby/jruby | 81.684±1.546 | 203.50±04.69 + 220.84±20.47 | 3624.22±70.81 |
Tcl (FP) | 192.447±0.637 | 4.06±00.02 + 0.00±00.00 | 8478.25±83.42 |
Perl | 225.331±1.241 | 6.90±00.05 + 0.00±00.00 | 10000.48±94.38 |
Tcl (OOP) | 384.789±1.841 | 4.06±00.01 + 0.00±00.00 | 17091.07±181.64 |
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
Scala (Staged) | 7.639±0.120 | 224.42±01.90 + 102.36±04.76 | 459.89±14.99 |
C++/g++ | 9.755±0.030 | 1.97±00.03 + 2.29±00.11 | 393.99±02.68 |
C#/.NET Core | 12.048±0.028 | 32.72±00.06 + 1.27±00.00 | 482.24±01.22 |
Java | 12.450±0.060 | 39.92±00.06 + 2.44±00.04 | 489.83±01.92 |
C/gcc | 12.513±0.013 | 0.99±00.02 + 0.84±00.06 | 505.46±02.16 |
Kotlin/JVM | 13.082±0.033 | 43.53±00.17 + 2.30±00.33 | 547.94±03.79 |
F#/.NET Core | 13.115±0.017 | 37.55±00.07 + 2.12±00.04 | 526.18±01.55 |
C/clang | 13.293±0.031 | 0.97±00.03 + 0.75±00.03 | 572.78±05.53 |
V/gcc | 13.591±0.007 | 1.91±00.03 + 1.18±00.04 | 547.49±03.65 |
C++/clang++ | 13.898±0.024 | 1.72±00.05 + 1.88±00.03 | 569.56±02.36 |
Racket (Staged) | 13.986±0.059 | 99.66±00.50 + 77.13±01.67 | 556.40±01.66 |
Crystal | 14.103±0.018 | 2.96±00.03 + 0.74±00.04 | 598.53±02.62 |
Go | 14.182±0.009 | 3.56±00.01 + 0.00±00.00 | 562.34±00.49 |
Rust | 14.229±0.012 | 1.04±00.01 + 1.20±00.04 | 564.72±02.02 |
D/gdc | 14.266±0.016 | 6.37±00.06 + 1.49±00.04 | 601.20±03.61 |
D/ldc2 | 14.278±0.013 | 3.05±00.02 + 0.83±00.02 | 572.27±01.12 |
Vala/gcc | 14.469±0.022 | 4.47±00.04 + 1.24±00.04 | 574.39±01.98 |
Zig | 14.481±0.014 | 1.02±00.02 + 1.42±00.06 | 600.92±02.27 |
Vala/clang | 14.845±0.010 | 4.46±00.02 + 1.22±00.03 | 607.68±02.95 |
Nim/gcc | 15.641±0.030 | 2.07±00.06 + 1.29±00.00 | 663.67±00.73 |
Scala | 16.365±0.031 | 71.96±00.25 + 139.77±00.37 | 735.09±03.15 |
Swift | 18.421±0.037 | 16.45±00.04 + 0.00±00.00 | 762.02±06.09 |
Go/gccgo | 19.091±0.406 | 23.58±00.05 + 0.00±00.00 | 796.35±13.26 |
Nim/clang | 19.696±0.204 | 2.38±00.01 + 1.29±00.00 | 806.37±09.58 |
V/clang | 20.895±0.225 | 1.94±00.05 + 1.17±00.00 | 901.27±12.94 |
OCaml | 24.390±0.016 | 4.53±00.08 + 2.82±00.00 | 1171.79±06.03 |
Julia | 26.058±0.070 | 256.43±00.04 + 0.30±00.00 | 1021.32±06.88 |
Chez Scheme | 27.698±0.056 | 25.55±00.06 + 3.68±00.01 | 1210.07±04.32 |
C#/Mono | 31.025±0.011 | 25.61±00.09 + 0.83±00.00 | 1289.92±08.22 |
MLton | 33.842±0.072 | 1.73±00.05 + 4.11±00.00 | 1537.44±17.63 |
Haskell (MArray) | 34.546±0.022 | 4.08±00.02 + 5.11±00.00 | 1400.92±05.61 |
Node.js | 34.598±0.063 | 44.44±00.07 + 5.25±00.00 | 1378.99±02.63 |
Lua/luajit | 34.656±0.034 | 2.56±00.04 + 0.38±00.00 | 1388.13±02.21 |
Racket | 35.183±0.294 | 113.90±00.09 + 1.45±00.65 | 1567.04±13.20 |
D/dmd | 37.942±0.003 | 3.35±00.03 + 0.87±00.02 | 1378.02±00.92 |
Python/pypy | 41.463±0.088 | 58.86±00.09 + 30.60±00.03 | 1837.98±07.63 |
Ruby/truffleruby | 48.320±0.851 | 203.25±00.90 + 581.44±30.40 | 2360.50±62.00 |
Ruby/truffleruby (JVM) | 50.546±1.177 | 374.73±03.49 + 489.39±89.50 | 2202.33±46.95 |
Idris | 66.241±0.299 | 22.03±00.05 + 9.54±00.00 | 2848.93±05.21 |
Haskell (FP) | 77.093±0.126 | 4.19±00.02 + 75.74±00.03 | 3195.67±11.41 |
Testing base64 encoding/decoding of the large blob into the newly allocated buffers.
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
C/gcc (aklomp) | 0.098±0.000 | 2.20±00.03 + 0.00±00.00 | 4.60±00.03 |
C/clang (aklomp) | 0.099±0.000 | 2.10±00.06 + 0.00±00.00 | 4.68±00.05 |
PHP | 0.105±0.000 | 18.56±00.06 + 0.00±00.00 | 4.76±00.07 |
Go (base64x) | 0.267±0.001 | 6.61±00.02 + 0.00±00.00 | 12.80±00.11 |
Zig | 0.698±0.000 | 1.66±00.04 + 0.00±00.00 | 26.49±00.19 |
Rust | 0.849±0.000 | 2.42±00.02 + 0.00±00.00 | 35.03±00.11 |
Node.js | 0.913±0.001 | 43.17±00.03 + 40.43±00.21 | 40.38±00.37 |
C/clang | 0.997±0.000 | 2.15±00.01 + 0.00±00.00 | 36.62±00.05 |
C/gcc | 1.101±0.016 | 2.10±00.05 + 0.00±00.00 | 40.24±00.60 |
Nim/clang | 1.103±0.001 | 2.06±00.02 + 5.79±00.03 | 44.95±00.27 |
Crystal | 1.107±0.003 | 3.61±00.04 + 1.26±00.03 | 44.93±00.29 |
D/ldc2 | 1.175±0.003 | 3.68±00.03 + 3.41±00.00 | 48.49±00.44 |
Nim/gcc | 1.364±0.002 | 1.75±00.04 + 4.96±00.01 | 55.70±00.32 |
Ruby (--jit) | 1.427±0.002 | 15.10±00.16 + 73.37±00.56 | 55.92±00.37 |
Java | 1.518±0.005 | 41.05±00.10 + 210.65±19.06 | 60.66±00.44 |
V/clang | 1.534±0.001 | 2.43±00.01 + 2386.75±01.65 | 57.81±00.29 |
V/gcc | 1.575±0.000 | 2.43±00.03 + 2385.17±00.74 | 57.34±00.19 |
Scala | 1.604±0.001 | 70.48±00.15 + 276.28±02.12 | 66.81±00.54 |
Go | 1.622±0.002 | 4.34±00.02 + 0.00±00.00 | 67.66±00.22 |
Ruby | 1.642±0.005 | 11.69±00.04 + 42.50±00.30 | 64.29±00.22 |
Vala/clang | 1.643±0.001 | 5.68±00.03 + 0.10±00.03 | 63.09±00.70 |
Vala/gcc | 1.644±0.001 | 5.66±00.01 + 0.08±00.05 | 62.92±00.40 |
Kotlin/JVM | 1.665±0.003 | 44.48±00.20 + 314.13±03.17 | 67.36±00.44 |
C++/g++ (libcrypto) | 1.711±0.003 | 6.05±00.16 + 0.76±00.03 | 68.53±00.38 |
C++/clang++ (libcrypto) | 1.713±0.002 | 5.51±00.04 + 0.73±00.02 | 69.23±00.37 |
Perl (MIME::Base64) | 1.863±0.044 | 14.57±00.04 + 0.12±00.03 | 75.20±01.07 |
F#/.NET Core | 2.038±0.018 | 38.32±00.07 + 11.92±00.65 | 80.42±00.77 |
C#/.NET Core | 2.181±0.021 | 33.77±00.03 + 12.54±02.09 | 85.47±00.28 |
D/gdc | 2.382±0.001 | 7.44±00.04 + 3.36±00.00 | 103.50±00.94 |
Go/gccgo | 2.949±0.007 | 24.81±00.17 + 0.00±00.00 | 139.67±00.50 |
Julia | 3.079±0.006 | 271.96±00.08 + 97.24±00.20 | 124.20±01.58 |
Python/pypy | 3.236±0.002 | 58.92±00.06 + 31.49±00.17 | 141.88±00.91 |
D/dmd | 3.387±0.004 | 3.17±00.02 + 3.84±00.02 | 141.65±00.88 |
Tcl | 3.438±0.001 | 5.20±00.02 + 0.00±00.00 | 137.53±00.49 |
Python | 3.512±0.001 | 10.78±00.16 + 0.07±00.07 | 133.94±00.44 |
Ruby/truffleruby (JVM) | 3.758±0.016 | 374.30±05.93 + 247.06±24.79 | 189.38±03.07 |
Racket | 3.885±0.006 | 95.36±00.20 + 21.80±00.40 | 154.27±00.56 |
C#/Mono | 4.632±0.005 | 26.39±00.04 + 18.66±00.02 | 190.60±01.36 |
Ruby/jruby | 6.005±0.008 | 195.03±02.11 + 165.97±10.93 | 263.57±01.49 |
Ruby/truffleruby | 8.324±0.018 | 201.01±02.82 + 554.96±13.20 | 400.06±01.94 |
Perl (MIME::Base64::Perl) | 10.148±0.055 | 15.93±00.11 + 0.26±00.04 | 448.06±03.65 |
Testing parsing and simple calculating of values from a big JSON file.
Few notes:
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
C++/clang++ (simdjson On-Demand) | 0.060±0.000 | 112.46±00.06 + 60.05±00.06 | 2.51±00.01 |
C++/g++ (simdjson On-Demand) | 0.061±0.000 | 113.66±00.03 + 59.81±00.00 | 2.57±00.02 |
C++/clang++ (DAW JSON Link NoCheck) | 0.083±0.000 | 112.54±00.05 + 0.00±00.00 | 3.37±00.03 |
C++/g++ (DAW JSON Link NoCheck) | 0.087±0.000 | 113.29±00.06 + 0.00±00.00 | 3.53±00.04 |
C++/clang++ (DAW JSON Link) | 0.093±0.000 | 112.47±00.03 + 0.00±00.00 | 3.90±00.07 |
C++/g++ (DAW JSON Link) | 0.093±0.000 | 113.30±00.05 + 0.00±00.00 | 3.88±00.03 |
Rust (Serde Custom) | 0.098±0.001 | 111.40±00.08 + 0.00±00.00 | 4.14±00.04 |
C++/clang++ (simdjson DOM) | 0.102±0.000 | 112.55±00.09 + 177.09±00.06 | 4.62±00.02 |
C++/g++ (simdjson DOM) | 0.108±0.000 | 113.54±00.06 + 172.73±00.13 | 4.90±00.02 |
Rust (Serde Typed) | 0.111±0.000 | 111.71±00.05 + 11.05±00.07 | 4.62±00.04 |
D/ldc2 (Mir Asdf DOM) | 0.131±0.000 | 112.93±00.04 + 61.22±00.00 | 5.43±00.05 |
C++/clang++ (gason) | 0.139±0.000 | 112.57±00.02 + 96.97±00.06 | 5.67±00.04 |
C++/g++ (gason) | 0.140±0.000 | 113.17±00.06 + 96.93±00.04 | 5.55±00.03 |
C++/g++ (RapidJSON) | 0.152±0.000 | 113.21±00.04 + 128.90±00.03 | 6.47±00.04 |
Scala (jsoniter-scala) | 0.157±0.002 | 290.75±00.24 + 20.43±00.24 | 8.36±00.07 |
Go (rjson custom) | 0.196±0.000 | 113.58±00.05 + 0.00±00.00 | 7.57±00.02 |
C++/clang++ (RapidJSON) | 0.200±0.000 | 112.47±00.04 + 128.91±00.09 | 8.55±00.08 |
Go (Sonic) | 0.208±0.002 | 122.51±00.12 + 0.00±00.00 | 8.94±00.09 |
C++/g++ (RapidJSON Precise) | 0.217±0.000 | 113.21±00.06 + 122.47±01.29 | 9.28±00.08 |
D/ldc2 (Mir Amazon's Ion DOM) | 0.219±0.000 | 113.03±00.08 + 80.70±00.00 | 9.26±00.03 |
Go (rjson) | 0.224±0.000 | 113.61±00.10 + 0.00±00.00 | 8.70±00.02 |
Zig | 0.241±0.000 | 111.03±00.02 + 39.25±00.00 | 10.37±00.07 |
Go (goccy/go-json) | 0.266±0.000 | 114.29±00.07 + 0.00±00.00 | 10.56±00.02 |
C++/clang++ (RapidJSON Precise) | 0.285±0.000 | 112.48±00.03 + 128.91±00.03 | 12.50±00.10 |
C++/g++ (RapidJSON SAX) | 0.346±0.001 | 113.11±00.08 + 0.00±00.00 | 15.32±00.06 |
C/clang (yajl) | 0.350±0.001 | 110.97±00.04 + 0.00±00.00 | 15.14±00.06 |
C/gcc (yajl) | 0.353±0.001 | 110.95±00.01 + 0.00±00.00 | 15.22±00.06 |
C++/g++ (Boost.JSON) | 0.363±0.000 | 113.32±00.04 + 308.09±00.02 | 15.44±00.10 |
C++/clang++ (Boost.JSON) | 0.371±0.001 | 112.68±00.03 + 308.09±00.03 | 15.74±00.10 |
Nim/clang (jsony) | 0.392±0.000 | 111.56±00.05 + 146.12±00.09 | 16.46±00.12 |
C++/g++ (RapidJSON SAX Precise) | 0.401±0.001 | 113.12±00.06 + 0.00±00.00 | 17.92±00.23 |
C++/clang++ (RapidJSON SAX) | 0.408±0.001 | 194.77±00.07 + 0.00±00.00 | 17.40±00.07 |
Nim/gcc (jsony) | 0.412±0.000 | 111.23±00.03 + 154.69±00.03 | 17.46±00.15 |
Node.js | 0.470±0.004 | 152.69±00.04 + 195.96±00.68 | 22.05±00.12 |
C++/clang++ (RapidJSON SAX Precise) | 0.489±0.001 | 194.79±00.07 + 0.00±00.00 | 21.84±00.23 |
Go (jsoniter) | 0.513±0.000 | 114.27±00.06 + 0.00±00.00 | 20.75±00.17 |
Rust (Serde Untyped) | 0.532±0.001 | 111.73±00.03 + 839.98±00.00 | 22.14±00.20 |
C#/.NET Core (System.Text.Json) | 0.543±0.002 | 489.53±00.11 + 140.62±00.22 | 24.07±00.23 |
Java (DSL-JSON) | 0.568±0.007 | 262.39±00.11 + 225.35±19.01 | 29.30±00.32 |
Python/pypy | 0.608±0.002 | 279.98±00.08 + 125.71±00.04 | 26.46±00.26 |
V/gcc | 0.615±0.001 | 111.47±00.04 + 496.21±00.00 | 25.70±00.23 |
V/clang | 0.637±0.001 | 111.87±00.01 + 495.83±00.00 | 26.74±00.19 |
Nim/gcc (Packedjson) | 0.638±0.003 | 111.93±00.02 + 294.16±00.00 | 27.28±00.40 |
Crystal (Pull) | 0.638±0.002 | 113.26±00.02 + 18.44±00.00 | 28.10±00.40 |
Crystal (Schema) | 0.656±0.001 | 113.23±00.03 + 48.86±00.13 | 28.63±00.07 |
Nim/clang (Packedjson) | 0.662±0.001 | 112.23±00.01 + 294.16±00.00 | 28.31±00.17 |
Perl (Cpanel::JSON::XS) | 0.765±0.003 | 125.17±00.05 + 402.77±00.00 | 31.84±00.14 |
Go | 0.813±0.001 | 113.95±00.06 + 0.00±00.00 | 33.60±00.06 |
PHP | 0.813±0.001 | 128.26±00.08 + 517.83±00.03 | 34.57±00.20 |
Crystal | 0.924±0.004 | 113.21±00.02 + 392.50±00.03 | 39.68±00.33 |
Nim/gcc | 1.037±0.002 | 111.91±00.03 + 1001.34±00.00 | 43.13±00.12 |
C#/.NET Core | 1.056±0.004 | 495.03±00.30 + 272.99±00.16 | 50.49±00.23 |
Nim/clang | 1.076±0.002 | 112.23±00.00 + 999.02±00.00 | 44.69±00.27 |
C++/g++ (json-c) | 1.159±0.003 | 113.42±00.05 + 1216.05±00.01 | 47.99±00.30 |
C++/clang++ (json-c) | 1.160±0.007 | 112.70±00.07 + 1216.04±00.01 | 48.35±00.66 |
Clojure | 1.161±0.019 | 452.13±02.50 + 587.46±09.25 | 61.05±00.99 |
C++/clang++ (Nlohmann) | 1.176±0.001 | 112.64±00.03 + 360.11±00.02 | 50.39±00.68 |
Ruby (--jit) | 1.280±0.005 | 127.59±00.08 + 212.39±00.01 | 53.84±00.51 |
Go/gccgo | 1.290±0.004 | 138.88±00.10 + 0.00±00.00 | 53.03±00.36 |
C++/g++ (Nlohmann) | 1.327±0.003 | 113.27±00.06 + 448.05±00.01 | 56.39±00.35 |
Ruby | 1.328±0.006 | 121.40±00.05 + 212.77±00.02 | 55.73±00.73 |
CPython (UltraJSON) | 1.331±0.004 | 123.14±00.05 + 497.28±01.55 | 51.19±00.22 |
Python | 1.374±0.002 | 120.99±00.03 + 325.95±00.01 | 55.23±00.37 |
F#/.NET Core (System.Text.Json) | 1.496±0.003 | 498.18±00.08 + 232.75±04.63 | 68.50±00.69 |
Ruby (YAJL) | 1.719±0.009 | 121.49±00.11 + 218.39±00.03 | 72.78±00.52 |
D/ldc2 | 1.723±0.003 | 112.86±00.06 + 708.76±00.04 | 72.08±00.65 |
C#/Mono | 1.809±0.013 | 252.32±00.17 + 31.51±00.01 | 78.45±00.96 |
Haskell | 1.982±0.007 | 115.63±00.14 + 723.73±00.48 | 84.53±00.77 |
Rust (jq) | 2.529±0.003 | 113.39±00.06 + 902.98±01.54 | 106.08±00.37 |
C++/g++ (Boost.PropertyTree) | 2.611±0.008 | 113.14±00.04 + 1440.09±00.04 | 111.72±00.36 |
C++/clang++ (Boost.PropertyTree) | 2.671±0.005 | 195.01±00.04 + 1232.80±00.03 | 112.86±00.70 |
Ruby/jruby | 2.884±0.022 | 464.98±06.74 + 972.45±49.49 | 148.64±02.99 |
D/dmd | 3.073±0.003 | 113.16±00.04 + 708.78±00.07 | 131.07±00.23 |
Vala/clang | 3.165±0.007 | 115.04±00.04 + 980.08±00.03 | 137.65±00.31 |
Vala/gcc | 3.166±0.011 | 115.06±00.03 + 980.07±00.03 | 136.93±00.87 |
D/gdc | 3.569±0.004 | 116.68±00.06 + 680.99±00.12 | 151.81±01.75 |
Racket | 3.818±0.022 | 222.34±00.45 + 261.38±27.93 | 159.23±01.29 |
Perl (JSON::Tiny) | 9.050±0.121 | 125.59±00.05 + 528.92±00.02 | 394.96±03.88 |
Ruby/truffleruby | 10.718±0.048 | 522.13±27.57 + 1942.99±149.89 | 617.74±02.74 |
Ruby/truffleruby (JVM) | 11.011±0.153 | 474.89±08.90 + 2201.50±87.97 | 693.97±13.20 |
Testing allocating and multiplying matrices.
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
D/ldc2 (lubeck) | 0.032±0.001 | 40.38±00.48 + 23.61±00.41 | 3.61±00.09 |
Nim/gcc (Arraymancer) | 0.064±0.004 | 5.41±00.06 + 57.41±00.10 | 5.27±00.20 |
Python (NumPy) | 0.065±0.000 | 33.37±00.06 + 57.87±00.04 | 5.54±00.01 |
Java (ND4J) | 0.076±0.001 | 114.91±01.04 + 92.22±00.01 | 5.97±00.05 |
Rust (ndarray) | 0.089±0.001 | 2.50±00.04 + 68.53±00.00 | 6.03±00.08 |
Julia (threads: 2) | 0.092±0.000 | 300.38±00.45 + 47.53±00.53 | 5.54±00.03 |
Nim/clang (Arraymancer) | 0.128±0.020 | 6.05±00.18 + 57.54±00.17 | 8.78±01.10 |
Julia (threads: 1) | 0.142±0.000 | 299.90±00.18 + 47.76±00.23 | 6.91±00.04 |
C++/g++ (Eigen) | 0.145±0.000 | 4.49±00.06 + 85.26±00.00 | 7.14±00.03 |
C++/clang++ (Eigen) | 0.145±0.000 | 4.72±00.03 + 85.37±00.00 | 7.07±00.04 |
V/clang (VSL + CBLAS) | 0.254±0.004 | 7.13±00.03 + 51.89±00.00 | 17.98±00.26 |
V/clang (VSL) | 0.258±0.002 | 7.21±00.17 + 51.57±00.13 | 17.89±00.11 |
V/gcc (VSL + CBLAS) | 0.459±0.002 | 7.25±00.06 + 51.57±00.00 | 34.63±00.06 |
V/gcc (VSL) | 0.466±0.001 | 6.90±00.04 + 51.89±00.00 | 31.75±00.07 |
Julia (no BLAS) | 1.146±0.023 | 271.40±00.23 + 52.30±00.01 | 49.02±00.55 |
D/ldc2 | 1.714±0.001 | 3.43±00.04 + 70.44±00.00 | 62.98±00.24 |
D/gdc | 1.866±0.001 | 7.40±00.14 + 70.16±00.01 | 72.94±00.11 |
D/dmd | 1.874±0.001 | 3.32±00.03 + 70.46±00.03 | 70.33±00.49 |
C/gcc | 3.025±0.000 | 1.53±00.02 + 68.65±00.02 | 109.57±01.18 |
V/gcc | 3.027±0.001 | 2.60±00.03 + 68.58±00.00 | 112.33±00.33 |
Vala/clang | 3.057±0.000 | 5.52±00.03 + 68.32±00.00 | 104.60±00.31 |
V/clang | 3.058±0.000 | 2.90±00.06 + 68.58±00.00 | 104.38±00.16 |
C/clang | 3.060±0.000 | 1.55±00.02 + 68.63±00.04 | 104.48±00.15 |
Rust | 3.061±0.000 | 2.14±00.07 + 68.69±00.00 | 105.19±00.52 |
Zig | 3.063±0.001 | 1.92±00.01 + 68.58±00.00 | 108.31±00.18 |
Nim/gcc | 3.086±0.001 | 2.60±00.01 + 58.91±01.16 | 114.12±00.58 |
Swift | 3.090±0.000 | 7.99±00.06 + 68.69±00.00 | 109.94±00.59 |
Nim/clang | 3.116±0.001 | 2.88±00.01 + 57.75±00.00 | 107.30±00.87 |
Vala/gcc | 3.123±0.000 | 5.49±00.10 + 68.32±00.00 | 114.09±00.15 |
Go | 3.145±0.000 | 4.09±00.01 + 0.00±00.00 | 112.66±01.59 |
Crystal | 3.147±0.001 | 3.63±00.06 + 60.02±00.02 | 115.55±00.61 |
Go/gccgo | 3.150±0.000 | 24.19±00.16 + 0.00±00.00 | 110.47±00.13 |
Java | 3.165±0.002 | 40.98±00.16 + 74.30±00.20 | 117.37±01.36 |
Kotlin/JVM | 3.209±0.003 | 42.32±00.21 + 73.03±00.22 | 129.95±00.33 |
Node.js | 3.252±0.002 | 51.38±00.26 + 70.87±00.47 | 130.78±00.37 |
Python/pypy | 3.254±0.001 | 59.96±00.07 + 68.90±00.03 | 135.03±00.17 |
Scala | 3.332±0.003 | 71.86±00.08 + 153.79±00.24 | 120.95±00.68 |
C#/.NET Core | 4.893±0.001 | 34.69±00.07 + 68.91±00.03 | 196.73±00.66 |
C#/Mono | 7.391±0.000 | 26.02±00.07 + 69.48±00.01 | 297.35±00.51 |
Ruby/truffleruby | 18.156±3.646 | 360.35±12.57 + 574.17±41.24 | 647.87±107.81 |
Ruby/truffleruby (JVM) | 25.014±0.847 | 407.30±16.57 + 334.68±58.11 | 875.73±22.67 |
Perl | 149.225±1.755 | 8.52±00.04 + 379.63±00.02 | 6447.37±59.99 |
Python | 155.211±1.106 | 10.91±00.03 + 68.84±00.00 | 7060.84±46.09 |
Ruby (--jit) | 156.368±0.039 | 21.96±00.04 + 68.35±00.02 | 6925.51±35.94 |
Ruby | 195.516±0.462 | 11.75±00.07 + 69.19±00.01 | 8795.03±25.92 |
Tcl | 202.975±0.793 | 7.47±00.02 + 400.44±00.00 | 9311.02±86.81 |
Ruby/jruby | 361.315±19.249 | 273.70±12.97 + 1142.23±73.73 | 14778.53±653.23 |
Testing:
Notes:
Language | Time, s | Memory, MiB | Energy, J |
---|---|---|---|
Zig | 0.056±0.000 | 1.03±00.02 + 52.80±00.19 | 2.34±00.03 |
C++/clang++ | 0.062±0.000 | 3.17±00.03 + 54.80±00.00 | 2.38±00.02 |
C++/g++ | 0.063±0.000 | 3.67±00.07 + 70.55±00.25 | 2.44±00.02 |
Go | 0.072±0.000 | 3.08±00.05 + 0.00±00.00 | 3.06±00.04 |
V/clang | 0.095±0.000 | 2.14±00.23 + 211.15±00.64 | 3.91±00.04 |
V/gcc | 0.099±0.000 | 1.95±00.03 + 201.60±00.17 | 4.10±00.03 |
Rust | 0.101±0.000 | 2.09±00.11 + 74.71±00.77 | 3.98±00.05 |
Java | 0.130±0.004 | 39.84±00.17 + 123.02±05.25 | 7.33±00.26 |
Crystal | 0.138±0.000 | 3.71±00.02 + 88.43±00.00 | 5.58±00.05 |
Node.js | 0.218±0.002 | 42.17±00.00 + 147.09±02.74 | 10.79±00.07 |
Scala | 0.228±0.003 | 72.13±00.08 + 230.29±00.88 | 13.20±00.22 |
Nim/clang | 0.278±0.000 | 2.10±00.10 + 587.68±01.03 | 10.75±00.04 |
Nim/gcc | 0.287±0.000 | 1.90±00.02 + 615.91±00.00 | 10.83±00.07 |
Lua/luajit | 0.298±0.001 | 1.33±00.02 + 156.66±00.63 | 12.00±00.08 |
Python/pypy | 0.616±0.003 | 58.72±00.13 + 249.30±00.15 | 25.12±00.16 |
Julia | 0.651±0.001 | 271.49±00.14 + 370.21±01.14 | 25.06±00.21 |
Racket | 0.752±0.002 | 109.49±00.82 + 247.11±00.45 | 29.59±00.17 |
Ruby/truffleruby | 0.890±0.012 | 203.57±01.55 + 800.19±11.11 | 60.49±00.76 |
Lua | 1.142±0.006 | 2.69±00.04 + 283.50±01.03 | 46.78±00.35 |
Ruby (--jit) | 1.187±0.002 | 22.98±00.02 + 163.71±00.14 | 47.95±00.43 |
Ruby/truffleruby (JVM) | 1.378±0.088 | 376.75±09.21 + 495.42±42.25 | 89.49±05.72 |
Ruby | 1.860±0.003 | 11.47±00.07 + 172.19±00.35 | 76.75±00.60 |
Ruby/jruby | 2.314±0.075 | 197.13±08.38 + 521.12±52.24 | 124.24±05.36 |
Python | 2.530±0.014 | 10.84±00.04 + 181.69±01.57 | 107.30±01.03 |
CPU: Intel(R) Xeon(R) E-2324G
Base Docker image: Debian GNU/Linux bookworm/sid
Language | Version |
---|---|
.NET Core | 8.0.300 |
C#/.NET Core | 4.10.0-3.24216.12 (3af0081a) |
C#/Mono | 6.12.0.200 |
Chez Scheme | 9.5.8 |
Clojure | "1.11.3" |
Crystal | 1.12.1 |
D/dmd | v2.108.1 |
D/gdc | 13.2.0 |
D/ldc2 | 1.38.0 |
Elixir | 1.14.0 |
F#/.NET Core | 12.8.300.0 for F# 8.0 |
Go | go1.22.3 |
Go/gccgo | 13.2.0 |
Haskell | 9.8.2 |
Idris 2 | 0.6.0 |
Java | 22.0.1 |
Julia | v"1.10.3" |
Kotlin | 2.0.0 |
Lua | 5.4.6 |
Lua/luajit | 2.1.1710398010 |
MLton | 20210117 |
Nim | 2.0.4 |
Node.js | v22.2.0 |
OCaml | 5.2.0 |
PHP | 8.2.18 |
Perl | v5.38.2 |
Python | 3.11.9 |
Python/pypy | 7.3.16-final0 for Python 3.10.14 |
Racket | "8.13" |
Ruby | 3.3.1p55 |
Ruby/jruby | 9.4.7.0 |
Ruby/truffleruby | 24.0.1 |
Rust | 1.78.0 |
Scala | 3.4.2 |
Swift | 5.10 |
Tcl | 8.6 |
V | 0.4.6 736067d |
Vala | 0.56.17 |
Zig | 0.12.0 |
clang/clang++ | 16.0.6 (27) |
gcc/g++ | 13.2.0 |
Build the image:
$ docker build docker/ -t benchmarks
Run the image:
$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>
where <cmd>
is:
versions
(print installed language versions);shell
(start the shell);brainfuck bench
(build and run Brainfuck bench.b benchmarks);brainfuck mandel
(build and run Brainfuck mandel.b benchmarks);base64
(build and run Base64 benchmarks);json
(build and run Json benchmarks);matmul
(build and run Matmul benchmarks);primes
(build and run Primes benchmarks);Please note that the actual measurements provided in the project are taken semi-manually (via shell
) as the full update takes days and could have occassional issues in Docker.
There is a ./run.sh
that could be used to simplify Docker usage:
./run.sh build
(build the image);./run.sh make versions
(run the image with the versions
command);sudo ./run.sh shell
(run the image with the `shell' command, sudo is required to read energy levels).Makefiles contain recipes for building and executing tests with the
proper dependencies. Please use make run
(and make run2
where applicable).
The measurements are taken using analyze.rb
script:
$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]
Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:
$ ATTEMPTS=1 ../analyze.rb make run
Please use Dockerfile as a reference regarding which packages and tools are required.
For all (optional):
powercap-utils
).For Python:
python3-numpy
).python3-ujson
).For C++:
libboost-dev
).libjson-c-dev
).For Rust:
libjq-dev
, libonig-dev
and environment variable
JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/
).For Java, Scala:
For Haskell:
For Perl:
cpanminus
).For Vala:
libjson-glib-dev
).Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:
If the test is compiled into a single binary, then two sections of
the Makefile
require changes:
executables
variable;If the test is compiled, but can't be executed directly as a binary, then
three sections of the Makefile
require changes:
artifacts
variable;run[<target_artifact>]
rule to run the test.If the test doesn't require compilation, then two sections of the Makefile
requires changes:
run[<script_file>]
into all_runners
variable;run[<script_file>]
rule to run the test.TOC is regenerated using git-markdown-toc:
./run.sh toc