kostya / benchmarks

Some benchmarks of different languages

MIT License

2.79k stars 253 forks source link

benchmarks languages

readme

Table of Content

Overview
- Measurements
Test Cases
- Brainfuck
- bench.b
- mandel.b
- Base64
- Json
- Matmul
- Primes
Tests Execution
Contribution

Overview

The benchmarks follow the criteria:

They are written as the average software developer would write them, i.e.
- The algorithms are implemented as cited in public sources;
- The libraries are used as described in the tutorials, documentation and examples;
- The used data structures are idiomatic.
The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.
All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

time spent for the benchmark execution (loading required data and code self-testing are not measured);
memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median_{±median absolute deviation}.

UPDATE: 2024-06-06

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

Verbose (default). Prints the output immediately.
Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

bench.b

Language	Time, s	Memory, MiB	Energy, J
Scala (Staged)	0.390_±0.015	222.77_±03.17 + 15.42_±01.67	24.89_±00.74
Racket (Staged)	0.893_±0.001	101.38_±00.97 + 0.00_±00.00	34.61_±00.07
Rust	1.011_±0.000	1.05_±00.01 + 0.00_±00.00	43.04_±00.28
V/gcc	1.060_±0.000	1.92_±00.03 + 0.00_±00.00	43.67_±00.15
C/gcc	1.111_±0.001	1.01_±00.01 + 0.00_±00.00	47.01_±00.65
C++/g++	1.113_±0.001	1.98_±00.04 + 0.00_±00.00	46.04_±00.34
D/gdc	1.120_±0.000	6.39_±00.05 + 0.00_±00.00	49.26_±00.57
C++/clang++	1.125_±0.001	1.74_±00.02 + 0.00_±00.00	46.90_±00.51
C/clang	1.140_±0.001	1.02_±00.01 + 0.00_±00.00	48.57_±00.37
D/ldc2	1.163_±0.003	2.32_±00.80 + 0.00_±00.00	49.08_±00.30
Nim/gcc	1.168_±0.001	1.01_±00.02 + 0.00_±00.00	49.66_±00.25
Java	1.193_±0.000	39.79_±00.09 + 1.55_±00.06	49.18_±00.26
Vala/gcc	1.210_±0.001	4.50_±00.02 + 0.00_±00.00	50.69_±00.11
Kotlin/JVM	1.222_±0.002	43.53_±00.14 + 0.52_±00.26	50.93_±00.10
Vala/clang	1.238_±0.006	4.52_±00.03 + 0.00_±00.00	52.25_±01.41
Go	1.247_±0.000	3.55_±00.03 + 0.00_±00.00	51.47_±00.33
Zig	1.265_±0.000	1.03_±00.02 + 0.00_±00.00	52.54_±00.44
C#/.NET Core	1.367_±0.001	32.66_±00.13 + 0.15_±00.02	58.23_±00.27
Go/gccgo	1.481_±0.002	23.56_±00.01 + 0.00_±00.00	62.86_±00.67
Nim/clang	1.565_±0.000	1.29_±00.01 + 0.00_±00.00	64.41_±00.46
F#/.NET Core	1.574_±0.004	37.41_±00.18 + 0.30_±00.00	67.12_±00.40
Crystal	1.597_±0.008	3.00_±00.05 + 0.00_±00.00	67.57_±00.71
OCaml	1.638_±0.005	3.01_±00.05 + 2.81_±00.06	76.01_±00.28
Julia	1.638_±0.003	256.65_±00.11 + 0.40_±00.03	69.57_±00.34
Chez Scheme	1.714_±0.004	24.77_±00.02 + 4.46_±00.12	72.48_±00.24
Racket	1.756_±0.028	113.87_±00.09 + 0.00_±00.00	71.30_±01.18
V/clang	1.973_±0.018	1.96_±00.03 + 0.00_±00.00	85.72_±01.29
C#/Mono	2.054_±0.011	25.60_±00.08 + 0.00_±00.00	87.72_±01.03
MLton	2.088_±0.027	1.77_±00.04 + 0.25_±00.00	86.67_±01.18
Scala	2.779_±0.003	72.03_±00.11 + 249.04_±00.15	121.64_±00.71
Node.js	2.920_±0.028	44.53_±00.02 + 4.37_±00.00	123.76_±01.03
Haskell (MArray)	3.095_±0.007	3.17_±00.04 + 4.92_±00.04	130.42_±01.60
D/dmd	3.326_±0.001	3.39_±00.03 + 0.00_±00.00	124.72_±00.18
Haskell (FP)	3.624_±0.002	3.27_±00.05 + 4.89_±00.04	154.06_±00.80
Ruby/truffleruby (JVM)	4.996_±0.396	374.17_±10.09 + 496.92_±17.09	243.84_±22.97
Ruby/truffleruby	5.193_±0.358	203.03_±01.33 + 550.75_±59.29	253.87_±16.16
Swift	5.656_±0.005	16.69_±00.04 + 0.00_±00.00	211.92_±00.58
Lua/luajit	5.881_±0.025	2.54_±00.02 + 0.00_±00.00	241.87_±05.05
Python/pypy	9.544_±0.044	58.79_±00.12 + 29.96_±00.01	422.77_±04.07
Idris	15.129_±0.051	20.71_±00.03 + 8.83_±00.04	661.53_±10.66
Elixir	20.395_±0.043	70.50_±00.79 + 0.00_±00.00	807.25_±05.02
Ruby (--jit)	31.817_±0.055	21.59_±00.04 + 4.76_±00.01	1311.22_±06.23
PHP	34.492_±0.160	17.78_±00.18 + 0.00_±00.00	1476.88_±31.23
Lua	36.833_±0.101	2.28_±00.03 + 0.00_±00.00	1512.87_±19.03
Ruby	68.430_±0.445	11.43_±00.04 + 0.00_±00.00	3008.91_±26.07
Python	69.075_±0.780	11.17_±00.08 + 0.00_±00.00	3039.83_±50.68
Ruby/jruby	81.684_±1.546	203.50_±04.69 + 220.84_±20.47	3624.22_±70.81
Tcl (FP)	192.447_±0.637	4.06_±00.02 + 0.00_±00.00	8478.25_±83.42
Perl	225.331_±1.241	6.90_±00.05 + 0.00_±00.00	10000.48_±94.38
Tcl (OOP)	384.789_±1.841	4.06_±00.01 + 0.00_±00.00	17091.07_±181.64

mandel.b

Mandel in Brainfuck

Language	Time, s	Memory, MiB	Energy, J
Scala (Staged)	7.639_±0.120	224.42_±01.90 + 102.36_±04.76	459.89_±14.99
C++/g++	9.755_±0.030	1.97_±00.03 + 2.29_±00.11	393.99_±02.68
C#/.NET Core	12.048_±0.028	32.72_±00.06 + 1.27_±00.00	482.24_±01.22
Java	12.450_±0.060	39.92_±00.06 + 2.44_±00.04	489.83_±01.92
C/gcc	12.513_±0.013	0.99_±00.02 + 0.84_±00.06	505.46_±02.16
Kotlin/JVM	13.082_±0.033	43.53_±00.17 + 2.30_±00.33	547.94_±03.79
F#/.NET Core	13.115_±0.017	37.55_±00.07 + 2.12_±00.04	526.18_±01.55
C/clang	13.293_±0.031	0.97_±00.03 + 0.75_±00.03	572.78_±05.53
V/gcc	13.591_±0.007	1.91_±00.03 + 1.18_±00.04	547.49_±03.65
C++/clang++	13.898_±0.024	1.72_±00.05 + 1.88_±00.03	569.56_±02.36
Racket (Staged)	13.986_±0.059	99.66_±00.50 + 77.13_±01.67	556.40_±01.66
Crystal	14.103_±0.018	2.96_±00.03 + 0.74_±00.04	598.53_±02.62
Go	14.182_±0.009	3.56_±00.01 + 0.00_±00.00	562.34_±00.49
Rust	14.229_±0.012	1.04_±00.01 + 1.20_±00.04	564.72_±02.02
D/gdc	14.266_±0.016	6.37_±00.06 + 1.49_±00.04	601.20_±03.61
D/ldc2	14.278_±0.013	3.05_±00.02 + 0.83_±00.02	572.27_±01.12
Vala/gcc	14.469_±0.022	4.47_±00.04 + 1.24_±00.04	574.39_±01.98
Zig	14.481_±0.014	1.02_±00.02 + 1.42_±00.06	600.92_±02.27
Vala/clang	14.845_±0.010	4.46_±00.02 + 1.22_±00.03	607.68_±02.95
Nim/gcc	15.641_±0.030	2.07_±00.06 + 1.29_±00.00	663.67_±00.73
Scala	16.365_±0.031	71.96_±00.25 + 139.77_±00.37	735.09_±03.15
Swift	18.421_±0.037	16.45_±00.04 + 0.00_±00.00	762.02_±06.09
Go/gccgo	19.091_±0.406	23.58_±00.05 + 0.00_±00.00	796.35_±13.26
Nim/clang	19.696_±0.204	2.38_±00.01 + 1.29_±00.00	806.37_±09.58
V/clang	20.895_±0.225	1.94_±00.05 + 1.17_±00.00	901.27_±12.94
OCaml	24.390_±0.016	4.53_±00.08 + 2.82_±00.00	1171.79_±06.03
Julia	26.058_±0.070	256.43_±00.04 + 0.30_±00.00	1021.32_±06.88
Chez Scheme	27.698_±0.056	25.55_±00.06 + 3.68_±00.01	1210.07_±04.32
C#/Mono	31.025_±0.011	25.61_±00.09 + 0.83_±00.00	1289.92_±08.22
MLton	33.842_±0.072	1.73_±00.05 + 4.11_±00.00	1537.44_±17.63
Haskell (MArray)	34.546_±0.022	4.08_±00.02 + 5.11_±00.00	1400.92_±05.61
Node.js	34.598_±0.063	44.44_±00.07 + 5.25_±00.00	1378.99_±02.63
Lua/luajit	34.656_±0.034	2.56_±00.04 + 0.38_±00.00	1388.13_±02.21
Racket	35.183_±0.294	113.90_±00.09 + 1.45_±00.65	1567.04_±13.20
D/dmd	37.942_±0.003	3.35_±00.03 + 0.87_±00.02	1378.02_±00.92
Python/pypy	41.463_±0.088	58.86_±00.09 + 30.60_±00.03	1837.98_±07.63
Ruby/truffleruby	48.320_±0.851	203.25_±00.90 + 581.44_±30.40	2360.50_±62.00
Ruby/truffleruby (JVM)	50.546_±1.177	374.73_±03.49 + 489.39_±89.50	2202.33_±46.95
Idris	66.241_±0.299	22.03_±00.05 + 9.54_±00.00	2848.93_±05.21
Haskell (FP)	77.093_±0.126	4.19_±00.02 + 75.74_±00.03	3195.67_±11.41

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Language	Time, s	Memory, MiB	Energy, J
C/gcc (aklomp)	0.098_±0.000	2.20_±00.03 + 0.00_±00.00	4.60_±00.03
C/clang (aklomp)	0.099_±0.000	2.10_±00.06 + 0.00_±00.00	4.68_±00.05
PHP	0.105_±0.000	18.56_±00.06 + 0.00_±00.00	4.76_±00.07
Go (base64x)	0.267_±0.001	6.61_±00.02 + 0.00_±00.00	12.80_±00.11
Zig	0.698_±0.000	1.66_±00.04 + 0.00_±00.00	26.49_±00.19
Rust	0.849_±0.000	2.42_±00.02 + 0.00_±00.00	35.03_±00.11
Node.js	0.913_±0.001	43.17_±00.03 + 40.43_±00.21	40.38_±00.37
C/clang	0.997_±0.000	2.15_±00.01 + 0.00_±00.00	36.62_±00.05
C/gcc	1.101_±0.016	2.10_±00.05 + 0.00_±00.00	40.24_±00.60
Nim/clang	1.103_±0.001	2.06_±00.02 + 5.79_±00.03	44.95_±00.27
Crystal	1.107_±0.003	3.61_±00.04 + 1.26_±00.03	44.93_±00.29
D/ldc2	1.175_±0.003	3.68_±00.03 + 3.41_±00.00	48.49_±00.44
Nim/gcc	1.364_±0.002	1.75_±00.04 + 4.96_±00.01	55.70_±00.32
Ruby (--jit)	1.427_±0.002	15.10_±00.16 + 73.37_±00.56	55.92_±00.37
Java	1.518_±0.005	41.05_±00.10 + 210.65_±19.06	60.66_±00.44
V/clang	1.534_±0.001	2.43_±00.01 + 2386.75_±01.65	57.81_±00.29
V/gcc	1.575_±0.000	2.43_±00.03 + 2385.17_±00.74	57.34_±00.19
Scala	1.604_±0.001	70.48_±00.15 + 276.28_±02.12	66.81_±00.54
Go	1.622_±0.002	4.34_±00.02 + 0.00_±00.00	67.66_±00.22
Ruby	1.642_±0.005	11.69_±00.04 + 42.50_±00.30	64.29_±00.22
Vala/clang	1.643_±0.001	5.68_±00.03 + 0.10_±00.03	63.09_±00.70
Vala/gcc	1.644_±0.001	5.66_±00.01 + 0.08_±00.05	62.92_±00.40
Kotlin/JVM	1.665_±0.003	44.48_±00.20 + 314.13_±03.17	67.36_±00.44
C++/g++ (libcrypto)	1.711_±0.003	6.05_±00.16 + 0.76_±00.03	68.53_±00.38
C++/clang++ (libcrypto)	1.713_±0.002	5.51_±00.04 + 0.73_±00.02	69.23_±00.37
Perl (MIME::Base64)	1.863_±0.044	14.57_±00.04 + 0.12_±00.03	75.20_±01.07
F#/.NET Core	2.038_±0.018	38.32_±00.07 + 11.92_±00.65	80.42_±00.77
C#/.NET Core	2.181_±0.021	33.77_±00.03 + 12.54_±02.09	85.47_±00.28
D/gdc	2.382_±0.001	7.44_±00.04 + 3.36_±00.00	103.50_±00.94
Go/gccgo	2.949_±0.007	24.81_±00.17 + 0.00_±00.00	139.67_±00.50
Julia	3.079_±0.006	271.96_±00.08 + 97.24_±00.20	124.20_±01.58
Python/pypy	3.236_±0.002	58.92_±00.06 + 31.49_±00.17	141.88_±00.91
D/dmd	3.387_±0.004	3.17_±00.02 + 3.84_±00.02	141.65_±00.88
Tcl	3.438_±0.001	5.20_±00.02 + 0.00_±00.00	137.53_±00.49
Python	3.512_±0.001	10.78_±00.16 + 0.07_±00.07	133.94_±00.44
Ruby/truffleruby (JVM)	3.758_±0.016	374.30_±05.93 + 247.06_±24.79	189.38_±03.07
Racket	3.885_±0.006	95.36_±00.20 + 21.80_±00.40	154.27_±00.56
C#/Mono	4.632_±0.005	26.39_±00.04 + 18.66_±00.02	190.60_±01.36
Ruby/jruby	6.005_±0.008	195.03_±02.11 + 165.97_±10.93	263.57_±01.49
Ruby/truffleruby	8.324_±0.018	201.01_±02.82 + 554.96_±13.20	400.06_±01.94
Perl (MIME::Base64::Perl)	10.148_±0.055	15.93_±00.11 + 0.26_±00.04	448.06_±03.65

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

gason mutates input strings;
simdjson requires input strings with batch of trailing zeros: a special zero padding for SIMD instructions;
DAW JSON Link "NoCheck" skips some JSON structure correctness checks;
DAW JSON Link, gason, default (not "Precise") RapidJSON, and D implementations except Mir-based have some inaccuracies in number parsing:

Language	Time, s	Memory, MiB	Energy, J
C++/clang++ (simdjson On-Demand)	0.060_±0.000	112.46_±00.06 + 60.05_±00.06	2.51_±00.01
C++/g++ (simdjson On-Demand)	0.061_±0.000	113.66_±00.03 + 59.81_±00.00	2.57_±00.02
C++/clang++ (DAW JSON Link NoCheck)	0.083_±0.000	112.54_±00.05 + 0.00_±00.00	3.37_±00.03
C++/g++ (DAW JSON Link NoCheck)	0.087_±0.000	113.29_±00.06 + 0.00_±00.00	3.53_±00.04
C++/clang++ (DAW JSON Link)	0.093_±0.000	112.47_±00.03 + 0.00_±00.00	3.90_±00.07
C++/g++ (DAW JSON Link)	0.093_±0.000	113.30_±00.05 + 0.00_±00.00	3.88_±00.03
Rust (Serde Custom)	0.098_±0.001	111.40_±00.08 + 0.00_±00.00	4.14_±00.04
C++/clang++ (simdjson DOM)	0.102_±0.000	112.55_±00.09 + 177.09_±00.06	4.62_±00.02
C++/g++ (simdjson DOM)	0.108_±0.000	113.54_±00.06 + 172.73_±00.13	4.90_±00.02
Rust (Serde Typed)	0.111_±0.000	111.71_±00.05 + 11.05_±00.07	4.62_±00.04
D/ldc2 (Mir Asdf DOM)	0.131_±0.000	112.93_±00.04 + 61.22_±00.00	5.43_±00.05
C++/clang++ (gason)	0.139_±0.000	112.57_±00.02 + 96.97_±00.06	5.67_±00.04
C++/g++ (gason)	0.140_±0.000	113.17_±00.06 + 96.93_±00.04	5.55_±00.03
C++/g++ (RapidJSON)	0.152_±0.000	113.21_±00.04 + 128.90_±00.03	6.47_±00.04
Scala (jsoniter-scala)	0.157_±0.002	290.75_±00.24 + 20.43_±00.24	8.36_±00.07
Go (rjson custom)	0.196_±0.000	113.58_±00.05 + 0.00_±00.00	7.57_±00.02
C++/clang++ (RapidJSON)	0.200_±0.000	112.47_±00.04 + 128.91_±00.09	8.55_±00.08
Go (Sonic)	0.208_±0.002	122.51_±00.12 + 0.00_±00.00	8.94_±00.09
C++/g++ (RapidJSON Precise)	0.217_±0.000	113.21_±00.06 + 122.47_±01.29	9.28_±00.08
D/ldc2 (Mir Amazon's Ion DOM)	0.219_±0.000	113.03_±00.08 + 80.70_±00.00	9.26_±00.03
Go (rjson)	0.224_±0.000	113.61_±00.10 + 0.00_±00.00	8.70_±00.02
Zig	0.241_±0.000	111.03_±00.02 + 39.25_±00.00	10.37_±00.07
Go (goccy/go-json)	0.266_±0.000	114.29_±00.07 + 0.00_±00.00	10.56_±00.02
C++/clang++ (RapidJSON Precise)	0.285_±0.000	112.48_±00.03 + 128.91_±00.03	12.50_±00.10
C++/g++ (RapidJSON SAX)	0.346_±0.001	113.11_±00.08 + 0.00_±00.00	15.32_±00.06
C/clang (yajl)	0.350_±0.001	110.97_±00.04 + 0.00_±00.00	15.14_±00.06
C/gcc (yajl)	0.353_±0.001	110.95_±00.01 + 0.00_±00.00	15.22_±00.06
C++/g++ (Boost.JSON)	0.363_±0.000	113.32_±00.04 + 308.09_±00.02	15.44_±00.10
C++/clang++ (Boost.JSON)	0.371_±0.001	112.68_±00.03 + 308.09_±00.03	15.74_±00.10
Nim/clang (jsony)	0.392_±0.000	111.56_±00.05 + 146.12_±00.09	16.46_±00.12
C++/g++ (RapidJSON SAX Precise)	0.401_±0.001	113.12_±00.06 + 0.00_±00.00	17.92_±00.23
C++/clang++ (RapidJSON SAX)	0.408_±0.001	194.77_±00.07 + 0.00_±00.00	17.40_±00.07
Nim/gcc (jsony)	0.412_±0.000	111.23_±00.03 + 154.69_±00.03	17.46_±00.15
Node.js	0.470_±0.004	152.69_±00.04 + 195.96_±00.68	22.05_±00.12
C++/clang++ (RapidJSON SAX Precise)	0.489_±0.001	194.79_±00.07 + 0.00_±00.00	21.84_±00.23
Go (jsoniter)	0.513_±0.000	114.27_±00.06 + 0.00_±00.00	20.75_±00.17
Rust (Serde Untyped)	0.532_±0.001	111.73_±00.03 + 839.98_±00.00	22.14_±00.20
C#/.NET Core (System.Text.Json)	0.543_±0.002	489.53_±00.11 + 140.62_±00.22	24.07_±00.23
Java (DSL-JSON)	0.568_±0.007	262.39_±00.11 + 225.35_±19.01	29.30_±00.32
Python/pypy	0.608_±0.002	279.98_±00.08 + 125.71_±00.04	26.46_±00.26
V/gcc	0.615_±0.001	111.47_±00.04 + 496.21_±00.00	25.70_±00.23
V/clang	0.637_±0.001	111.87_±00.01 + 495.83_±00.00	26.74_±00.19
Nim/gcc (Packedjson)	0.638_±0.003	111.93_±00.02 + 294.16_±00.00	27.28_±00.40
Crystal (Pull)	0.638_±0.002	113.26_±00.02 + 18.44_±00.00	28.10_±00.40
Crystal (Schema)	0.656_±0.001	113.23_±00.03 + 48.86_±00.13	28.63_±00.07
Nim/clang (Packedjson)	0.662_±0.001	112.23_±00.01 + 294.16_±00.00	28.31_±00.17
Perl (Cpanel::JSON::XS)	0.765_±0.003	125.17_±00.05 + 402.77_±00.00	31.84_±00.14
Go	0.813_±0.001	113.95_±00.06 + 0.00_±00.00	33.60_±00.06
PHP	0.813_±0.001	128.26_±00.08 + 517.83_±00.03	34.57_±00.20
Crystal	0.924_±0.004	113.21_±00.02 + 392.50_±00.03	39.68_±00.33
Nim/gcc	1.037_±0.002	111.91_±00.03 + 1001.34_±00.00	43.13_±00.12
C#/.NET Core	1.056_±0.004	495.03_±00.30 + 272.99_±00.16	50.49_±00.23
Nim/clang	1.076_±0.002	112.23_±00.00 + 999.02_±00.00	44.69_±00.27
C++/g++ (json-c)	1.159_±0.003	113.42_±00.05 + 1216.05_±00.01	47.99_±00.30
C++/clang++ (json-c)	1.160_±0.007	112.70_±00.07 + 1216.04_±00.01	48.35_±00.66
Clojure	1.161_±0.019	452.13_±02.50 + 587.46_±09.25	61.05_±00.99
C++/clang++ (Nlohmann)	1.176_±0.001	112.64_±00.03 + 360.11_±00.02	50.39_±00.68
Ruby (--jit)	1.280_±0.005	127.59_±00.08 + 212.39_±00.01	53.84_±00.51
Go/gccgo	1.290_±0.004	138.88_±00.10 + 0.00_±00.00	53.03_±00.36
C++/g++ (Nlohmann)	1.327_±0.003	113.27_±00.06 + 448.05_±00.01	56.39_±00.35
Ruby	1.328_±0.006	121.40_±00.05 + 212.77_±00.02	55.73_±00.73
CPython (UltraJSON)	1.331_±0.004	123.14_±00.05 + 497.28_±01.55	51.19_±00.22
Python	1.374_±0.002	120.99_±00.03 + 325.95_±00.01	55.23_±00.37
F#/.NET Core (System.Text.Json)	1.496_±0.003	498.18_±00.08 + 232.75_±04.63	68.50_±00.69
Ruby (YAJL)	1.719_±0.009	121.49_±00.11 + 218.39_±00.03	72.78_±00.52
D/ldc2	1.723_±0.003	112.86_±00.06 + 708.76_±00.04	72.08_±00.65
C#/Mono	1.809_±0.013	252.32_±00.17 + 31.51_±00.01	78.45_±00.96
Haskell	1.982_±0.007	115.63_±00.14 + 723.73_±00.48	84.53_±00.77
Rust (jq)	2.529_±0.003	113.39_±00.06 + 902.98_±01.54	106.08_±00.37
C++/g++ (Boost.PropertyTree)	2.611_±0.008	113.14_±00.04 + 1440.09_±00.04	111.72_±00.36
C++/clang++ (Boost.PropertyTree)	2.671_±0.005	195.01_±00.04 + 1232.80_±00.03	112.86_±00.70
Ruby/jruby	2.884_±0.022	464.98_±06.74 + 972.45_±49.49	148.64_±02.99
D/dmd	3.073_±0.003	113.16_±00.04 + 708.78_±00.07	131.07_±00.23
Vala/clang	3.165_±0.007	115.04_±00.04 + 980.08_±00.03	137.65_±00.31
Vala/gcc	3.166_±0.011	115.06_±00.03 + 980.07_±00.03	136.93_±00.87
D/gdc	3.569_±0.004	116.68_±00.06 + 680.99_±00.12	151.81_±01.75
Racket	3.818_±0.022	222.34_±00.45 + 261.38_±27.93	159.23_±01.29
Perl (JSON::Tiny)	9.050_±0.121	125.59_±00.05 + 528.92_±00.02	394.96_±03.88
Ruby/truffleruby	10.718_±0.048	522.13_±27.57 + 1942.99_±149.89	617.74_±02.74
Ruby/truffleruby (JVM)	11.011_±0.153	474.89_±08.90 + 2201.50_±87.97	693.97_±13.20

Matmul

Testing allocating and multiplying matrices.

Language	Time, s	Memory, MiB	Energy, J
D/ldc2 (lubeck)	0.032_±0.001	40.38_±00.48 + 23.61_±00.41	3.61_±00.09
Nim/gcc (Arraymancer)	0.064_±0.004	5.41_±00.06 + 57.41_±00.10	5.27_±00.20
Python (NumPy)	0.065_±0.000	33.37_±00.06 + 57.87_±00.04	5.54_±00.01
Java (ND4J)	0.076_±0.001	114.91_±01.04 + 92.22_±00.01	5.97_±00.05
Rust (ndarray)	0.089_±0.001	2.50_±00.04 + 68.53_±00.00	6.03_±00.08
Julia (threads: 2)	0.092_±0.000	300.38_±00.45 + 47.53_±00.53	5.54_±00.03
Nim/clang (Arraymancer)	0.128_±0.020	6.05_±00.18 + 57.54_±00.17	8.78_±01.10
Julia (threads: 1)	0.142_±0.000	299.90_±00.18 + 47.76_±00.23	6.91_±00.04
C++/g++ (Eigen)	0.145_±0.000	4.49_±00.06 + 85.26_±00.00	7.14_±00.03
C++/clang++ (Eigen)	0.145_±0.000	4.72_±00.03 + 85.37_±00.00	7.07_±00.04
V/clang (VSL + CBLAS)	0.254_±0.004	7.13_±00.03 + 51.89_±00.00	17.98_±00.26
V/clang (VSL)	0.258_±0.002	7.21_±00.17 + 51.57_±00.13	17.89_±00.11
V/gcc (VSL + CBLAS)	0.459_±0.002	7.25_±00.06 + 51.57_±00.00	34.63_±00.06
V/gcc (VSL)	0.466_±0.001	6.90_±00.04 + 51.89_±00.00	31.75_±00.07
Julia (no BLAS)	1.146_±0.023	271.40_±00.23 + 52.30_±00.01	49.02_±00.55
D/ldc2	1.714_±0.001	3.43_±00.04 + 70.44_±00.00	62.98_±00.24
D/gdc	1.866_±0.001	7.40_±00.14 + 70.16_±00.01	72.94_±00.11
D/dmd	1.874_±0.001	3.32_±00.03 + 70.46_±00.03	70.33_±00.49
C/gcc	3.025_±0.000	1.53_±00.02 + 68.65_±00.02	109.57_±01.18
V/gcc	3.027_±0.001	2.60_±00.03 + 68.58_±00.00	112.33_±00.33
Vala/clang	3.057_±0.000	5.52_±00.03 + 68.32_±00.00	104.60_±00.31
V/clang	3.058_±0.000	2.90_±00.06 + 68.58_±00.00	104.38_±00.16
C/clang	3.060_±0.000	1.55_±00.02 + 68.63_±00.04	104.48_±00.15
Rust	3.061_±0.000	2.14_±00.07 + 68.69_±00.00	105.19_±00.52
Zig	3.063_±0.001	1.92_±00.01 + 68.58_±00.00	108.31_±00.18
Nim/gcc	3.086_±0.001	2.60_±00.01 + 58.91_±01.16	114.12_±00.58
Swift	3.090_±0.000	7.99_±00.06 + 68.69_±00.00	109.94_±00.59
Nim/clang	3.116_±0.001	2.88_±00.01 + 57.75_±00.00	107.30_±00.87
Vala/gcc	3.123_±0.000	5.49_±00.10 + 68.32_±00.00	114.09_±00.15
Go	3.145_±0.000	4.09_±00.01 + 0.00_±00.00	112.66_±01.59
Crystal	3.147_±0.001	3.63_±00.06 + 60.02_±00.02	115.55_±00.61
Go/gccgo	3.150_±0.000	24.19_±00.16 + 0.00_±00.00	110.47_±00.13
Java	3.165_±0.002	40.98_±00.16 + 74.30_±00.20	117.37_±01.36
Kotlin/JVM	3.209_±0.003	42.32_±00.21 + 73.03_±00.22	129.95_±00.33
Node.js	3.252_±0.002	51.38_±00.26 + 70.87_±00.47	130.78_±00.37
Python/pypy	3.254_±0.001	59.96_±00.07 + 68.90_±00.03	135.03_±00.17
Scala	3.332_±0.003	71.86_±00.08 + 153.79_±00.24	120.95_±00.68
C#/.NET Core	4.893_±0.001	34.69_±00.07 + 68.91_±00.03	196.73_±00.66
C#/Mono	7.391_±0.000	26.02_±00.07 + 69.48_±00.01	297.35_±00.51
Ruby/truffleruby	18.156_±3.646	360.35_±12.57 + 574.17_±41.24	647.87_±107.81
Ruby/truffleruby (JVM)	25.014_±0.847	407.30_±16.57 + 334.68_±58.11	875.73_±22.67
Perl	149.225_±1.755	8.52_±00.04 + 379.63_±00.02	6447.37_±59.99
Python	155.211_±1.106	10.91_±00.03 + 68.84_±00.00	7060.84_±46.09
Ruby (--jit)	156.368_±0.039	21.96_±00.04 + 68.35_±00.02	6925.51_±35.94
Ruby	195.516_±0.462	11.75_±00.07 + 69.19_±00.01	8795.03_±25.92
Tcl	202.975_±0.793	7.47_±00.02 + 400.44_±00.00	9311.02_±86.81
Ruby/jruby	361.315_±19.249	273.70_±12.97 + 1142.23_±73.73	14778.53_±653.23

Primes

Testing:

generating primes using the optimized sieve of Atkin;
prefix search for their decimal numbers using Trie data structure.

Notes:

All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
The results are always sorted (could be unstable or stable though).

Language	Time, s	Memory, MiB	Energy, J
Zig	0.056_±0.000	1.03_±00.02 + 52.80_±00.19	2.34_±00.03
C++/clang++	0.062_±0.000	3.17_±00.03 + 54.80_±00.00	2.38_±00.02
C++/g++	0.063_±0.000	3.67_±00.07 + 70.55_±00.25	2.44_±00.02
Go	0.072_±0.000	3.08_±00.05 + 0.00_±00.00	3.06_±00.04
V/clang	0.095_±0.000	2.14_±00.23 + 211.15_±00.64	3.91_±00.04
V/gcc	0.099_±0.000	1.95_±00.03 + 201.60_±00.17	4.10_±00.03
Rust	0.101_±0.000	2.09_±00.11 + 74.71_±00.77	3.98_±00.05
Java	0.130_±0.004	39.84_±00.17 + 123.02_±05.25	7.33_±00.26
Crystal	0.138_±0.000	3.71_±00.02 + 88.43_±00.00	5.58_±00.05
Node.js	0.218_±0.002	42.17_±00.00 + 147.09_±02.74	10.79_±00.07
Scala	0.228_±0.003	72.13_±00.08 + 230.29_±00.88	13.20_±00.22
Nim/clang	0.278_±0.000	2.10_±00.10 + 587.68_±01.03	10.75_±00.04
Nim/gcc	0.287_±0.000	1.90_±00.02 + 615.91_±00.00	10.83_±00.07
Lua/luajit	0.298_±0.001	1.33_±00.02 + 156.66_±00.63	12.00_±00.08
Python/pypy	0.616_±0.003	58.72_±00.13 + 249.30_±00.15	25.12_±00.16
Julia	0.651_±0.001	271.49_±00.14 + 370.21_±01.14	25.06_±00.21
Racket	0.752_±0.002	109.49_±00.82 + 247.11_±00.45	29.59_±00.17
Ruby/truffleruby	0.890_±0.012	203.57_±01.55 + 800.19_±11.11	60.49_±00.76
Lua	1.142_±0.006	2.69_±00.04 + 283.50_±01.03	46.78_±00.35
Ruby (--jit)	1.187_±0.002	22.98_±00.02 + 163.71_±00.14	47.95_±00.43
Ruby/truffleruby (JVM)	1.378_±0.088	376.75_±09.21 + 495.42_±42.25	89.49_±05.72
Ruby	1.860_±0.003	11.47_±00.07 + 172.19_±00.35	76.75_±00.60
Ruby/jruby	2.314_±0.075	197.13_±08.38 + 521.12_±52.24	124.24_±05.36
Python	2.530_±0.014	10.84_±00.04 + 181.69_±01.57	107.30_±01.03

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux bookworm/sid

Language	Version
.NET Core	8.0.300
C#/.NET Core	4.10.0-3.24216.12 (3af0081a)
C#/Mono	6.12.0.200
Chez Scheme	9.5.8
Clojure	"1.11.3"
Crystal	1.12.1
D/dmd	v2.108.1
D/gdc	13.2.0
D/ldc2	1.38.0
Elixir	1.14.0
F#/.NET Core	12.8.300.0 for F# 8.0
Go	go1.22.3
Go/gccgo	13.2.0
Haskell	9.8.2
Idris 2	0.6.0
Java	22.0.1
Julia	v"1.10.3"
Kotlin	2.0.0
Lua	5.4.6
Lua/luajit	2.1.1710398010
MLton	20210117
Nim	2.0.4
Node.js	v22.2.0
OCaml	5.2.0
PHP	8.2.18
Perl	v5.38.2
Python	3.11.9
Python/pypy	7.3.16-final0 for Python 3.10.14
Racket	"8.13"
Ruby	3.3.1p55
Ruby/jruby	9.4.7.0
Ruby/truffleruby	24.0.1
Rust	1.78.0
Scala	3.4.2
Swift	5.10
Tcl	8.6
V	0.4.6 736067d
Vala	0.56.17
Zig	0.12.0
clang/clang++	16.0.6 (27)
gcc/g++	13.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

versions (print installed language versions);
shell (start the shell);
brainfuck bench (build and run Brainfuck bench.b benchmarks);
brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
base64 (build and run Base64 benchmarks);
json (build and run Json benchmarks);
matmul (build and run Matmul benchmarks);
primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

./run.sh build (build the image);
./run.sh make versions (run the image with the versions command);
sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

NumPy for matmul tests (Debian package python3-numpy).
UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

Boost for JSON tests (Debian package libboost-dev).
JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

Coursier for downloading Maven artifacts.

For Haskell:

network for TCP connectivity between the tests and the test runner.
raw-strings-qq for raw string literals used in tests.

For Perl:

cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

The test runner listens on localhost:9001;
All messages are sent using TCP sockets closed immediately after the message has been sent;
There are two messages sent from a test (it establishes the measurement boundary):
1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
2. The end message with any content (mostly it's "stop" for consistency).
The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

append a new target (the final binary location) into executables variable;
append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

append a new target (the final artifact location) into artifacts variable;
append the proper target rule to compile the test;
append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

append run[<script_file>] into all_runners variable;
append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc