Build error in benchmarking

[carun]

## D-Templated-Build-dmd:
- Build took 0.875 seconds (using "/usr/bin/dmd" version v2.088.0)
## D-Templated-Build-ldmd2:
output: b'(6): Error: only one `main` function allowed\n'
- Build took 0.970 seconds (using "/opt/dev-setup/ldc2-1.17.0-linux-x86_64/bin/ldmd2" version 1.17.0)
## D-Templated-Build-gdc:
- Build took 8.017 seconds (using "/usr/bin/gdc" version 10.0.1)

## Java-Untemplated-Build-/usr/bin/javac:
javac 11.0.8
Traceback (most recent call last):
  File "./benchmark", line 1260, in <module>
    main()
  File "./benchmark", line 260, in main
    results += benchmark_Java(execs=execs, durs=durs, gpaths=gpaths, args=args, op='Build', templated=False)
  File "./benchmark", line 670, in benchmark_Java
    version = sp.run([exe, '-version'], stderr=sp.PIPE).stderr.decode('utf-8').split()[1]
IndexError: list index out of range

[nordlow]

What is the output of the command javac -version on your system?

[carun]

javac 11.0.8

[nordlow]

Can you try again now with master and tell me what the output of the command

./benchmark --languages=Java --function-count=2 --function-depth=2 --run-count=1

is?

[carun]

2020-08-31 13:05:22 ~/code/d/compiler-benchmark (master)
$ ./benchmark --languages=Java --function-count=2 --function-depth=2 --run-count=1
# Code-generation:
- Generating generated/java/main.java took 0.015 seconds (Java)
- Generating generated/java/main_t.java took 0.000 seconds (Java)

## Java-Untemplated-Build-/usr/bin/javac:
javac 11.0.8
Traceback (most recent call last):
  File "./benchmark", line 1260, in <module>
    main()
  File "./benchmark", line 260, in main
    results += benchmark_Java(execs=execs, durs=durs, gpaths=gpaths, args=args, op='Build', templated=False)
  File "./benchmark", line 670, in benchmark_Java
    version = sp.run([exe, '-version'], stderr=sp.PIPE).stderr.decode('utf-8').split()[1]
IndexError: list index out of range
2020-08-31 13:05:32 ~/code/d/compiler-benchmark (master)
$

[nordlow]

You can't be on Git master.

This is the contents of the last version:

https://github.com/nordlow/compiler-benchmark/blob/master/benchmark

The call to sp.run is on line 673, not 670 as your version says.

[carun]

May be worth documenting it in README or something.

[nordlow]

May be worth documenting it in README or something.

I mean things shall work if you are on git master.

[carun]

OK, I pulled origin and here is the latest.

2020-09-04 12:34:19 ~/code/d/compiler-benchmark (master)
$ git rev-parse --short HEAD
8f9d5ba
2020-09-04 12:35:53 ~/code/d/compiler-benchmark (master)
$ ./benchmark --languages=Java --function-count=2 --function-depth=2 --run-count=1
# Code-generation:
- Generating generated/java/main.java took 0.031 seconds (Java)
- Generating generated/java/main_t.java took 0.001 seconds (Java)

# Benchmark:
javac 11.0.8
WARNING: Failed to decode result b'' of command ['/usr/bin/javac', '-version'], defaulting version of Java to `none`
## Java-Untemplated-Build-/usr/bin/javac:
- Build took 1.106 seconds (using "/usr/bin/javac" version unknown)

| Lang-uage | Oper-ation | Temp-lated | Time [us/#fn] | Slowdown vs [Best] | Version | Exec |
| :---: | :---: | --- | :---: | :---: | :---: | :---: |
| Java | Build | No | 276462.5 | 1.0 [Java] | unknown | `javac` |

2020-09-04 12:36:11 ~/code/d/compiler-benchmark (master)
$

[nordlow]

Ok, got it. The benchmark will work nicely for you. You just will get the version unknown in the table output.

Regarding the warning still occurring: My guess is that the command javac -version outputs to stdout on Windows and to stderr on Linux. That's the only reasonable explanation I have. What OS are you running on? I've updated master with even more verbose diagnostics. Please do a git pull and rerun your benchmark.

[carun]

2020-09-08 09:07:46 ~/code/d/compiler-benchmark (master)
$ git pull
remote: Enumerating objects: 54, done.
remote: Counting objects: 100% (54/54), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 51 (delta 34), reused 41 (delta 24), pack-reused 0
Unpacking objects: 100% (51/51), 11.22 KiB | 52.00 KiB/s, done.
From https://github.com/nordlow/compiler-benchmark
   8f9d5ba..e2287ee  master     -> origin/master
Updating 8f9d5ba..e2287ee
Fast-forward
 README.md | 135 +++++++++++++++++++++++++++++++++++++++++++++--------------------------------------------
 benchmark | 296 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------------------------------------------
 2 files changed, 263 insertions(+), 168 deletions(-)
Current branch master is up to date.
2020-09-08 13:49:04 ~/code/d/compiler-benchmark (master)
2020-09-08 13:49:31 ~/code/d/compiler-benchmark (master)
$ ./benchmark --languages=Java --function-count=2 --function-depth=2 --run-count=1
# Code-generation:
- Generating generated/java/main.java took 0.043 seconds (Java)
- Generating generated/java/main_t.java took 0.000 seconds (Java)

# Benchmark:
## Java-Untemplated-Build-/usr/bin/javac:
- Build took 1.508 seconds (using "/usr/bin/javac" version 11.0.8)

| Lang-uage | Oper-ation | Temp-lated | Op Time [us/#fn] | Slowdown vs [Best] | Run Time [us/#fn] | Version | Exec |
| :---: | :---: | --- | :---: | :---: | :---: | :---: | :---: |
| Java | Build | No | 377093.2 | 1.0 [Java] | 36907534 | 11.0.8 | `javac` |

2020-09-08 13:49:38 ~/code/d/compiler-benchmark (master)
$ uname -a
Linux AIM-ES 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
2020-09-08 13:49:45 ~/code/d/compiler-benchmark (master)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal
2020-09-08 13:49:48 ~/code/d/compiler-benchmark (master)
$

[nordlow]

LGTM. :)

[carun]

Thanks for the help.

[carun]

FWIW

2020-09-08 14:29:19 ~/code/d/compiler-benchmark (master)
$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   40 bits physical, 48 bits virtual
CPU(s):                          16
On-line CPU(s) list:             0-15
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       8
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           63
Model name:                      Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
Stepping:                        0
CPU MHz:                         2095.078
BogoMIPS:                        4190.15
Hypervisor vendor:               VMware
Virtualization type:             full
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        8 MiB
L3 cache:                        176 MiB
NUMA node0 CPU(s):               0-15
Vulnerability Itlb multihit:     KVM: Vulnerable
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl tsc_reliabl
                                 e nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm pti arat
2020-09-08 14:29:33 ~/code/d/compiler-benchmark (master)
$

Lang-uage	Oper-ation	Temp-lated	Op Time [us/#fn]	Slowdown vs [Best]	Run Time [us/#fn]	Version	Exec
D	Check	No	50.0	1.3 [D]	N/A	v2.088.0	dmd
D	Check	No	50.9	1.3 [D]	N/A	1.17.0	ldmd2
D	Check	No	128.6	3.3 [D]	N/A	10.0.1	gdc
D	Check	Yes	64.0	1.7 [D]	N/A	v2.088.0	dmd
D	Check	Yes	53.0	1.4 [D]	N/A	1.17.0	ldmd2
D	Check	Yes	38.6	1.0 [D]	N/A	10.0.1	gdc
D	Build	No	328.2	2.0 [D]	432	v2.088.0	dmd
D	Build	No	513.0	3.1 [D]	501	1.17.0	ldmd2
D	Build	No	839.2	5.0 [D]	22603	10.0.1	gdc
D	Build	Yes	166.6	1.0 [D]	330	v2.088.0	dmd
D	Build	Yes	343.2	2.1 [D]	302	1.17.0	ldmd2
D	Build	Yes	837.2	5.0 [D]	756	10.0.1	gdc
C	Check	No	78.9	2.0 [D]	N/A	5.5.0	gcc-5
C	Check	No	289.2	7.5 [D]	N/A	7.5.0	gcc-7
C	Check	No	134.3	3.5 [D]	N/A	8.4.0	gcc-8
C	Check	No	81.1	2.1 [D]	N/A	9.3.0	gcc-9
C	Check	No	69.5	1.8 [D]	N/A	10	gcc-10
C	Check	No	105.0	2.7 [D]	N/A	10.0.1	clang-10
C	Build	No	738.3	4.4 [D]	333	5.5.0	gcc-5
C	Build	No	644.7	3.9 [D]	256	7.5.0	gcc-7
C	Build	No	640.4	3.8 [D]	257	8.4.0	gcc-8
C	Build	No	722.9	4.3 [D]	251	9.3.0	gcc-9
C	Build	No	782.3	4.7 [D]	357	10	gcc-10
C	Build	No	440.4	2.6 [D]	351	10.0.1	clang-10
C++	Check	No	95.8	2.5 [D]	N/A	9.3.0	g++-9
C++	Check	No	103.1	2.7 [D]	N/A	10	g++-10
C++	Check	No	128.5	3.3 [D]	N/A	10.0.0	clang++-10
C++	Check	Yes	116.3	3.0 [D]	N/A	9.3.0	g++-9
C++	Check	Yes	120.5	3.1 [D]	N/A	10	g++-10
C++	Check	Yes	110.3	2.9 [D]	N/A	10.0.0	clang++-10
C++	Build	No	1006.5	6.0 [D]	16749	9.3.0	g++-9
C++	Build	No	884.9	5.3 [D]	333	10	g++-10
C++	Build	No	459.2	2.8 [D]	526	10.0.0	clang++-10
C++	Build	Yes	921.5	5.5 [D]	232	9.3.0	g++-9
C++	Build	Yes	932.6	5.6 [D]	260	10	g++-10
C++	Build	Yes	456.0	2.7 [D]	356	10.0.0	clang++-10
Go	Build	No	515.4	3.1 [D]	375	1.13.8	go
Rust	Check	No	1072.6	27.8 [D]	N/A	1.47.0-nightly	rustc
Rust	Check	Yes	716.9	18.6 [D]	N/A	1.47.0-nightly	rustc
Rust	Build	No	1965.8	11.8 [D]	529	1.46.0	rustc
Rust	Build	No	1580.7	9.5 [D]	520	1.47.0-nightly	rustc
Rust	Build	Yes	1245.4	7.5 [D]	1365	1.46.0	rustc
Rust	Build	Yes	1701.6	10.2 [D]	450	1.47.0-nightly	rustc
Java	Build	No	318.5	1.9 [D]	16019	11.0.8	javac