search

goblint / bench

The benchmark suite
4 stars 5 forks source link

CPU Spec as benchmarks? #49

Open michael-schwarz opened 1 year ago

michael-schwarz commented 1 year ago

I was talking to one of the authors of an OOPSLA paper on alias analysis in the context of compiler optimizations today https://dl.acm.org/doi/10.1145/3563316 (fany title, btw) and asked her how she evaluates her things:

They apparently use the CPU Spec 2017 benchmark set that has several C programs with Loc up to >1M. I know we at some point bought Spec 2006 or something, is there a specific reason we are not using these benchmarks these days?

sim642 commented 1 year ago

They are proprietary for one.

Also if you look at the benchmark list, those > 1M LoC programs are just GCC and Blender.

michael-schwarz commented 1 year ago

Both valid points. One could probably even include the tests in supplementary artefacts for the reviewers and then publish one without them. If this is not a dealbreaker at OOPSLA, it likely also won't be in other venues.

On the other hand, having GCC and blender combined into a single file makes it infinitely more easy to analyze. Might well be worth the 50 bucks if you estimate how much time one of us fixing build scripts for hours on end costs us.

sim642 commented 1 year ago

On the other hand, having GCC and blender combined into a single file makes it infinitely more easy to analyze

Are they combined though? The SPEC 2006 programs in this repo have still been combined by us using CIL into _comb.c files.