search

ecraven / r7rs-benchmarks

Benchmarks for various Scheme implementations. Taken with kind permission from the Larceny project, based on the Gabriel and Gambit benchmarks.
270 stars 32 forks source link

CPU limit penalizes parallel GC #44

Open wingo opened 5 years ago

wingo commented 5 years ago

The CPU limit in the bench script is 300 seconds of CPU time (5 minutes). Compilation has 5 minutes, then running has 5 minutes. But unfortunately this penalizes parallelization, whose goal is usually to minimize wall-clock time rather than CPU time; and the end result of the benchmarking process is wall-clock time. Of course this result is worse the more cores you have.

I think Guile has hit this limit in ctak at run-time, and now that some optimizations allow for better visibility into compile, we hit it there at compile-time as well.

I think the limit should be raised to 15 minutes. It's a decent trade-off between the concern of wanting short benchmark runs and allowing for parallelism in GC.

ecraven commented 1 month ago

Does anyone know whether there's a simple way to limit wall-clock time, not user time?

lockbox commented 1 week ago

could https://linux.die.net/man/1/timeout help here? (probably not going to be entirely as precise as ulimit but may work well enough)