search

renaissance-benchmarks / renaissance

The Renaissance Benchmark Suite
https://renaissance.dev
GNU General Public License v3.0
312 stars 58 forks source link

Remove or increase timeouts #142

Open Fithos opened 5 years ago

Fithos commented 5 years ago

In some benchmarks (particularly those running on Spark), there are timeouts that trigger (e.g., causing a TimeoutException) if a response does not come within a given timeframe. While the timeout won't be triggered during a normal benchmark execution, expensive dynamic analyses performing heavy instrumentation may significantly slowdown the application, triggering the timeouts. As a result, the benchmark will fail prematurely, and the analysis won't be possible for such benchmark.

To support such kinds of analyses, this behavior should be minimized. All benchmarks should be revised to see whether they make use of timeouts and, if possible, remove them or significantly increase them.

FJShen commented 2 years ago

I support this proposal. Is there something technically tricky that keeps Renaissance from realizing this?

lbulej commented 2 years ago

It depends. There are many libraries, so reviewing all of them (and after each update) for potential timeouts is not a feasible approach. The most sensible way to proceed is to report timeouts triggered by expensive analyses and see if the timeout can be a) configured without having to modify the library code, b) safely disabled. Then we have to proceed on case by case basis.