Running multiple benchmarks on the online-instance

[HamidJalali]

Is it possible to run multiple benchmarks on the online instance in parallel? Or do one user has to wait until the benchmark which is currently in work to be finished? Thank you in advance.

[MichaelRoeder]

The experiments are carried out sequentially to make sure that the results are comparable. Otherwise, one system / benchmark could influence the result (especially runtime performance ) of parallel benchmarks / systems running in parallel.

[HamidJalali]

So the only way to achieve parallel benchmarking would be to have multiple independent instances of HOBBIT, correct?

[MichaelRoeder]

Yes. The platforms should even be deployed on different servers to make sure that the single benchmarks are not influencing each other (e.g., if both are stressing the same hardware resource).