as discussed on Discourse it would be nice to implement a timeout within pyperfomance.
If a benchmark run hangs, pyperformance doesn’t have a mechanism to time out and terminate the underlying process. We’ve had a few cases where a benchmark was hanging literally forever (deadlock) and the job was terminated by the CI system eventually (after many hours!). This is not ideal because:
it’s hard to set an appropriate timeout for the whole process as it might vary from machine to machine
sometimes folks might not have full control of the CI system hence impossible to set a sensible timeout
having an early failure by pyperformance is preferred than having a timeout at process (pyperformance) level. This allow better usage of CI resources.
Requirements:
Ideally the timeout would be applied at benchmark run level and not for the whole pyperformance process allowing a more fine grained control in case of timeout.
If a benchmark times out, it should not fail the whole pyperformance suite but we should mark it as failed (there is already a mechanism to do that).
There should be able to provide an option at cli level to override the default timeout value.
diegorusso
commented
5 days ago
Hello,
as discussed on Discourse it would be nice to implement a timeout within pyperfomance.
If a benchmark run hangs, pyperformance doesn’t have a mechanism to time out and terminate the underlying process. We’ve had a few cases where a benchmark was hanging literally forever (deadlock) and the job was terminated by the CI system eventually (after many hours!). This is not ideal because:
Requirements: