Currently if a test takes longer than runner-timeout (or worse, server-timeout), our infrastructure does Weird Things. This is unavoidable, since at some point in time you need to assume that the test is hung and will never finish.
What we can (and should) do is mark tests as failed when they take more time than a certain threshold below where the infrastructure would explode. I propose to default this to 50% of min(server-timeout, runner-timeout, retry-limit), with the percentage being configurable.
This means that tests that are taking too long become visible before they become an infrastructural problem. Hopefully this means we don't need to keep bumping our timeouts upward as tests silently creep up to the limit. In fact, with this in place, I expect that we can ratchet the timeouts downward again, resulting in an overall more responsive test infrastructure.
asottile
commented
8 years ago
Currently if a test takes longer than runner-timeout (or worse, server-timeout), our infrastructure does Weird Things. This is unavoidable, since at some point in time you need to assume that the test is hung and will never finish.
What we can (and should) do is mark tests as failed when they take more time than a certain threshold below where the infrastructure would explode. I propose to default this to 50% of min(server-timeout, runner-timeout, retry-limit), with the percentage being configurable.
This means that tests that are taking too long become visible before they become an infrastructural problem. Hopefully this means we don't need to keep bumping our timeouts upward as tests silently creep up to the limit. In fact, with this in place, I expect that we can ratchet the timeouts downward again, resulting in an overall more responsive test infrastructure.