run_fuzzer and reproduce always use timeout=25

[rlohning]

When I use run_fuzzer or reproduce with an explicit -timeout=60, the fuzzers still get the parameter -timeout=25:

qtrob@ubuntu:~/oss-fuzz$ python infra/helper.py run_fuzzer qt qtbase_corelib_serialization_qcborvalue_fromcbor -timeout=60
(...)
Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e RUN_FUZZER_MODE=interactive -v /home/qtrob/oss-fuzz/build/out/qt:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer qtbase_corelib_serialization_qcborvalue_fromcbor -timeout=60
Using seed corpus: qtbase_corelib_serialization_qcborvalue_fromcbor_seed_corpus.zip
/out/qtbase_corelib_serialization_qcborvalue_fromcbor -rss_limit_mb=2560 -timeout=25 -timeout=60 /tmp/qtbase_corelib_serialization_qcborvalue_fromcbor_corpus < /dev/null
(...)

qtrob@ubuntu:~/oss-fuzz$ python infra/helper.py reproduce qt qtbase_corelib_serialization_qcborvalue_fromcbor /tmp/34953.cbor -timeout=60
(...)
+ run_fuzzer qtbase_corelib_serialization_qcborvalue_fromcbor -runs=100 -timeout=60 /testcase
/out/qtbase_corelib_serialization_qcborvalue_fromcbor -rss_limit_mb=2560 -timeout=25 -runs=100 -timeout=60 /testcase < /dev/null
(...)

My explicitly set timeout seems to be ignored. Even if it wasn't, reading both values would be confusing to the user, at least.

[inferno-chromium]

We use higher timeout values in clusterfuzz now, like 65 sec [https://github.com/google/clusterfuzz/blob/master/src/local/butler/scripts/setup.py#L28], @jonathanmetzman do we need to bump these in oss-fuzz infra scripts - https://github.com/google/oss-fuzz/search?q=timeout%3D25

[inferno-chromium]

@rlohning - as @oliverchang pointed out, in clusterfuzz we explicitly don't allow more than 25 sec timeout during fuzzing as it impacts overall fuzzing efficiency - https://github.com/google/clusterfuzz/blob/481c78b28fb2d9a401a6f3fe302746a8f6110954/src/python/bot/fuzzers/libFuzzer/fuzzer.py#L54. what is the usecase for more than 25 sec timeout ?

[rlohning]

No actual use case I guess. I just played around with the parameters and was surprised by the behavior.

Wouldn't "we explicitly don't allow" mean to exit with an error message when I pass -timeout=60?

[inferno-chromium]

No actual use case I guess. I just played around with the parameters and was surprised by the behavior.

Wouldn't "we explicitly don't allow" mean to exit with an error message when I pass -timeout=60?

yes that would be ideal to exit out, can keep this open for enhancement

[zhangskz]

Isn't this useful for debugging timeout bugs? e.g. the instructions at https://google.github.io/oss-fuzz/advanced-topics/reproducing/#fuzz-target-bugs instruct setting --timeout=65 (which doesn't seem possible when using using python infra/helper.py reproduce ....

[oliverchang]

Hmm, I'm thinking we should revert that --timeout=65 to 25 seconds in the documentation too.

For ClusterFuzz, the reason we have 65 seconds for reproduction and 25 seconds during fuzzing is to not file flaky timeout bugs which take very close to 25 seconds to complete. These timeouts may be flaky to reproduce consistently, especially if different CPUs/machine types run the same testcase.

When we do file them and get a developer to reproduce them however, it seems beneficial to set a lower timeout (i.e. 25 seconds) instead so that these reproduce more easily.

@jonathanmetzman @inferno-chromium WDYT?