Non-interactive go-fuzz usage

[michael-schaller]

Currently go-fuzz doesn't terminate on its own and so it is harder than necessary to use in a non-interactive context like for an instance in a fuzz script or for continuous integration.

As workaround one can use timeout --signal int <timeout in seconds> go-fuzz ... to limit the overall runtime of go-fuzz and then afterwards check if the crashes directory has new content. It would be better though if go-fuzz would support this use case as it could be smarter about it.

How about adding these features to go-fuzz? 1) Add an auto-stop command line option to stop once the coverage hasn't increased for a configurable amount of seconds. 2) Add a deadline command line option to stop after a fixed time of seconds. This command should exit with a dedicated non-zero exit code to be able to check for this condition. 3) Exit with a dedicated non-zero exit code in case new crashes have been encountered during this run.

Both command line options shouldn't have default values as the value depends on the complexity of the code to fuzz, the go-fuzz settings (number of workers, ...) and the performance of the machine this runs on.

[michael-schaller]

The previous issue #166 is related.

[dvyukov]

Hi Michael,

As pointed out in #166 at this point it does not make sense to add something to go-fuzz that won't be accepted as part of implementation in the standard library. Which in particular means interfaces need to be simple, not too bloated and do limited set of things good.

Integration with various CI systems should probably be handled outside of the core system. There are too many CI systems, and one also wants to copy in/out corpus, notify CI about new crashes, etc. In the end, something needs to capture actual crashes/inputs too (if you take travis-ci, it gives you a temp VM, all local artifacts will be discarded after a run). I think this should be handled by some intermediate program which will interact with go-fuzz and with various CI systems. Such system will allow to have simpler go-fuzz interfaces.

Now the question is: if we don't stuff everything into go-fuzz, what should be minimal go-fuzz interface that will allow implementation of different testing/integration strategies on top of go-fuzz? It seems to me that the current go-fuzz interface is actually enough. Runner can always kill it and it probably wants to parse crashes dir anyway (to say what crashes exactly happened and to save them somewhere). If you write such proxy, I will be happy to reference it from go-fuzz docs as a recommended solution for integration with particular CI system.

[michael-schaller]

That would indeed work if go-fuzz would have stable output so that the caller can detect that the coverage hasn't increased within a certain time. Maybe go-fuzz should have a machine readable output option where it prints the status updates in a machine readable format like for an instance JSON.

[dvyukov]

There is already something like this here: https://github.com/dvyukov/go-fuzz/blob/master/go-fuzz/coordinator.go#L76 This was contributed by somebody else and I never used it. The interfaces were added somewhat chaotically over time.

[everestmz]

I was having some similar ideas and came across this issue - what would you think about implementing something like afl's AFL_EXIT_WHEN_DONE env var?

It kills AFL when there are no more pending paths and no crashes have been found in 100 cycles. I couild see it being really useful for non-interactive use cases.

(Implementation here: https://github.com/mcarpenter/afl/blob/ec1e27672ed7f625452c9cf6c6d6774c4b36e61f/afl-fuzz.c#L3954)

What are your thoughts on adding an option like this?