search

dobin / ffw

A fuzzing framework for network servers
GNU General Public License v3.0
116 stars 24 forks source link

Restart honggfuzz on crash #20

Closed dobin closed 6 years ago

dobin commented 6 years ago

If the communication with honggfuzz drops, it should be restarted.

Reference: https://github.com/dobin/ffw/issues/11

dobin commented 6 years ago

Could reproduce. Honggfuzz seem to crash/exit after like 3 minutes of fuzzing.

FFW: 

DEBUG:root:SOCKET: Try to recv
DEBUG:root:SOCKET:   Recieved: Fuzz
DEBUG:root:Fuzzing the data
DEBUG:root:selected input: 0  from: cli  len: 19
INFO:root:Call fuzzer, seed: 3438846681769109650
DEBUG:root:CMD: -s 3438846681769109650 -o /Development/ffw-examples/mongoose-6.9/temp/3438846681769109650.out.raw "/Development/ffw-examples/mongoose-6.9/temp/3438846681769109650.in.raw"
DEBUG:root:Read fuzzing data: MQTT��<dummy
INFO:root:Open connection on localhost:1883
INFO:root:Send data: 
DEBUG:root:  Sending message: 0
INFO:root:ReceiveData err on msg 1: timed out
DEBUG:root:SOCKET: Send: okay
DEBUG:root:A fuzzing loop...
DEBUG:root:SOCKET: Try to recv
DEBUG:root:SOCKET:   Recieved: 
INFO:root:Hongfuzz quit, exiting too

Honggfuzz: 

No input file corpus loaded, the external socket_fuzzer is responsible for creating the fuzz data
[2018-04-04T12:15:26+0000][D][14292] arch_archInit():421 Glibc version:'2.26', OK
[2018-04-04T12:15:26+0000][D][14292] files_readFileToBufMax():61 Read '2' bytes from '/sys/bus/event_source/devices/intel_pt/type'
[2018-04-04T12:15:26+0000][D][14292] arch_perfInit():376 perfIntelPtPerfType = 7
...
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():487 ------[ 1: subproc_run
[2018-04-04T12:15:28+0000][D][14293] arch_checkWait():301 PID '14307' returned with status: SIGNALED, signal: 6 (Aborted)
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParseRaw():332 sancov map file not found
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParse():582 Target sancov file not found
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():495 ------[ 2: fetch input
[2018-04-04T12:15:28+0000][D][14293] fuzz_prepareSocketFuzzer():56 fuzz_prepareSocketFuzzer: SEND Fuzz
[2018-04-04T12:15:28+0000][D][14293] fuzz_waitforSocketFuzzer():78 fuzz_waitforSocketFuzzer: RECV: okay
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():504 ------[ 3: feedback
[2018-04-04T12:15:28+0000][D][14293] fuzz_perfFeedback():189 New file size: 0, Perf feedback new/cur (instr,branch): 0/0/0/0, BBcnt new/total: 0/0
[2018-04-04T12:15:28+0000][D][14293] fuzz_sanCovFeedback():249 File size (Best/New): 0, SanCov feedback (bb,dso): Best: [0,0] / New: [0,0], newBBs:0
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():525 ------[ 4: reap child
[2018-04-04T12:15:28+0000][D][14293] arch_checkWait():301 PID '14302' returned with status: STOPPED with signal: 5 (Trace/breakpoint trap)
[2018-04-04T12:15:28+0000][D][14293] arch_traceEvent():1164 PID: 14302, Ptrace event: 6
[2018-04-04T12:15:28+0000][D][14293] arch_traceEvent():1181 PID: 14302 terminated with signal: 6
[2018-04-04T12:15:28+0000][D][14293] arch_checkWait():301 PID '14309' returned with status: STOPPED with signal: 5 (Trace/breakpoint trap)
[2018-04-04T12:15:28+0000][D][14293] arch_traceEvent():1164 PID: 14309, Ptrace event: 6
[2018-04-04T12:15:28+0000][D][14293] arch_traceEvent():1181 PID: 14309 terminated with signal: 6
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParseRaw():332 sancov map file not found
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParse():582 Target sancov file not found
------------------------------------------------------
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():487 ------[ 1: subproc_run
[2018-04-04T12:15:28+0000][D][14293] arch_checkWait():301 PID '14302' returned with status: SIGNALED, signal: 6 (Aborted)
[2018-04-04T12:15:28+0000][D][14293] arch_checkWait():301 PID '14309' returned with status: SIGNALED, signal: 6 (Aborted)
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParseRaw():332 sancov map file not found
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParse():582 Target sancov file not found
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():495 ------[ 2: fetch input
[2018-04-04T12:15:28+0000][D][14293] fuzz_prepareSocketFuzzer():56 fuzz_prepareSocketFuzzer: SEND Fuzz
[2018-04-04T12:15:28+0000][D][14293] fuzz_waitforSocketFuzzer():78 fuzz_waitforSocketFuzzer: RECV: okay
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():504 ------[ 3: feedback
[2018-04-04T12:15:28+0000][D][14293] fuzz_perfFeedback():189 New file size: 0, Perf feedback new/cur (instr,branch): 0/0/0/0, BBcnt new/total: 0/0
[2018-04-04T12:15:28+0000][D][14293] fuzz_sanCovFeedback():249 File size (Best/New): 0, SanCov feedback (bb,dso): Best: [0,0] / New: [0,0], newBBs:0
[2018-04-04T12:15:28+0000][D][14293] fuzz_fuzzLoopSocket():525 ------[ 4: reap child
[2018-04-04T12:15:28+0000][D][14293] arch_checkWait():301 PID '14314' returned with status: STOPPED with signal: 5 (Trace/breakpoint trap)
[2018-04-04T12:15:28+0000][D][14293] arch_traceEvent():1164 PID: 14314, Ptrace event: 6
[2018-04-04T12:15:28+0000][D][14293] arch_traceEvent():1181 PID: 14314 terminated with signal: 6
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParseRaw():332 sancov map file not found
[2018-04-04T12:15:28+0000][D][14293] sancov_sanCovParse():582 Target sancov file not found
------------------------------------------------------

Is it a timeout of 1min?

dobin commented 6 years ago

No, next run went 7m. 

[2018-04-04T12:38:29+0000][D][24625] arch_traceEvent():1164 PID: 24293, Ptrace event: 6
[2018-04-04T12:38:29+0000][D][24625] arch_traceEvent():1181 PID: 24293 terminated with signal: 6
[2018-04-04T12:38:29+0000][W][24625] arch_reapChild():366 Un-handled ASan report due to compiler-rt internal error - retry with './HF.sanitizer.log.24390'
Ok, that's interesting, saved new crash as './SAN.PC.0.STACK.0.CODE.UNKNOWN.ADDR.(nil).INSTR.[UNKNOWN].2018-04-04.12:38:29.fuzz'
dobin commented 6 years ago 
cat bin/HF.sanitizer.log.24293 
==24293==LeakSanitizer has encountered a fatal error.
==24293==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==24293==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
root@fuzzing:/Development/ffw-examples/mongoose-6.9# grep fatal bin/HF*
bin/HF.sanitizer.log.12916:==12916==LeakSanitizer has encountered a fatal error.
bin/HF.sanitizer.log.1771:==1771==LeakSanitizer has encountered a fatal error.
bin/HF.sanitizer.log.24293:==24293==LeakSanitizer has encountered a fatal error.
bin/HF.sanitizer.log.6971:==6971==LeakSanitizer has encountered a fatal error.
bin/HF.sanitizer.log.8382:==8382==LeakSanitizer has encountered a fatal error.
bin/HF.sanitizer.log.9495:==9495==LeakSanitizer has encountered a fatal error.
grep: bin/HF_SANCOV: Is a directory
dobin commented 6 years ago

Implemented the restart. But i'm pretty sure that the LeakSanitizer is the main reason. Sadly i seem to be unable to disable it, e.g. via environment variable, like implemented in https://github.com/dobin/ffw/commit/a0612ad447901f221a1cf0975cdc716f6430884a.

I think it kills some bugs - if process crashes (?), LSan will try to do its thing:

LSan lies dormant until the very end of the process, at which point there is an extra leak detection phase.

It is also enabled by default:

LeakSanitizer is enabled by default in ASan builds of x86_64 Linux