go-fuzz-build with '.'

[andrewchambers]

andrew@andrew-laptop:fuzz $ go-fuzz-build .
failed to execute go build: exit status 1
can't load package: /tmp/go-fuzz-build703235414/src/go-fuzz-main/main.go:5:2: local import "." in non-local package
import cycle not allowed
package go-fuzz-main
    imports .
    imports .

I don't mind if you don't care to support this.

[dvyukov]

Yeah, I have no idea how to fix it. Go-fuzz-build is hacky enough already. Please use full package name.

[andrewchambers]

Maybe put a readme note if there isn't one already. I may have missed it.

[dvyukov]

Local import paths always start with ".", right? I think it's better to provide readable diagnostic in go-fuzz-build if package name starts with ".". What do you think?

[andrewchambers]

Sounds good to me. I am up and running with my own fuzzing now btw.

[dvyukov]

Great! Do you have a good corpus? It is important. Does it utilize all cores? Please copy-paste few log lines here. Want to check sanity.

[andrewchambers]

andrew@andrew-laptop:fuzz $ go-fuzz -bin=./Fuzz-fuzz -corpus=./corpus -workdir=./workdir
2015/04/26 00:26:28 slaves: 4, corpus: 29 (3s ago), crashers: 1, restarts: 1/0, execs: 0 (0/sec), cover: 0.00%, uptime: 3s
2015/04/26 00:26:31 slaves: 4, corpus: 29 (6s ago), crashers: 1, restarts: 1/0, execs: 0 (0/sec), cover: 0.32%, uptime: 6s
2015/04/26 00:26:34 slaves: 4, corpus: 29 (9s ago), crashers: 1, restarts: 1/3571, execs: 14287 (1587/sec), cover: 0.32%, uptime: 9s
2015/04/26 00:26:37 slaves: 4, corpus: 29 (12s ago), crashers: 1, restarts: 1/4727, execs: 18908 (1574/sec), cover: 0.32%, uptime: 12s

Is there a way to see which code the profiler is actually running? something like go tool cover -html=prof.html but for the current "best" mutation, perhaps via a status http server.

[andrewchambers]

My corpus is currently not good. I have code which converts files into a tree data structure based on features of the data, I want to see if the coverage guided mutation can trigger the case where the maximum tree depth is met.

[dvyukov]

Most of the bugs that I've found triggered by relatively simple inputs. I've not yet seen fuzzer generating a tricky malicious input. Eager to hear about your experience. Leave it at least for a night.

[dvyukov]

2015/04/26 00:26:37 slaves: 4, corpus: 29 (12s ago), crashers: 1, restarts: 1/4727, execs: 18908 (1574/sec), cover: 0.32%, uptime: 12s

restarts seems to be OK, the test is not crashing too frequently cover is OK execs/sec depends on execution speed of the test. 1574/sec may or may not be OK. Is CPU fully utilized? On some tests I got 6000/sec/core. But others do time.Sleep and so I had to run with -procs=500 to get good CPU utilization.

[dvyukov]

Is there a way to see which code the profiler is actually running?

No, there is no anything like this atm. There is no "best" input, there is a minimized corpus that collectively covers as much as possible. Do you actually want to look at all them? Why? There is a simple solution. You can write a test function that reads a file from disk and feeds it into Fuzz function, run it on files from workdir/corpus and then use cover tool as usual to check coverage for a single input or for the whole corpus collectively.

[andrewchambers]

Ah, I understand. That is a good solution. Viewing just gives an idea of what sorts of code paths are being taken. In my current case, my function is already randomly tested with full coverage, so I don't think it will find any bugs. My problem was that in order to get full test coverage, I had to artificially modify probabilities to make certain things more likely.

I was curious if using this will allow me to remove the artificial special cases from my test code.

[andrewchambers]

btw:

andrew@andrew-laptop:~ $ top

top - 00:44:38 up 12:31,  5 users,  load average: 2.67, 1.06, 1.31
Tasks: 238 total,   3 running, 235 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.6 us,  1.4 sy, 18.1 ni,  3.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   3861852 total,  3341544 used,   520308 free,   141568 buffers
KiB Swap:  4008956 total,   464388 used,  3544568 free.  1372272 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
20578 andrew    20   0   18652  15708   2132 S  93.9  0.4   0:13.56 Fuzz-fuzz   
20575 andrew    20   0   18652  15804   2100 S  92.5  0.4   0:21.09 Fuzz-fuzz   
20571 andrew    20   0   19744  16820   2140 R  89.2  0.4   0:21.61 Fuzz-fuzz   
20528 andrew    39  19  661088 120372   7372 S  16.9  3.1   0:07.84 go-fuzz     
20643 andrew    20   0   16476  13124   2152 R  12.6  0.3   0:00.38 Fuzz-fuzz   
 3381 andrew    20   0  862292  97980  27120 S   4.6  2.5  68:08.41 chrome     

[andrewchambers]

I think once knowledge of this tool spreads and the tool is easy enough to use (easy after those few fixes), hobbyists will start security fuzzing any code they can get their hands on.

[dvyukov]

top looks reasonable

I was curious if using this will allow me to remove the artificial special cases from my code.

I would suggest to delete delete any of you current code. Generators usually can generate different things that fuzzers can, so they are complementary. I bet that go-fuzz won't be able to generate everything your generator can. That's based on my experience with GoSmith (random Go program generator) and Trinity (linux kernel syscall fuzzer).

Btw, I have an idea to combine fuzzing and generation in symbiosis. You also write a function of the form: func Generate() []byte And then go-fuzz alternates between fuzzing current corpus and asking generator for more inputs. New generated inputs are feed into test function, if they produce new coverage, then are added to fuzzing corpus. As I see it, generator part will allow to cover lots of various correct inputs, while fuzzer part will allows to (1) rigorously test error handling and validation code for all these inputs, (2) generate something that generator cannot possibly generate (developer did not code generation of some aspects).

What do you think?

[andrewchambers]

I like that idea. One example that comes to mind is that most compilers have really bad test suites checking it rejects bad input, instead they check for correct compilation only. I can see how something like that could detect missing validation checks of mostly correct inputs.

Another idea would be to generate a correct program, mutate it slightly, and then compare gccgo with gc to make sure both either accept, or reject the mutated program to ensure consensus and no crashes. This case doesn't rely on coverage info though.

[andrewchambers]

Although with regard to fuzz testing the Go compilers, it is less useful unless it is a security flaw, and not just an edge case. The bugs you found in the image packages could potentially stop denial of service attacks on servers so are really valuable imo.