search

rust-fuzz / cargo-fuzz

Command line helpers for fuzzing
https://rust-fuzz.github.io/book/cargo-fuzz.html
Apache License 2.0
1.51k stars 107 forks source link

libFuzzer out of memory #270

Open lopopolo opened 3 years ago

lopopolo commented 3 years ago

I run cargo-fuzz in several github actions on a daily cron and invoke it as:

cargo fuzz run encode -- -max_total_time=1800

For the past week or so, I've been getting failures of the following sort:

==3918== ERROR: libFuzzer: out-of-memory (used: 2114Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 24233890 bytes in 183 chunks; quarantined: 28023994 bytes in 206841 chunks; 1768194 other chunks; total chunks: 1975218; showing top 95% (at most 8 unique contexts)
24121507 byte(s) (99%) in 15 allocation(s)
    #0 0x5642e8dcec8d  (/home/runner/work/boba/boba/fuzz/target/x86_64-unknown-linux-gnu/release/encode+0xa8c8d)
    #1 0x7f4e460c5b38  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xaab38)
    #2 0x5642e8d53fb6  (/home/runner/work/boba/boba/fuzz/target/x86_64-unknown-linux-gnu/release/encode+0x2dfb6)
    #3 0x7f4e45cbb0b2  (/lib/x86_64-linux-gnu/libc.so.6+0x270b2)

MS: 5 InsertByte-CrossOver-CopyPart-ChangeBit-InsertByte-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc
0xa,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,0xa0,
\x88\xc7\xc7\x88\x88\x88f\x88E\x88\xc7\xfa\xc7\xc7\xc7\x91\xc7\xc7\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96\x96t\x96\x96\x96\x96\x96\x96\x96\x96\x96\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff
artifact_prefix='/home/runner/work/boba/boba/fuzz/artifacts/encode/'; Test unit written to /home/runner/work/boba/boba/fuzz/artifacts/encode/oom-8fdc30c936d6f750b40fff1eaa9c28451cc174ec
Base64: iIiIiIiIMP////////////////////9dCv///zD/////////////Cv////8AAAAA/+///////////zD/////////////////////XQr///8w/////////////wr//////////////////////////////////////////////////w==
SUMMARY: libFuzzer: out-of-memory

Sample failing run: https://github.com/artichoke/boba/runs/3154113135?check_suite_focus=true

dallenng commented 3 years ago

I have a similar problem in my GitHub actions, and when I fuzz on my computer there is no oom.

In CI, between the run 1 through 600k, the fuzzer behave the same as on my computer with rss going from 30Mb to 600Mb. Then, suddenly the rss value start to increase quickly until it reaches the limit or no more RAM is available, while on my computer, the rss value continues to increase slowly until it converges to some value.

Here is my repo if you want to see my CI config or what changes I tried in this PR.

I got a successful workflow run the first time (7 days ago), but all subsequent runs failed due to fuzz oom.

Sample failing run

nagisa commented 2 years ago

libfuzzer needs the memory to store the knowledge it has gathered about the program. Fuzzing extremely branchy code is prone to using extreme amounts of memory.

The only way I know of to deal with this problem is to reduce the problem size (i.e. fuzz smaller portions of your system) or give it more memory. Improvements to libfuzzer itself are of course a possibility as well, but it isn't maintained by ourselves. You could try different fuzzing implementations too.

dallenng commented 2 years ago

I didn't do a lot of testing but the problem is not happening anymore on my side.