Closed Caesurus closed 1 year ago
Minor update. It looks like I don't need to provide the bin file at all for coverage to be consumed. But I've run into a problem.
When trying to do coverage across multiple memory segments, the coverage blocks found only show coverage in one specific section. So for instance, if I generate covbps
for libc, and those get hit first, the coverage for other memory regions aren't showing up.
If I get time I'll try to come up with a concrete example of this. Just curious if this is by design or whether it's a bug of some sort.
Thanks
Very cool! Glad that snapchange is working out for you.
Yeah, the .bin
file is mostly for automated symbol gathering for debugging. Only the .covbps
file(s) is needed for coverage.
Interesting case. I haven't come across this specifically, but there could be a few things we can try to attempt to nail down what's going on.
When you have your covbps
for libc
, can you do a single step trace (cargo run -r -- trace <INPUT_FILE>
)? Does the coverage point that you are expecting get executed in the single step trace?
Likewise, as a check to see if the coverage breakpoints are being set at all, in your set_input
function, we could dump a few specific coverage breakpoints that you are expecting and see if the breakpoint has been written.
Something like this:
impl Fuzzer for Example1Fuzzer {
fn set_input(&mut self, input: &Self::Input, fuzzvm: &mut FuzzVm<Self>) -> Result<()> {
for addr in [/* fill with expected coverage points that aren't showing up */] {
fuzzvm.hexdump(VirtAddr(addr), fuzzvm.cr3(), 0x4)?;
}
}
}
And ensure the first byte at each of those coverage points is a 0xcc
Thanks so much for the response. That's great debug information. I'm hoping to have spare cycles to investigate further early next week.
Ok, so bear with me... I've tried to reproduce the problem. And it's not conventional code, so I could totally be doing something wrong. I modified the example1.c
code in an attempt to simplify the usecase.
// Example test case for snapshot fuzzing
//
// Test the ability to write arbitrary memory and registers into a snapshot
//
// clang -ggdb -O0 example1.c -o example1
#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <string.h>
#include <asm/unistd.h>
pid_t local_getpid() {
pid_t pid;
asm (
"movl $39, %%eax\n" // syscall number for getpid
"syscall\n"
"movl %%eax, %0\n"
: "=r"(pid)
:
: "rax"
);
return pid;
}
void fuzzme(char* data) {
int pid = local_getpid();
// Correct solution: data == "fuzzmetosolveme!", pid == 0xdeadbeef
if (data[0] == 'f') {
if (data[1] == 'u') {
if (data[2] == 'z') {
if (data[3] == 'z') {
if (data[4] == 'm') {
if (data[5] == 'e') {
if (data[6] == 't') {
if (data[7] == 'o') {
if (data[8] == 's') {
if (data[9] == 'o') {
if (data[10] == 'l') {
if (data[11] == 'v') {
if (data[12] == 'e') {
if (data[13] == 'm') {
if (data[14] == 'e') {
if (data[15] == '!') {
pid = local_getpid();
if (pid == 0xdeadbeef) {
*(int*)0xcafecafe = 0x41414141;
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
return;
}
int main() {
char* data = "aaaaaaaaaaaaaaaa";
size_t size = 4096;
void *ptr = mmap(NULL, size, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (ptr == MAP_FAILED) {
return 1;
}
int offset = (char*)fuzzme - (char*)local_getpid;
printf("offset = 0x%x - 0x%x = 0x%x\n", (char*)fuzzme, (char*)local_getpid, offset);
memcpy(ptr, local_getpid, size);
if (mprotect(ptr, size, PROT_READ | PROT_EXEC) == -1) {
return 1;
}
void (*func_ptr)() = ptr+offset;
if(getenv("SNAPSHOT") != 0) {
// Print the memory buffer address and pid address for fuzzing
printf("SNAPSHOT Data buffer: %p\n", data);
// Ensure the stdout has been flushed
fflush(stdout);
// Snapshot taken here
__asm("int3 ; vmcall");
}
// Call the fuzzme function
func_ptr(data);
//fuzzme(data);
munmap(ptr, size);
return 0;
}
As you can see, I'm copying the two functions into a freshly mmap'd section and making sure it's executable. Then calling the copy of fuzzme
that exists in that new section. I had to rewrite getpid so I didn't have to resolve symbols to libc etc... but the functionality should be the same.
I added the following function to gdbsnapshot.py
:
def dumpmem():
pid = gdb.selected_inferior().pid
maps_path = f"/proc/{pid}/maps"
try:
with open(maps_path, "r") as maps_file:
for line in maps_file:
print(line)
parts = line.strip().split()
addr_range, permissions = parts[0], parts[1]
# Check if the section has r-x permissions
if 'x' in permissions and 'r' in permissions:
start_addr, end_addr = [
int(x, 16) for x in addr_range.split('-')
]
filename = f"/tmp/dump_{start_addr:x}.dmp"
print(
f"Dumping section {start_addr:x}-{end_addr:x} to {filename}"
)
# Read the section memory and write to file
with open(filename, "wb") as file:
mem = gdb.selected_inferior().read_memory(
start_addr, end_addr - start_addr)
file.write(mem)
print("Dumping completed.")
except Exception as e:
print(f"An error occurred: {str(e)}")
print(
"Make sure you have the necessary permissions to read the memory maps."
)
And then the line to call the function...
And then obviously the necessary code to copy those *.dmp files out of the image in the utils/extract.sh
script... This gives me the executable memory regions. In my snapshot, I have the following vmmap:
Start Addr End Addr Size Offset Perms objfile
0x400000 0x401000 0x1000 0x0 r--p /root/example1
0x401000 0x402000 0x1000 0x1000 r-xp /root/example1
0x402000 0x403000 0x1000 0x2000 r--p /root/example1
0x403000 0x404000 0x1000 0x2000 r--p /root/example1
0x404000 0x405000 0x1000 0x3000 rw-p /root/example1
0x405000 0x426000 0x21000 0x0 rw-p [heap]
0x7ffff7ddb000 0x7ffff7dde000 0x3000 0x0 rw-p
0x7ffff7dde000 0x7ffff7e04000 0x26000 0x0 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7e04000 0x7ffff7f59000 0x155000 0x26000 r-xp /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7f59000 0x7ffff7fac000 0x53000 0x17b000 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7fac000 0x7ffff7fb0000 0x4000 0x1ce000 r--p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7fb0000 0x7ffff7fb2000 0x2000 0x1d2000 rw-p /usr/lib/x86_64-linux-gnu/libc.so.6
0x7ffff7fb2000 0x7ffff7fbf000 0xd000 0x0 rw-p
0x7ffff7fc2000 0x7ffff7fc3000 0x1000 0x0 r-xp
0x7ffff7fc3000 0x7ffff7fc5000 0x2000 0x0 rw-p
0x7ffff7fc5000 0x7ffff7fc9000 0x4000 0x0 r--p [vvar]
0x7ffff7fc9000 0x7ffff7fcb000 0x2000 0x0 r-xp [vdso]
0x7ffff7fcb000 0x7ffff7fcc000 0x1000 0x0 r--p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7fcc000 0x7ffff7ff1000 0x25000 0x1000 r-xp /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7ff1000 0x7ffff7ffb000 0xa000 0x26000 r--p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7ffb000 0x7ffff7ffd000 0x2000 0x30000 r--p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffff7ffd000 0x7ffff7fff000 0x2000 0x32000 rw-p /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
0x7ffffffde000 0x7ffffffff000 0x21000 0x0 rw-p [stack]
0xffffffffff600000 0xffffffffff601000 0x1000 0x0 --xp [vsyscall]
The mmap'd region is:
0x7ffff7fc2000 0x7ffff7fc3000 0x1000 0x0 r-xp
Now I load that into binja, with some minor fixes to the bn_snapchange.py
:
https://github.com/awslabs/snapchange/blob/main/coverage_scripts/bn_snapchange.py#L696-L704
these use rebase
, which are no longer parameters in the base SnapchangeTask
class... (I may look at opening a PR).
Anyway... I open the memory dump with the correct base addresse 0x7ffff7fc2000
, and architecture x86_64
, and platform linux-x86_64
. Then generate a covbps file named dump_7ffff7fc2000.covbps
that looks like this:
0x7ffff7fc2000
0x7ffff7fc2020
0x7ffff7fc21b3
0x7ffff7fc2044
0x7ffff7fc21ae
0x7ffff7fc2055
0x7ffff7fc21a9
0x7ffff7fc2066
0x7ffff7fc21a4
0x7ffff7fc2077
0x7ffff7fc219f
0x7ffff7fc2088
0x7ffff7fc219a
0x7ffff7fc2099
0x7ffff7fc2195
0x7ffff7fc20aa
0x7ffff7fc2190
0x7ffff7fc20bb
0x7ffff7fc218b
0x7ffff7fc20cc
0x7ffff7fc2186
0x7ffff7fc20dd
0x7ffff7fc2181
0x7ffff7fc20ee
0x7ffff7fc217c
0x7ffff7fc20ff
0x7ffff7fc2177
0x7ffff7fc2110
0x7ffff7fc2172
0x7ffff7fc2121
0x7ffff7fc216d
0x7ffff7fc2132
0x7ffff7fc2168
0x7ffff7fc2143
0x7ffff7fc2163
0x7ffff7fc2158
0x7ffff7fc23b4
0x7ffff7fc2edf
0x7ffff7fc2f07
Ok, so that file is in the snapshot
directory. If that is the only covbps file in the directory and I run the fuzzer, I don't get any coverage blocks hit at all.
When I do a trace, I see these addresses are in fact hit (selectively grep'd so that it's not too much output):
➜ snapshot grep 7ffff7fc traces/f467b5ff9c99cd88_trace|grep INSTRUC
INSTRUCTION 006 0x00007ffff7fc2020 0x107488000 | libc.so.6!__libc_initial+0x30d2
INSTRUCTION 007 0x00007ffff7fc2021 0x107488000 | libc.so.6!__libc_initial+0x30d3
INSTRUCTION 008 0x00007ffff7fc2024 0x107488000 | libc.so.6!__libc_initial+0x30d6
INSTRUCTION 009 0x00007ffff7fc2028 0x107488000 | libc.so.6!__libc_initial+0x30da
INSTRUCTION 010 0x00007ffff7fc202c 0x107488000 | libc.so.6!__libc_initial+0x30de
INSTRUCTION 011 0x00007ffff7fc2000 0x107488000 | libc.so.6!__libc_initial+0x30b2
INSTRUCTION 012 0x00007ffff7fc2001 0x107488000 | libc.so.6!__libc_initial+0x30b3
INSTRUCTION 013 0x00007ffff7fc2004 0x107488000 | libc.so.6!__libc_initial+0x30b6
INSTRUCTION 014 0x00007ffff7fc2009 0x107488000 | libc.so.6!__libc_initial+0x30bb
INSTRUCTION 333 0x00007ffff7fc200b 0x107488000 | libc.so.6!__libc_initial+0x30bd
INSTRUCTION 334 0x00007ffff7fc200d 0x107488000 | libc.so.6!__libc_initial+0x30bf
INSTRUCTION 335 0x00007ffff7fc2010 0x107488000 | libc.so.6!__libc_initial+0x30c2
INSTRUCTION 336 0x00007ffff7fc2013 0x107488000 | libc.so.6!__libc_initial+0x30c5
INSTRUCTION 337 0x00007ffff7fc2014 0x107488000 | libc.so.6!__libc_initial+0x30c6
INSTRUCTION 338 0x00007ffff7fc2031 0x107488000 | libc.so.6!__libc_initial+0x30e3
INSTRUCTION 339 0x00007ffff7fc2034 0x107488000 | libc.so.6!__libc_initial+0x30e6
INSTRUCTION 340 0x00007ffff7fc2038 0x107488000 | libc.so.6!__libc_initial+0x30ea
INSTRUCTION 341 0x00007ffff7fc203b 0x107488000 | libc.so.6!__libc_initial+0x30ed
INSTRUCTION 342 0x00007ffff7fc203e 0x107488000 | libc.so.6!__libc_initial+0x30f0
INSTRUCTION 343 0x00007ffff7fc21b3 0x107488000 | libc.so.6!__libc_initial+0x3265
INSTRUCTION 344 0x00007ffff7fc21b7 0x107488000 | libc.so.6!__libc_initial+0x3269
INSTRUCTION 345 0x00007ffff7fc21b8 0x107488000 | libc.so.6!__libc_initial+0x326a
I additionally added the following to the fuzzer:
fn set_input(&mut self, input: &Self::Input, fuzzvm: &mut FuzzVm<Self>) -> Result<()> {
for addr in [0x7ffff7fc2020, 0x7ffff7fc2000, 0x7ffff7fc21b3, 0x7ffff7fc2044] {
fuzzvm.hexdump(VirtAddr(addr), fuzzvm.cr3(), 0x4)?;
}
// Write the mutated input
fuzzvm.write_bytes_dirty(VirtAddr(0x402004), CR3, &input)?;
Ok(())
}
The output kind of get sprayed over the UI but here is a snippet:
0x00007ffff7fc2000: 55 48 89 e5 | UH..
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc21b3: 48 83 c4 10 | H...
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2044: 48 8b 45 f8 | H.E.
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5 | UH..
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5 | UH..
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc21b3: 48 83 c4 10 | H...
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2044: 48 8b 45 f8 | H.E.
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5 | UH..
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5 | UH..
None of the coverage spots have breakpoints (0xcc
).
That was quite the journey and if you've gotten this far, I really appreciate your time, and I hope this is enough information to reproduce the issue (if it's something you're interested in fixing). I'm happy to provide more info, or clarify any information that isn't clear.
Thanks again.
Stellar writeup! Thanks for the explanation. Fun to see a use case of gathering coverage in a JIT like environment.
This looks like what I would expect. Let me try to reproduce this as well, but all the steps you've shown look correct. While I'm checking this out, three small things to try:
Can you translate the JIT region and does it come back with a valid translation?
cargo run -r -- project translate 0x00007ffff7fc2020
Second, can you rerun with -vvv
to see if there is an error message about not being able to set a coverage breakpoint?
Thanks :D Here is the output of the translate. I picked the lower address because i wanted to see both functions:
➜ snapchange-example-01 cargo run -r -- project translate 0x00007ffff7fc2000
warning: unused import: `BreakpointType`
--> src/fuzzer.rs:10:56
|
10 | use snapchange::fuzzer::{Breakpoint, BreakpointLookup, BreakpointType, Fuzzer};
| ^^^^^^^^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused import: `snapchange::Execution`
--> src/fuzzer.rs:12:5
|
12 | use snapchange::Execution;
| ^^^^^^^^^^^^^^^^^^^^^
warning: `fuzzer_template` (bin "fuzzer_template") generated 2 warnings (run `cargo fix --bin "fuzzer_template"` to apply 2 suggestions)
Finished release [optimized] target(s) in 0.06s
Running `target/release/fuzzer_template project translate 0x00007ffff7fc2000`
[2023-09-06T20:10:40Z INFO snapchange::stack_unwinder] "./snapshot/vmlinux" has no .eh_frame_hdr
[2023-09-06T20:10:40Z INFO snapchange::cmdline] Gathering unwinders took: 77.383µs
[2023-09-06T20:10:40Z INFO snapchange::memory] Open memory backing: 0x7ffeb6f19000..0x7ffff6f19000
[2023-09-06T20:10:40Z INFO snapchange::commands::project] Translating VirtAddr 0x7ffff7fc2000 Cr3 0x107488000
[2023-09-06T20:10:40Z INFO snapchange::commands::project] VirtAddr 0x7ffff7fc2000 -> PhysAddr 0x1083fa000
HEXDUMP
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5 b8 27 00 00 00 0f 05 89 c1 89 4d fc | UH...'........M.
0x00007ffff7fc2010: 8b 45 fc 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 | .E.].f..........
0x00007ffff7fc2020: 55 48 89 e5 48 83 ec 10 48 89 7d f8 e8 cf ff ff | UH..H...H.}.....
0x00007ffff7fc2030: ff 89 45 f4 48 8b 4d f8 0f be 01 83 f8 66 0f 85 | ..E.H.M......f..
POTENTIAL INSTRUCTIONS
0x00007ffff7fc2000: 55 section_0x7ffff7fc2000_start+0x0 | push rbp
0x00007ffff7fc2001: 4889e5 section_0x7ffff7fc2000_start+0x1 | mov rbp, rsp
0x00007ffff7fc2004: b827000000 section_0x7ffff7fc2000_start+0x4 | mov eax, 0x27
0x00007ffff7fc2009: 0f05 section_0x7ffff7fc2000_start+0x9 | syscall
0x00007ffff7fc200b: 89c1 section_0x7ffff7fc2000_start+0xb | mov ecx, eax
0x00007ffff7fc200d: 894dfc section_0x7ffff7fc2000_start+0xd | mov dword ptr [rbp-0x4], ecx
0x00007ffff7fc2010: 8b45fc section_0x7ffff7fc2000_start+0x10 | mov eax, dword ptr [rbp-0x4]
0x00007ffff7fc2013: 5d section_0x7ffff7fc2000_start+0x13 | pop rbp
0x00007ffff7fc2014: c3 section_0x7ffff7fc2000_start+0x14 | ret
0x00007ffff7fc2015: 662e0f1f840000000000 section_0x7ffff7fc2000_start+0x15 | nop word ptr cs:[rax+rax]
0x00007ffff7fc201f: 90 section_0x7ffff7fc2000_start+0x1f | nop
0x00007ffff7fc2020: 55 section_0x7ffff7fc2000_start+0x20 | push rbp
0x00007ffff7fc2021: 4889e5 section_0x7ffff7fc2000_start+0x21 | mov rbp, rsp
0x00007ffff7fc2024: 4883ec10 section_0x7ffff7fc2000_start+0x24 | sub rsp, 0x10
0x00007ffff7fc2028: 48897df8 section_0x7ffff7fc2000_start+0x28 | mov qword ptr [rbp-0x8], rdi
0x00007ffff7fc202c: e8cfffffff section_0x7ffff7fc2000_start+0x2c | call 0xffffffffffffffd4
0x00007ffff7fc2031: 8945f4 section_0x7ffff7fc2000_start+0x31 | mov dword ptr [rbp-0xc], eax
0x00007ffff7fc2034: 488b4df8 section_0x7ffff7fc2000_start+0x34 | mov rcx, qword ptr [rbp-0x8]
0x00007ffff7fc2038: 0fbe01 section_0x7ffff7fc2000_start+0x38 | movsx eax, byte ptr [rcx]
0x00007ffff7fc203b: 83f866 section_0x7ffff7fc2000_start+0x3b | cmp eax, 0x66
This looks correct to me, the local_getpid asm is there, and the start of the fuzzme function is also correct.
I ran with the -vvv
and I'm assuming the debug output shows up in the logs:
Tui Log [log=51.1/s]
16:13:46:I:snapchange::commands::fuzz:Config { stats: Stats { merge_coverage_timer: 60s, maximum_new_corpus_size: fa, minimum_total_corpus_percentage_sync: a, maximum_total_corpus_percentage_sync: 32 }, guest_memory_size: 140000000 }
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:811:Linux user symbol: libc.so.6!__GI_raise @ 0x7ffff7e19f20 0x107488000 Crash
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:811:Linux user symbol: libc.so.6!__GI_exit @ 0x7ffff7e1c600 0x107488000 Reset
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: __die_body @ 0xffffffff81026a20 0x107488000 KernelDie
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: force_sig_fault @ 0xffffffff8107e830 0x107488000 ForceSigFault
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: do_idle @ 0xffffffff810b0b40 0x107488000 Reset
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: univ8250_console_write @ 0xffffffff81559610 0x107488000 ConsoleWrite
16:13:46:W:snapchange::commands::fuzz:/home/user/snapchange/src/commands/fuzz.rs:110:Starting all 1 worker threads
16:13:46:W:snapchange::commands::fuzz:/home/user/snapchange/src/commands/fuzz.rs:162:No dictionary in use. "./snapshot/dict" not found.
16:13:46:I:snapchange::commands::fuzz:Starting corpus: Total 1 Per core 1
16:13:46:I:snapchange::commands::fuzz:Execution timeout: 10s
16:13:46:I:snapchange::cmdline:Total coverage: 1 Coverage seen: 0 Redqueen coverage: 0
16:13:46:I:snapchange::commands::fuzz:Coverage left: 1
16:13:46:I:snapchange::commands::fuzz:Pre-populating coverage breakpoints
16:13:46:I:snapchange::commands::fuzz:Given 1 | Valid 1 | Can write 36933.08 covbps/sec
16:13:46:I:snapchange::commands::fuzz:Cloning corpus of 1 for 1 cores
16:13:46:I:snapchange::commands::fuzz:...took 976ns
16:13:46:I:snapchange::commands::fuzz:Cloned 1 symbols in 10.750864ms
16:13:46:I:snapchange::commands::fuzz:Cloned 1 covbps in 1.703µs
16:13:46:I:snapchange::commands::fuzz:Cloned 1 dictionaries in 358ns
16:13:46:I:snapchange::commands::fuzz:Cloned 1 configs in 207ns
16:13:46:I:snapchange::commands::fuzz:Cloned 1 unwinders in 1.097µs
16:13:46:I:snapchange::commands::fuzz:Cloned 1 previous coverage in 271ns
16:13:46:I:snapchange::commands::fuzz:Cloned 1 symbol breakpoints in 361ns
16:13:46:D:snapchange::fuzzvm:/home/user/snapchange/src/fuzzvm.rs:3062:01: Initializing the fuzzer breakpoint cache
16:13:46:D:snapchange::fuzzvm:/home/user/snapchange/src/fuzzvm.rs:3132:01: Initializing the fuzzer breakpoint cache of 0 breakpoints took 82ns
16:13:47:I:snapchange::stats:Found vmlinux addr2line context
16:13:47:I:snapchange::stats:Found File { fd: 13, path: "/home/user/snapchange-example-01/snapshot/example1.bin", read: true, write: false } addr2line context
16:13:47:D:snapchange::stats:/home/user/snapchange/src/stats.rs:1344:0 New corpus len: 1
Ah! Looking back, I think the hexdump
output is promising, since if the coverage has been seen, the breakpoint won't be initialized again. So the coverage might actually has been hit already?
Do you happen to see any of the dump_7ffff7fc2000.covbps
breakpoints in ./snapshot/coverage.addresses
? If so, then that coverage has been hit.
If it isn't in snapshot/coverage.addresses
, can we reset the coverage (rm ./snapshot/coverage.*
) and then add back the hexdump check in set_input
to see if, without any fuzz runs, if it initializes to 0xcc
.
For some of this debugging, we can also use cargo run -r -- fuzz -c 8 --ascii-stats
to not clobber the TUI.
So no addresses are in coverage.addresses at all.
➜ snapshot ls -la coverage.*
-rw-rw-r-- 1 caesurus caesurus 0 Sep 6 16:21 coverage.addresses
-rw-rw-r-- 1 caesurus caesurus 0 Sep 6 16:21 coverage.in_order
-rw-rw-r-- 1 caesurus caesurus 4 Sep 6 16:21 coverage.lcov
-rw-rw-r-- 1 caesurus caesurus 0 Sep 6 16:21 coverage.lighthouse
-rw-rw-r-- 1 caesurus caesurus 0 Sep 6 16:21 coverage.redqueen
-rw-rw-r-- 1 caesurus caesurus 0 Sep 6 16:21 coverage.src
➜ snapshot cat coverage.lcov
TN:
I did what you mentioned, and removed the coverage files. Re-enabled the set_input
hexdumps and did the following:
➜ snapchange-example-01 rm ./snapshot/coverage.*
➜ snapchange-example-01 rm ./snapshot/coverage.*
zsh: no matches found: ./snapshot/coverage.*
➜ snapchange-example-01 cargo run -r -- fuzz -c 8 --ascii-stats > logme.txt
warning: unused import: `BreakpointType`
--> src/fuzzer.rs:10:56
|
10 | use snapchange::fuzzer::{Breakpoint, BreakpointLookup, BreakpointType, Fuzzer};
| ^^^^^^^^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
warning: unused import: `snapchange::Execution`
--> src/fuzzer.rs:12:5
|
12 | use snapchange::Execution;
| ^^^^^^^^^^^^^^^^^^^^^
warning: `fuzzer_template` (bin "fuzzer_template") generated 2 warnings (run `cargo fix --bin "fuzzer_template"` to apply 2 suggestions)
Finished release [optimized] target(s) in 0.06s
Running `target/release/fuzzer_template fuzz -c 8 --ascii-stats`
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Config { stats: Stats { merge_coverage_timer: 60s, maximum_new_corpus_size: fa, minimum_total_corpus_percentage_sync: a, maximum_total_corpus_percentage_sync: 32 }, guest_memory_size: 140000000 }
[2023-09-07T00:46:21Z WARN snapchange::commands::fuzz] Starting all 8 worker threads
[2023-09-07T00:46:21Z WARN snapchange::commands::fuzz] No dictionary in use. "./snapshot/dict" not found.
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Starting corpus: Total 1 Per core 1
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Execution timeout: 1s
[2023-09-07T00:46:21Z INFO snapchange::cmdline] Total coverage: 1 Coverage seen: 0 Redqueen coverage: 0
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Coverage left: 1
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Pre-populating coverage breakpoints
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Given 1 | Valid 1 | Can write 8537.60 covbps/sec
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloning corpus of 1 for 8 cores
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] ...took 19.908µs
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 symbols in 89.598181ms
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 covbps in 910ns
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 dictionaries in 371ns
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 configs in 260ns
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 unwinders in 3.19µs
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 previous coverage in 1.42µs
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Cloned 8 symbol breakpoints in 638ns
[2023-09-07T00:46:23Z INFO snapchange::stats] Found vmlinux addr2line context
[2023-09-07T00:46:23Z INFO snapchange::stats] Found File { fd: 26, path: "/home/caesurus/sambashare/snapchange-example-01/snapshot/example1.bin", read: true, write: false } addr2line context
^C[2023-09-07T00:46:25Z INFO snapchange::commands::fuzz] CTRL C PRESSED!
[2023-09-07T00:46:25Z INFO snapchange] [kick_cores] FINISHED
[2023-09-07T00:46:25Z INFO snapchange] [kick_cores] Waiting for all threads to die....
[2023-09-07T00:46:25Z INFO snapchange] [kick_cores] FINISHED
➜ snapchange-example-01 head logme.txt
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5 | UH..
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5 | UH..
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc21b3: 48 83 c4 10 | H...
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2044: 48 8b 45 f8 | H.E.
---- address ----- 0 1 2 3 4 5 6 7 8 9 A B C D E F 0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5 | UH..
I redirected the stdout since stderr still sort of clobbered things, but splitting it up got a nice output. I still do not see the breakpoints get set
Interesting, interesting here. It looks like we are only finding one coverage breakpoint address from the .covbps
files in ./snapshot
:
[2023-09-07T00:46:21Z INFO snapchange::commands::fuzz] Coverage left: 1
Just as a confirmation, do we still have the ./snapshot/dump_7ffff7fc2000.covbps
file in that directory? Are there other .covbps
files in the ./snapshot
directory?
Yes... but I was laying awake thinking about this early this morning.... I know that other coverage points worked so thought I should check the following:
➜ snapshot hexdump -C manual_test.covbps
00000000 30 78 37 66 66 66 66 37 65 30 34 30 32 30 0a 30 |0x7ffff7e04020.0|
00000010 78 37 66 66 66 66 37 65 30 34 30 32 36 0a 30 78 |x7ffff7e04026.0x|
00000020 37 66 66 66 66 37 65 30 34 30 33 30 0a 30 78 37 |7ffff7e04030.0x7|
00000030 66 66 66 66 37 66 63 32 30 30 30 0a 30 78 37 66 |ffff7fc2000.0x7f|
00000040 66 66 66 37 66 63 32 30 32 30 0a 30 78 37 66 66 |fff7fc2020.0x7ff|
00000050 66 66 37 66 63 32 31 62 33 0a 30 78 37 66 66 66 |ff7fc21b3.0x7fff|
00000060 66 37 66 63 32 30 34 34 0a 30 78 37 66 66 66 66 |f7fc2044.0x7ffff|
00000070 37 66 63 32 31 61 65 0a 30 78 37 66 66 66 66 37 |7fc21ae.0x7ffff7|
00000080 66 63 32 30 35 35 0a 30 78 37 66 66 66 66 37 66 |fc2055.0x7ffff7f|
00000090 63 32 31 61 39 0a 0a |c21a9..|
00000097
➜ snapshot hexdump -C dump_7ffff7fc2000.covbps
00000000 30 78 37 66 66 66 66 37 66 63 32 30 30 30 0d 0a |0x7ffff7fc2000..|
00000010 30 78 37 66 66 66 66 37 66 63 32 30 32 30 0d 0a |0x7ffff7fc2020..|
00000020 30 78 37 66 66 66 66 37 66 63 32 31 62 33 0d 0a |0x7ffff7fc21b3..|
00000030 30 78 37 66 66 66 66 37 66 63 32 30 34 34 0d 0a |0x7ffff7fc2044..|
00000040 30 78 37 66 66 66 66 37 66 63 32 31 61 65 0d 0a |0x7ffff7fc21ae..|
00000050 30 78 37 66 66 66 66 37 66 63 32 30 35 35 0d 0a |0x7ffff7fc2055..|
00000060 30 78 37 66 66 66 66 37 66 63 32 31 61 39 0d 0a |0x7ffff7fc21a9..|
00000070 30 78 37 66 66 66 66 37 66 63 32 30 36 36 0d 0a |0x7ffff7fc2066..|
00000080 30 78 37 66 66 66 66 37 66 63 32 31 61 34 0d 0a |0x7ffff7fc21a4..|
00000090 30 78 37 66 66 66 66 37 66 63 32 30 37 37 0d 0a |0x7ffff7fc2077..|
000000a0 30 78 37 66 66 66 66 37 66 63 32 31 39 66 0d 0a |0x7ffff7fc219f..|
000000b0 30 78 37 66 66 66 66 37 66 63 32 30 38 38 0d 0a |0x7ffff7fc2088..|
and sure enough, converted the line endings:
sed -i 's/\r//' dump_7ffff7fc2000.covbps
and now they are showing up!
[2023-09-07T12:41:04Z INFO snapchange::commands::fuzz] Coverage left: 39
[2023-09-07T12:41:04Z INFO snapchange::commands::fuzz] Pre-populating coverage breakpoints
[2023-09-07T12:41:04Z INFO snapchange::commands::fuzz] Given 39 | Valid 39 | Can write 377639.85 covbps/sec
and
➜ snapshot cat coverage.addresses
0x7ffff7fc2000
0x7ffff7fc2020
0x7ffff7fc2044
0x7ffff7fc2055
0x7ffff7fc21a9
0x7ffff7fc21ae
So thank you for your help in getting to the bottom of this. Would you like me to create a separate issue stating the problem with /r/n
line endings? Or would you prefer to handle it a different way? I'm happy to close this specific issue.
Thanks again
Not a worry at all. Glad we were able to nail it down.
Looks like there is already a PR to fix the issue, so no worries on creating another one.
👋 I'm working with snapchange and finding it super easy to use and think it's really great. I had a question regarding coverage.
I can generate coverage for binaries that I have, including shared libraries. I just place the binary/library in the snapshot directory with a
.bin
extension, along with the coverage file. This seems to work well because I can specify the base address from the vmmap for loaded libraries.However, what if I have something that is custom packed. So there is a loader that does mmap and loads code into the memory region, and then executes code from there. I can dump the executable memory regions to disk, and can even generate code coverage for those memory regions. However I can not just load the memory dump into the snapshot directory with a
.bin
extension because it gives me an error.This honestly seems pretty reasonable to me. But I'd like to be able to give the fuzzer information about coverage in these regions and don't want to have to try to create an ELF file just to do this.
Do you have any ideas how I can approach this use case? Any suggestions are greatly appreciated.
Thanks again for releasing this tool.