Coverage without .bin file

[Caesurus]

👋 I'm working with snapchange and finding it super easy to use and think it's really great. I had a question regarding coverage.

I can generate coverage for binaries that I have, including shared libraries. I just place the binary/library in the snapshot directory with a .bin extension, along with the coverage file. This seems to work well because I can specify the base address from the vmmap for loaded libraries.

However, what if I have something that is custom packed. So there is a loader that does mmap and loads code into the memory region, and then executes code from there. I can dump the executable memory regions to disk, and can even generate code coverage for those memory regions. However I can not just load the memory dump into the snapshot directory with a .bin extension because it gives me an error.

     Finished release [optimized] target(s) in 0.09s
     Running `target/release/fuzzer_template fuzz --cores 12 --timeout 60s`
thread 'main' panicked at src/main.rs:6:49:
called `Result::unwrap()` on an `Err` value: Object error: Error("Unknown file magic")

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: snapchange::cmdline::get_project_state
   2: snapchange::snapchange_main
   3: fuzzer_template::main
   4: std::sys_common::backtrace::__rust_begin_short_backtrace
   5: std::rt::lang_start::{{closure}}
   6: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/core/src/ops/function.rs:284:13
   7: std::panicking::try::do_call
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panicking.rs:524:40
   8: std::panicking::try
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panicking.rs:488:19
   9: std::panic::catch_unwind
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panic.rs:142:14
  10: std::rt::lang_start_internal::{{closure}}
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/rt.rs:148:48
  11: std::panicking::try::do_call
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panicking.rs:524:40
  12: std::panicking::try
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panicking.rs:488:19
  13: std::panic::catch_unwind
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panic.rs:142:14
  14: std::rt::lang_start_internal
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/rt.rs:148:20
  15: main
  16: __libc_start_main
             at /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
  17: _start
stack backtrace:
   0: rust_begin_unwind
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/std/src/panicking.rs:617:5
   1: core::panicking::panic_fmt
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/core/src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/d12c6e947ceacf3b22c154caf9532b390d8dc88a/library/core/src/result.rs:1652:5
   3: fuzzer_template::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

This honestly seems pretty reasonable to me. But I'd like to be able to give the fuzzer information about coverage in these regions and don't want to have to try to create an ELF file just to do this.

Do you have any ideas how I can approach this use case? Any suggestions are greatly appreciated.

Thanks again for releasing this tool.

[Caesurus]

Minor update. It looks like I don't need to provide the bin file at all for coverage to be consumed. But I've run into a problem.

When trying to do coverage across multiple memory segments, the coverage blocks found only show coverage in one specific section. So for instance, if I generate covbps for libc, and those get hit first, the coverage for other memory regions aren't showing up.

If I get time I'll try to come up with a concrete example of this. Just curious if this is by design or whether it's a bug of some sort.

Thanks

[corydu]

Very cool! Glad that snapchange is working out for you.

Yeah, the .bin file is mostly for automated symbol gathering for debugging. Only the .covbps file(s) is needed for coverage.

Interesting case. I haven't come across this specifically, but there could be a few things we can try to attempt to nail down what's going on.

When you have your covbps for libc, can you do a single step trace (cargo run -r -- trace <INPUT_FILE>)? Does the coverage point that you are expecting get executed in the single step trace?

Likewise, as a check to see if the coverage breakpoints are being set at all, in your set_input function, we could dump a few specific coverage breakpoints that you are expecting and see if the breakpoint has been written.

Something like this:

impl Fuzzer for Example1Fuzzer {
    fn set_input(&mut self, input: &Self::Input, fuzzvm: &mut FuzzVm<Self>) -> Result<()> {
        for addr in [/* fill with expected coverage points that aren't showing up */] {
            fuzzvm.hexdump(VirtAddr(addr), fuzzvm.cr3(), 0x4)?;
        }
    }
}

And ensure the first byte at each of those coverage points is a 0xcc

[Caesurus]

Thanks so much for the response. That's great debug information. I'm hoping to have spare cycles to investigate further early next week.

[Caesurus]

Ok, so bear with me... I've tried to reproduce the problem. And it's not conventional code, so I could totally be doing something wrong. I modified the example1.c code in an attempt to simplify the usecase.

// Example test case for snapshot fuzzing
//
// Test the ability to write arbitrary memory and registers into a snapshot
//
// clang -ggdb -O0 example1.c -o example1

#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <string.h>
#include <asm/unistd.h>

pid_t local_getpid() {
    pid_t pid;
    asm (
        "movl $39, %%eax\n"  // syscall number for getpid
        "syscall\n"
        "movl %%eax, %0\n"
        : "=r"(pid)
        :
        : "rax"
    );
    return pid;
}

void fuzzme(char* data) {
    int pid = local_getpid();

    // Correct solution: data == "fuzzmetosolveme!", pid == 0xdeadbeef
    if (data[0]  == 'f') {
    if (data[1]  == 'u') {
    if (data[2]  == 'z') {
    if (data[3]  == 'z') {
    if (data[4]  == 'm') {
    if (data[5]  == 'e') {
    if (data[6]  == 't') {
    if (data[7]  == 'o') {
    if (data[8]  == 's') {
    if (data[9]  == 'o') {
    if (data[10] == 'l') {
    if (data[11] == 'v') {
    if (data[12] == 'e') {
    if (data[13] == 'm') {
    if (data[14] == 'e') {
    if (data[15] == '!') {
        pid = local_getpid();
        if (pid == 0xdeadbeef) {
            *(int*)0xcafecafe = 0x41414141;
        }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }
    }

    return;
}

int main() {
    char* data = "aaaaaaaaaaaaaaaa";

    size_t size = 4096;  
    void *ptr = mmap(NULL, size, PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

    if (ptr == MAP_FAILED) {
        return 1;
    }

    int offset = (char*)fuzzme - (char*)local_getpid;
    printf("offset = 0x%x - 0x%x = 0x%x\n", (char*)fuzzme, (char*)local_getpid, offset);
    memcpy(ptr, local_getpid, size);

    if (mprotect(ptr, size, PROT_READ | PROT_EXEC) == -1) {
        return 1;
    }

    void (*func_ptr)() = ptr+offset;

    if(getenv("SNAPSHOT") != 0) {
        // Print the memory buffer address and pid address for fuzzing
        printf("SNAPSHOT Data buffer: %p\n", data);

        // Ensure the stdout has been flushed
        fflush(stdout);

        // Snapshot taken here
        __asm("int3 ; vmcall"); 
    }

    // Call the fuzzme function
    func_ptr(data);
    //fuzzme(data);

    munmap(ptr, size);
    return 0;
}

As you can see, I'm copying the two functions into a freshly mmap'd section and making sure it's executable. Then calling the copy of fuzzme that exists in that new section. I had to rewrite getpid so I didn't have to resolve symbols to libc etc... but the functionality should be the same.

I added the following function to gdbsnapshot.py:

def dumpmem():
    pid = gdb.selected_inferior().pid
    maps_path = f"/proc/{pid}/maps"

    try:
        with open(maps_path, "r") as maps_file:
            for line in maps_file:
                print(line)
                parts = line.strip().split()
                addr_range, permissions = parts[0], parts[1]

                # Check if the section has r-x permissions
                if 'x' in permissions and 'r' in permissions:
                    start_addr, end_addr = [
                        int(x, 16) for x in addr_range.split('-')
                    ]
                    filename = f"/tmp/dump_{start_addr:x}.dmp"
                    print(
                        f"Dumping section {start_addr:x}-{end_addr:x} to {filename}"
                    )

                    # Read the section memory and write to file
                    with open(filename, "wb") as file:
                        mem = gdb.selected_inferior().read_memory(
                            start_addr, end_addr - start_addr)
                        file.write(mem)

        print("Dumping completed.")

    except Exception as e:
        print(f"An error occurred: {str(e)}")
        print(
            "Make sure you have the necessary permissions to read the memory maps."
        )

And then the line to call the function...

And then obviously the necessary code to copy those *.dmp files out of the image in the utils/extract.sh script... This gives me the executable memory regions. In my snapshot, I have the following vmmap:

          Start Addr           End Addr       Size     Offset  Perms  objfile
            0x400000           0x401000     0x1000        0x0  r--p   /root/example1
            0x401000           0x402000     0x1000     0x1000  r-xp   /root/example1
            0x402000           0x403000     0x1000     0x2000  r--p   /root/example1
            0x403000           0x404000     0x1000     0x2000  r--p   /root/example1
            0x404000           0x405000     0x1000     0x3000  rw-p   /root/example1
            0x405000           0x426000    0x21000        0x0  rw-p   [heap]
      0x7ffff7ddb000     0x7ffff7dde000     0x3000        0x0  rw-p   
      0x7ffff7dde000     0x7ffff7e04000    0x26000        0x0  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7e04000     0x7ffff7f59000   0x155000    0x26000  r-xp   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7f59000     0x7ffff7fac000    0x53000   0x17b000  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7fac000     0x7ffff7fb0000     0x4000   0x1ce000  r--p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7fb0000     0x7ffff7fb2000     0x2000   0x1d2000  rw-p   /usr/lib/x86_64-linux-gnu/libc.so.6
      0x7ffff7fb2000     0x7ffff7fbf000     0xd000        0x0  rw-p   
      0x7ffff7fc2000     0x7ffff7fc3000     0x1000        0x0  r-xp   
      0x7ffff7fc3000     0x7ffff7fc5000     0x2000        0x0  rw-p   
      0x7ffff7fc5000     0x7ffff7fc9000     0x4000        0x0  r--p   [vvar]
      0x7ffff7fc9000     0x7ffff7fcb000     0x2000        0x0  r-xp   [vdso]
      0x7ffff7fcb000     0x7ffff7fcc000     0x1000        0x0  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7fcc000     0x7ffff7ff1000    0x25000     0x1000  r-xp   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7ff1000     0x7ffff7ffb000     0xa000    0x26000  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7ffb000     0x7ffff7ffd000     0x2000    0x30000  r--p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffff7ffd000     0x7ffff7fff000     0x2000    0x32000  rw-p   /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
      0x7ffffffde000     0x7ffffffff000    0x21000        0x0  rw-p   [stack]
  0xffffffffff600000 0xffffffffff601000     0x1000        0x0  --xp   [vsyscall]

The mmap'd region is:

0x7ffff7fc2000     0x7ffff7fc3000     0x1000        0x0  r-xp 

Now I load that into binja, with some minor fixes to the bn_snapchange.py: https://github.com/awslabs/snapchange/blob/main/coverage_scripts/bn_snapchange.py#L696-L704 these use rebase, which are no longer parameters in the base SnapchangeTask class... (I may look at opening a PR).

Anyway... I open the memory dump with the correct base addresse 0x7ffff7fc2000 , and architecture x86_64, and platform linux-x86_64. Then generate a covbps file named dump_7ffff7fc2000.covbps that looks like this:

0x7ffff7fc2000
0x7ffff7fc2020
0x7ffff7fc21b3
0x7ffff7fc2044
0x7ffff7fc21ae
0x7ffff7fc2055
0x7ffff7fc21a9
0x7ffff7fc2066
0x7ffff7fc21a4
0x7ffff7fc2077
0x7ffff7fc219f
0x7ffff7fc2088
0x7ffff7fc219a
0x7ffff7fc2099
0x7ffff7fc2195
0x7ffff7fc20aa
0x7ffff7fc2190
0x7ffff7fc20bb
0x7ffff7fc218b
0x7ffff7fc20cc
0x7ffff7fc2186
0x7ffff7fc20dd
0x7ffff7fc2181
0x7ffff7fc20ee
0x7ffff7fc217c
0x7ffff7fc20ff
0x7ffff7fc2177
0x7ffff7fc2110
0x7ffff7fc2172
0x7ffff7fc2121
0x7ffff7fc216d
0x7ffff7fc2132
0x7ffff7fc2168
0x7ffff7fc2143
0x7ffff7fc2163
0x7ffff7fc2158
0x7ffff7fc23b4
0x7ffff7fc2edf
0x7ffff7fc2f07

Ok, so that file is in the snapshot directory. If that is the only covbps file in the directory and I run the fuzzer, I don't get any coverage blocks hit at all.

When I do a trace, I see these addresses are in fact hit (selectively grep'd so that it's not too much output):

➜  snapshot grep 7ffff7fc traces/f467b5ff9c99cd88_trace|grep INSTRUC
INSTRUCTION 006 0x00007ffff7fc2020 0x107488000 | libc.so.6!__libc_initial+0x30d2
INSTRUCTION 007 0x00007ffff7fc2021 0x107488000 | libc.so.6!__libc_initial+0x30d3
INSTRUCTION 008 0x00007ffff7fc2024 0x107488000 | libc.so.6!__libc_initial+0x30d6
INSTRUCTION 009 0x00007ffff7fc2028 0x107488000 | libc.so.6!__libc_initial+0x30da
INSTRUCTION 010 0x00007ffff7fc202c 0x107488000 | libc.so.6!__libc_initial+0x30de
INSTRUCTION 011 0x00007ffff7fc2000 0x107488000 | libc.so.6!__libc_initial+0x30b2
INSTRUCTION 012 0x00007ffff7fc2001 0x107488000 | libc.so.6!__libc_initial+0x30b3
INSTRUCTION 013 0x00007ffff7fc2004 0x107488000 | libc.so.6!__libc_initial+0x30b6
INSTRUCTION 014 0x00007ffff7fc2009 0x107488000 | libc.so.6!__libc_initial+0x30bb
INSTRUCTION 333 0x00007ffff7fc200b 0x107488000 | libc.so.6!__libc_initial+0x30bd
INSTRUCTION 334 0x00007ffff7fc200d 0x107488000 | libc.so.6!__libc_initial+0x30bf
INSTRUCTION 335 0x00007ffff7fc2010 0x107488000 | libc.so.6!__libc_initial+0x30c2
INSTRUCTION 336 0x00007ffff7fc2013 0x107488000 | libc.so.6!__libc_initial+0x30c5
INSTRUCTION 337 0x00007ffff7fc2014 0x107488000 | libc.so.6!__libc_initial+0x30c6
INSTRUCTION 338 0x00007ffff7fc2031 0x107488000 | libc.so.6!__libc_initial+0x30e3
INSTRUCTION 339 0x00007ffff7fc2034 0x107488000 | libc.so.6!__libc_initial+0x30e6
INSTRUCTION 340 0x00007ffff7fc2038 0x107488000 | libc.so.6!__libc_initial+0x30ea
INSTRUCTION 341 0x00007ffff7fc203b 0x107488000 | libc.so.6!__libc_initial+0x30ed
INSTRUCTION 342 0x00007ffff7fc203e 0x107488000 | libc.so.6!__libc_initial+0x30f0
INSTRUCTION 343 0x00007ffff7fc21b3 0x107488000 | libc.so.6!__libc_initial+0x3265
INSTRUCTION 344 0x00007ffff7fc21b7 0x107488000 | libc.so.6!__libc_initial+0x3269
INSTRUCTION 345 0x00007ffff7fc21b8 0x107488000 | libc.so.6!__libc_initial+0x326a

I additionally added the following to the fuzzer:

    fn set_input(&mut self, input: &Self::Input, fuzzvm: &mut FuzzVm<Self>) -> Result<()> {
        for addr in [0x7ffff7fc2020, 0x7ffff7fc2000, 0x7ffff7fc21b3, 0x7ffff7fc2044] {
            fuzzvm.hexdump(VirtAddr(addr), fuzzvm.cr3(), 0x4)?;
        }

        // Write the mutated input
        fuzzvm.write_bytes_dirty(VirtAddr(0x402004), CR3, &input)?;
        Ok(())
    }

The output kind of get sprayed over the UI but here is a snippet:

0x00007ffff7fc2000: 55 48 89 e5                                      | UH..
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc21b3: 48 83 c4 10                                      | H...
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2044: 48 8b 45 f8                                      | H.E.
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5                                      | UH..
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5                                      | UH..
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc21b3: 48 83 c4 10                                      | H...
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2044: 48 8b 45 f8                                      | H.E.
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5                                      | UH..
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5                                      | UH..

None of the coverage spots have breakpoints (0xcc).

That was quite the journey and if you've gotten this far, I really appreciate your time, and I hope this is enough information to reproduce the issue (if it's something you're interested in fixing). I'm happy to provide more info, or clarify any information that isn't clear.

Thanks again.

[corydu]

Stellar writeup! Thanks for the explanation. Fun to see a use case of gathering coverage in a JIT like environment.

This looks like what I would expect. Let me try to reproduce this as well, but all the steps you've shown look correct. While I'm checking this out, three small things to try:

Can you translate the JIT region and does it come back with a valid translation?

cargo run -r -- project translate 0x00007ffff7fc2020

Second, can you rerun with -vvv to see if there is an error message about not being able to set a coverage breakpoint?

[Caesurus]

Thanks :D Here is the output of the translate. I picked the lower address because i wanted to see both functions:

➜  snapchange-example-01 cargo run -r -- project translate 0x00007ffff7fc2000
warning: unused import: `BreakpointType`
  --> src/fuzzer.rs:10:56
   |
10 | use snapchange::fuzzer::{Breakpoint, BreakpointLookup, BreakpointType, Fuzzer};
   |                                                        ^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: unused import: `snapchange::Execution`
  --> src/fuzzer.rs:12:5
   |
12 | use snapchange::Execution;
   |     ^^^^^^^^^^^^^^^^^^^^^

warning: `fuzzer_template` (bin "fuzzer_template") generated 2 warnings (run `cargo fix --bin "fuzzer_template"` to apply 2 suggestions)
    Finished release [optimized] target(s) in 0.06s
     Running `target/release/fuzzer_template project translate 0x00007ffff7fc2000`
[2023-09-06T20:10:40Z INFO  snapchange::stack_unwinder] "./snapshot/vmlinux" has no .eh_frame_hdr
[2023-09-06T20:10:40Z INFO  snapchange::cmdline] Gathering unwinders took: 77.383µs
[2023-09-06T20:10:40Z INFO  snapchange::memory] Open memory backing: 0x7ffeb6f19000..0x7ffff6f19000
[2023-09-06T20:10:40Z INFO  snapchange::commands::project] Translating VirtAddr 0x7ffff7fc2000 Cr3 0x107488000
[2023-09-06T20:10:40Z INFO  snapchange::commands::project] VirtAddr 0x7ffff7fc2000 -> PhysAddr 0x1083fa000
 HEXDUMP
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5 b8 27 00 00 00 0f 05 89 c1 89 4d fc  | UH...'........M.
0x00007ffff7fc2010: 8b 45 fc 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90  | .E.].f..........
0x00007ffff7fc2020: 55 48 89 e5 48 83 ec 10 48 89 7d f8 e8 cf ff ff  | UH..H...H.}.....
0x00007ffff7fc2030: ff 89 45 f4 48 8b 4d f8 0f be 01 83 f8 66 0f 85  | ..E.H.M......f..
 POTENTIAL INSTRUCTIONS
0x00007ffff7fc2000: 55                     section_0x7ffff7fc2000_start+0x0                   | push rbp
0x00007ffff7fc2001: 4889e5                 section_0x7ffff7fc2000_start+0x1                   | mov rbp, rsp
0x00007ffff7fc2004: b827000000             section_0x7ffff7fc2000_start+0x4                   | mov eax, 0x27
0x00007ffff7fc2009: 0f05                   section_0x7ffff7fc2000_start+0x9                   | syscall
0x00007ffff7fc200b: 89c1                   section_0x7ffff7fc2000_start+0xb                   | mov ecx, eax
0x00007ffff7fc200d: 894dfc                 section_0x7ffff7fc2000_start+0xd                   | mov dword ptr [rbp-0x4], ecx
0x00007ffff7fc2010: 8b45fc                 section_0x7ffff7fc2000_start+0x10                  | mov eax, dword ptr [rbp-0x4]
0x00007ffff7fc2013: 5d                     section_0x7ffff7fc2000_start+0x13                  | pop rbp
0x00007ffff7fc2014: c3                     section_0x7ffff7fc2000_start+0x14                  | ret
0x00007ffff7fc2015: 662e0f1f840000000000   section_0x7ffff7fc2000_start+0x15                  | nop word ptr cs:[rax+rax]
0x00007ffff7fc201f: 90                     section_0x7ffff7fc2000_start+0x1f                  | nop
0x00007ffff7fc2020: 55                     section_0x7ffff7fc2000_start+0x20                  | push rbp
0x00007ffff7fc2021: 4889e5                 section_0x7ffff7fc2000_start+0x21                  | mov rbp, rsp
0x00007ffff7fc2024: 4883ec10               section_0x7ffff7fc2000_start+0x24                  | sub rsp, 0x10
0x00007ffff7fc2028: 48897df8               section_0x7ffff7fc2000_start+0x28                  | mov qword ptr [rbp-0x8], rdi
0x00007ffff7fc202c: e8cfffffff             section_0x7ffff7fc2000_start+0x2c                  | call 0xffffffffffffffd4
0x00007ffff7fc2031: 8945f4                 section_0x7ffff7fc2000_start+0x31                  | mov dword ptr [rbp-0xc], eax
0x00007ffff7fc2034: 488b4df8               section_0x7ffff7fc2000_start+0x34                  | mov rcx, qword ptr [rbp-0x8]
0x00007ffff7fc2038: 0fbe01                 section_0x7ffff7fc2000_start+0x38                  | movsx eax, byte ptr [rcx]
0x00007ffff7fc203b: 83f866                 section_0x7ffff7fc2000_start+0x3b                  | cmp eax, 0x66

This looks correct to me, the local_getpid asm is there, and the start of the fuzzme function is also correct.

I ran with the -vvv and I'm assuming the debug output shows up in the logs:

Tui Log [log=51.1/s]
16:13:46:I:snapchange::commands::fuzz:Config { stats: Stats { merge_coverage_timer: 60s, maximum_new_corpus_size: fa, minimum_total_corpus_percentage_sync: a, maximum_total_corpus_percentage_sync: 32 }, guest_memory_size: 140000000 }
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:811:Linux user symbol: libc.so.6!__GI_raise @ 0x7ffff7e19f20 0x107488000 Crash      
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:811:Linux user symbol: libc.so.6!__GI_exit @ 0x7ffff7e1c600 0x107488000 Reset       
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: __die_body @ 0xffffffff81026a20 0x107488000 KernelDie      
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: force_sig_fault @ 0xffffffff8107e830 0x107488000 ForceSigFault        
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: do_idle @ 0xffffffff810b0b40 0x107488000 Reset  
16:13:46:D:snapchange::cmdline:/home/user/snapchange/src/cmdline.rs:836:Linux kernel symbol: univ8250_console_write @ 0xffffffff81559610 0x107488000 ConsoleWrite  
16:13:46:W:snapchange::commands::fuzz:/home/user/snapchange/src/commands/fuzz.rs:110:Starting all 1 worker threads     
16:13:46:W:snapchange::commands::fuzz:/home/user/snapchange/src/commands/fuzz.rs:162:No dictionary in use. "./snapshot/dict" not found.      
16:13:46:I:snapchange::commands::fuzz:Starting corpus: Total 1 Per core 1      
16:13:46:I:snapchange::commands::fuzz:Execution timeout: 10s        
16:13:46:I:snapchange::cmdline:Total coverage: 1 Coverage seen: 0 Redqueen coverage: 0    
16:13:46:I:snapchange::commands::fuzz:Coverage left: 1   
16:13:46:I:snapchange::commands::fuzz:Pre-populating coverage breakpoints      
16:13:46:I:snapchange::commands::fuzz:Given 1 | Valid 1 | Can write         36933.08 covbps/sec      
16:13:46:I:snapchange::commands::fuzz:Cloning corpus of 1 for 1 cores          
16:13:46:I:snapchange::commands::fuzz:...took 976ns      
16:13:46:I:snapchange::commands::fuzz:Cloned 1 symbols in 10.750864ms          
16:13:46:I:snapchange::commands::fuzz:Cloned 1 covbps in 1.703µs    
16:13:46:I:snapchange::commands::fuzz:Cloned 1 dictionaries in 358ns
16:13:46:I:snapchange::commands::fuzz:Cloned 1 configs in 207ns     
16:13:46:I:snapchange::commands::fuzz:Cloned 1 unwinders in 1.097µs 
16:13:46:I:snapchange::commands::fuzz:Cloned 1 previous coverage in 271ns      
16:13:46:I:snapchange::commands::fuzz:Cloned 1 symbol breakpoints in 361ns     
16:13:46:D:snapchange::fuzzvm:/home/user/snapchange/src/fuzzvm.rs:3062:01: Initializing the fuzzer breakpoint cache    
16:13:46:D:snapchange::fuzzvm:/home/user/snapchange/src/fuzzvm.rs:3132:01: Initializing the fuzzer breakpoint cache of 0 breakpoints took 82ns          
16:13:47:I:snapchange::stats:Found vmlinux addr2line context        
16:13:47:I:snapchange::stats:Found File { fd: 13, path: "/home/user/snapchange-example-01/snapshot/example1.bin", read: true, write: false } addr2line context     
16:13:47:D:snapchange::stats:/home/user/snapchange/src/stats.rs:1344:0 New corpus len: 1 

[corydu]

Ah! Looking back, I think the hexdump output is promising, since if the coverage has been seen, the breakpoint won't be initialized again. So the coverage might actually has been hit already?

Do you happen to see any of the dump_7ffff7fc2000.covbps breakpoints in ./snapshot/coverage.addresses? If so, then that coverage has been hit.

If it isn't in snapshot/coverage.addresses, can we reset the coverage (rm ./snapshot/coverage.*) and then add back the hexdump check in set_input to see if, without any fuzz runs, if it initializes to 0xcc.

For some of this debugging, we can also use cargo run -r -- fuzz -c 8 --ascii-stats to not clobber the TUI.

[Caesurus]

So no addresses are in coverage.addresses at all.

➜  snapshot ls -la coverage.*
-rw-rw-r-- 1 caesurus caesurus 0 Sep  6 16:21 coverage.addresses
-rw-rw-r-- 1 caesurus caesurus 0 Sep  6 16:21 coverage.in_order
-rw-rw-r-- 1 caesurus caesurus 4 Sep  6 16:21 coverage.lcov
-rw-rw-r-- 1 caesurus caesurus 0 Sep  6 16:21 coverage.lighthouse
-rw-rw-r-- 1 caesurus caesurus 0 Sep  6 16:21 coverage.redqueen
-rw-rw-r-- 1 caesurus caesurus 0 Sep  6 16:21 coverage.src
➜  snapshot cat coverage.lcov
TN:

I did what you mentioned, and removed the coverage files. Re-enabled the set_input hexdumps and did the following:

➜  snapchange-example-01 rm ./snapshot/coverage.*
➜  snapchange-example-01 rm ./snapshot/coverage.*
zsh: no matches found: ./snapshot/coverage.*
➜  snapchange-example-01 cargo run -r -- fuzz -c 8 --ascii-stats > logme.txt
warning: unused import: `BreakpointType`
  --> src/fuzzer.rs:10:56
   |
10 | use snapchange::fuzzer::{Breakpoint, BreakpointLookup, BreakpointType, Fuzzer};
   |                                                        ^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: unused import: `snapchange::Execution`
  --> src/fuzzer.rs:12:5
   |
12 | use snapchange::Execution;
   |     ^^^^^^^^^^^^^^^^^^^^^

warning: `fuzzer_template` (bin "fuzzer_template") generated 2 warnings (run `cargo fix --bin "fuzzer_template"` to apply 2 suggestions)
    Finished release [optimized] target(s) in 0.06s
     Running `target/release/fuzzer_template fuzz -c 8 --ascii-stats`
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Config { stats: Stats { merge_coverage_timer: 60s, maximum_new_corpus_size: fa, minimum_total_corpus_percentage_sync: a, maximum_total_corpus_percentage_sync: 32 }, guest_memory_size: 140000000 }
[2023-09-07T00:46:21Z WARN  snapchange::commands::fuzz] Starting all 8 worker threads
[2023-09-07T00:46:21Z WARN  snapchange::commands::fuzz] No dictionary in use. "./snapshot/dict" not found.
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Starting corpus: Total 1 Per core 1
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Execution timeout: 1s
[2023-09-07T00:46:21Z INFO  snapchange::cmdline] Total coverage: 1 Coverage seen: 0 Redqueen coverage: 0
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Coverage left: 1
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Pre-populating coverage breakpoints
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Given 1 | Valid 1 | Can write          8537.60 covbps/sec
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloning corpus of 1 for 8 cores
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] ...took 19.908µs
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 symbols in 89.598181ms
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 covbps in 910ns
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 dictionaries in 371ns
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 configs in 260ns
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 unwinders in 3.19µs
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 previous coverage in 1.42µs
[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Cloned 8 symbol breakpoints in 638ns
[2023-09-07T00:46:23Z INFO  snapchange::stats] Found vmlinux addr2line context
[2023-09-07T00:46:23Z INFO  snapchange::stats] Found File { fd: 26, path: "/home/caesurus/sambashare/snapchange-example-01/snapshot/example1.bin", read: true, write: false } addr2line context
^C[2023-09-07T00:46:25Z INFO  snapchange::commands::fuzz] CTRL C PRESSED!
[2023-09-07T00:46:25Z INFO  snapchange] [kick_cores] FINISHED
[2023-09-07T00:46:25Z INFO  snapchange] [kick_cores] Waiting for all threads to die....
[2023-09-07T00:46:25Z INFO  snapchange] [kick_cores] FINISHED
➜  snapchange-example-01 head logme.txt
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5                                      | UH..
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2000: 55 48 89 e5                                      | UH..
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc21b3: 48 83 c4 10                                      | H...
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2044: 48 8b 45 f8                                      | H.E.
---- address -----   0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F    0123456789ABCDEF
0x00007ffff7fc2020: 55 48 89 e5                                      | UH..

I redirected the stdout since stderr still sort of clobbered things, but splitting it up got a nice output. I still do not see the breakpoints get set

[corydu]

Interesting, interesting here. It looks like we are only finding one coverage breakpoint address from the .covbps files in ./snapshot:

[2023-09-07T00:46:21Z INFO  snapchange::commands::fuzz] Coverage left: 1

Just as a confirmation, do we still have the ./snapshot/dump_7ffff7fc2000.covbps file in that directory? Are there other .covbps files in the ./snapshot directory?

[Caesurus]

Yes... but I was laying awake thinking about this early this morning.... I know that other coverage points worked so thought I should check the following:

➜  snapshot hexdump -C manual_test.covbps
00000000  30 78 37 66 66 66 66 37  65 30 34 30 32 30 0a 30  |0x7ffff7e04020.0|
00000010  78 37 66 66 66 66 37 65  30 34 30 32 36 0a 30 78  |x7ffff7e04026.0x|
00000020  37 66 66 66 66 37 65 30  34 30 33 30 0a 30 78 37  |7ffff7e04030.0x7|
00000030  66 66 66 66 37 66 63 32  30 30 30 0a 30 78 37 66  |ffff7fc2000.0x7f|
00000040  66 66 66 37 66 63 32 30  32 30 0a 30 78 37 66 66  |fff7fc2020.0x7ff|
00000050  66 66 37 66 63 32 31 62  33 0a 30 78 37 66 66 66  |ff7fc21b3.0x7fff|
00000060  66 37 66 63 32 30 34 34  0a 30 78 37 66 66 66 66  |f7fc2044.0x7ffff|
00000070  37 66 63 32 31 61 65 0a  30 78 37 66 66 66 66 37  |7fc21ae.0x7ffff7|
00000080  66 63 32 30 35 35 0a 30  78 37 66 66 66 66 37 66  |fc2055.0x7ffff7f|
00000090  63 32 31 61 39 0a 0a                              |c21a9..|
00000097
➜  snapshot hexdump -C dump_7ffff7fc2000.covbps
00000000  30 78 37 66 66 66 66 37  66 63 32 30 30 30 0d 0a  |0x7ffff7fc2000..|
00000010  30 78 37 66 66 66 66 37  66 63 32 30 32 30 0d 0a  |0x7ffff7fc2020..|
00000020  30 78 37 66 66 66 66 37  66 63 32 31 62 33 0d 0a  |0x7ffff7fc21b3..|
00000030  30 78 37 66 66 66 66 37  66 63 32 30 34 34 0d 0a  |0x7ffff7fc2044..|
00000040  30 78 37 66 66 66 66 37  66 63 32 31 61 65 0d 0a  |0x7ffff7fc21ae..|
00000050  30 78 37 66 66 66 66 37  66 63 32 30 35 35 0d 0a  |0x7ffff7fc2055..|
00000060  30 78 37 66 66 66 66 37  66 63 32 31 61 39 0d 0a  |0x7ffff7fc21a9..|
00000070  30 78 37 66 66 66 66 37  66 63 32 30 36 36 0d 0a  |0x7ffff7fc2066..|
00000080  30 78 37 66 66 66 66 37  66 63 32 31 61 34 0d 0a  |0x7ffff7fc21a4..|
00000090  30 78 37 66 66 66 66 37  66 63 32 30 37 37 0d 0a  |0x7ffff7fc2077..|
000000a0  30 78 37 66 66 66 66 37  66 63 32 31 39 66 0d 0a  |0x7ffff7fc219f..|
000000b0  30 78 37 66 66 66 66 37  66 63 32 30 38 38 0d 0a  |0x7ffff7fc2088..|

and sure enough, converted the line endings:

sed -i 's/\r//' dump_7ffff7fc2000.covbps

and now they are showing up!

[2023-09-07T12:41:04Z INFO  snapchange::commands::fuzz] Coverage left: 39
[2023-09-07T12:41:04Z INFO  snapchange::commands::fuzz] Pre-populating coverage breakpoints
[2023-09-07T12:41:04Z INFO  snapchange::commands::fuzz] Given 39 | Valid 39 | Can write        377639.85 covbps/sec

and

➜  snapshot cat coverage.addresses
0x7ffff7fc2000
0x7ffff7fc2020
0x7ffff7fc2044
0x7ffff7fc2055
0x7ffff7fc21a9
0x7ffff7fc21ae

So thank you for your help in getting to the bottom of this. Would you like me to create a separate issue stating the problem with /r/n line endings? Or would you prefer to handle it a different way? I'm happy to close this specific issue.

Thanks again

[corydu]

Not a worry at all. Glad we were able to nail it down.

Looks like there is already a PR to fix the issue, so no worries on creating another one.