Add support for Linux userland ELF snapshots and fuzzing

jasocrow commented 9 months ago

This is based on Kasamir123's pull request at https://github.com/0vercl0k/wtf/pull/102 plus some scripts in Snapchange for automatically setting up a Linux VM target.

The following improvements have been made as compared to Kasamir123's original pull request:

Fixed bug when calling mlockall, allowing us to remove page touching code
Code requires no custom #ifdefs in wtf
Linux snapshots work w/fuzzing via KVM. Kasamir123's code had some issues with gathering segment registers, and our updates fix these issues, allowing for KVM support
Kasamir123's code injects shellcode into the target process by overwriting code, but never restored the original code. We now restore the original code
Snapshotting is more streamlined, only taking a few manual steps once everything is configured
Some improvements from 0vercl0k's suggestions from ELF Snapshotting and Fuzzing 0vercl0k/wtf#102, like implementing raw2dmp in Python
Support for setting breakpoints on symbols in ELF targets plus use of symbols in fuzz harnesses
IDA script for generating coverage breakpoints list so that targets can be fuzzed with KVM
Target VM can run with HW acceleration enabled; Kasamir123's scripts for running the VM and taking a snapshot only worked with SW emulation
Works with recent Linux kernel versions

0vercl0k commented 9 months ago

Damn, this is an amazing contribution and I'm looking forward to diving into it!

Thanks again for working on this and contributing back.

It's probably going to take me a minute to go through all of this 😅, but bear with me; I really want to get it in!

Cheers

0vercl0k commented 9 months ago

Okay I tried to run through the readme / instructions today and I might be hitting something unexpected; or I am doing something wrong.

The set-up part works fine, QEMU / Linux kernel sources are properly downloaded / compiled as far as I can tell.

I am starting the VM and it seems to diverge from the README; I hit a bunch of SIGUSR1 signals but I keep hitting continue and I see the VM booting so maybe that's fine:

user@pc~/wtf-linux-snapshot/linux_mode/crash_test$ ../qemu_snapshot/gdb_server.sh
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../qemu_snapshot/target_vm/qemu/build/qemu-system-x86_64...
Starting program: /home/user/wtf-linux-snapshot/linux_mode/qemu_snapshot/target_vm/qemu/build/qemu-system-x86_64 -m 2G -smp 1 -kernel ../qemu_snapshot/target_vm/linux/ad

Program stopped.
0x00007ffff7fe3290 in _start () from /lib64/ld-linux-x86-64.so.2
Hardware assisted breakpoint 1 at 0x555555bcbdc0: file ../accel/kvm/kvm-all.c, line 2866.
Hardware assisted breakpoint 2 at 0x555555bad7c0: file ../accel/tcg/cpu-exec.c, line 893.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7606640 (LWP 63564)]
[New Thread 0x7ffff6c03640 (LWP 63565)]
WARNING: Image format was not specified for '../qemu_snapshot/target_vm/image/bookworm.img' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
[New Thread 0x7ffff60ff640 (LWP 63568)]

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
[Switching to Thread 0x7ffff60ff640 (LWP 63568)]
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac58) at ./nptl/futex-internal.c:57
57      ./nptl/futex-internal.c: No such file or directory.
(gdb) c
Continuing.

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac5c) at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) c
Continuing.

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac58) at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) c
Continuing.

Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55555682ac5c) at ./nptl/futex-internal.c:57
57      in ./nptl/futex-internal.c
(gdb) c

Then, I start the gdb client:

user@pc:~/wtf-linux-snapshot/linux_mode/crash_test$ ../qemu_snapshot/gdb_client.sh
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../qemu_snapshot/target_vm/linux/vmlinux...
Remote debugging using localhost:1234

This triggers a SIGUSR1 in the first shell:

         Starting systemd-update-ut… Record Runlevel Change in UTMP...
[  OK  ] Finished systemd-update-ut… - Record Runlevel Change in UTMP.
[    4.774337] audit: type=1400 audit(1707688808.601:6): avc:  denied  { watch_reads } for  pid=194 comm="(agetty)" path="/dev/tty6" dev="devtmpfs" ino=26 scontext=syst1
[    4.790192] audit: type=1400 audit(1707688808.616:7): avc:  denied  { checkpoint_restore } for  pid=194 comm="agetty" capability=40  scontext=system_u:system_r:kerne1

Debian GNU/Linux 12 linux ttyS0

linux login:
Thread 4 "qemu-system-x86" received signal SIGUSR1, User defined signal 1.
__GI___ioctl (fd=15, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
36      ../sysdeps/unix/sysv/linux/ioctl.c: No such file or directory.
(gdb)

Again, if I hit continue I get this on the gdb-client shell:

Remote debugging using localhost:1234
Ignoring packet error, continuing...
warning: unrecognized item "timeout" in "qSupported" response
Ignoring packet error, continuing...
Remote replied unexpectedly to 'vMustReplyEmpty': timeout
add symbol table from file "a.out" at
        .text_addr = 0x555555555100
--Type <RET> for more, q to quit, c to continue without paging--
Removing regs.json file if it exists...not found
Traceback (most recent call last):
  File "./bkpt.py", line 16, in <module>
    FuzzBkpt(target_dir, break_address, file_name, sym_path=file_name)
  File "/home/user/wtf-linux-snapshot/linux_mode/qemu_snapshot/gdb_fuzzbkpt.py", line 233, in __init__
    gdb.Breakpoint.__init__(
gdb.error: No hardware breakpoint support in the target.
The program is not being run.

What gdb version should I use? Or maybe it's something else? My test VM is a Ubuntu 22.04.3 LTS.

Cheers

jasocrow commented 9 months ago

Thanks for trying this out. I'll work on reproducing these issues and getting fixes in soon.

jasocrow commented 9 months ago

Ok, I think both issues you ran into were related to gdb trying to process many SIGUSR1 signals. I've updated gdb_server.sh to ignore these signals and hopefully this will get things to work for you. I must have had something in my ~/.gdbinit file to ignore these signals previously when I did my testing, and that's why I didn't see this. Please let me know if you run into any more problems!

Confirmed latest changes work on a fresh Ubuntu 22.04.2 LTS VM:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:    22.04
Codename:   jammy

0vercl0k commented 9 months ago

Awesome - I was able to generate a dump, thanks! Working through the rest of the readme now..

Cheers

0vercl0k commented 9 months ago

@jasocrow the README mentions that one of the limitation is that it doesn't support guests with more than 2GB of RAM - yet the start.sh script specifies 4G, my mem.dmp is 4GB and the pmemsave command seems to dump from 0 to 0xff'ff'ff'ff. Is the README not accurate then?

Also, I was wondering what's been your trace debugging workflow? Traces work fine but not sure how you've been symbolizing them?

Cheers

jasocrow commented 8 months ago

@0vercl0k Awesome, thanks for trying it out and starting to change things up to your liking.

Good catch on the README. I guess 4 GB started working at some point and we didn't notice. Perhaps it started working when we moved to using a VM generated from the Snapchange scripts.

Ah yeah, we never implemented a symbolizer. We've been using tenet traces and loading them in IDA.

jasocrow commented 8 months ago

@0vercl0k I can take a look at adding a symbolizer, but it might be a while before I can get to it.

0vercl0k commented 8 months ago

No worries - I was more asking from the angle of documenting the limitations and how people should go about debugging their testcases :)

Cheers

On Thu, Feb 22, 2024 at 8:55 AM jasocrow @.***> wrote:

@0vercl0k https://github.com/0vercl0k I can take a look at adding a symbolizer, but it might be a while before I can get to it.

— Reply to this email directly, view it on GitHub https://github.com/0vercl0k/wtf/pull/192#issuecomment-1959864710, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALIORM4DPNG2W5SYQZGS73YU5Z6VAVCNFSM6AAAAABCJFQPLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJZHA3DINZRGA . You are receiving this because you were mentioned.Message ID: @.***>

0vercl0k commented 7 months ago

All right @jasocrow I think this is ready to go - take a look and if this looks good to you, let's merge this!

Cheers

jasocrow commented 7 months ago

Everything looks good to me, happy if you're ready to merge. Thanks!!

jasocrow commented 7 months ago

Oh, I just noticed one small thing, looks like the link to Kyle's site in linux_mode/README.md is not showing correctly.

0vercl0k commented 7 months ago

Great catch, thank you - fixed. Ok cool, thanks for taking a look - will merge this next week!

Cheers

On Fri, Mar 29, 2024 at 6:52 AM jasocrow @.***> wrote:

Oh, I just noticed one small thing, looks like the link to Kyle's site in linux_mode/README.md is not showing correctly. Screenshot.2024-03-29.at.8.50.33.AM.png (view on web) https://github.com/0vercl0k/wtf/assets/153134172/54828663-49d3-44fc-9681-9e0eafbf6dac

— Reply to this email directly, view it on GitHub https://github.com/0vercl0k/wtf/pull/192#issuecomment-2027276161, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALIORJ5ZWCRHYLWRQVNHB3Y2VW3XAVCNFSM6AAAAABCJFQPLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXGI3TMMJWGE . You are receiving this because you were mentioned.Message ID: @.***>

0vercl0k / wtf

Add support for Linux userland ELF snapshots and fuzzing #192