search

SerenityOS / serenity

The Serenity Operating System 🐞
https://serenityos.org
BSD 2-Clause "Simplified" License
30.39k stars 3.18k forks source link

Enlighten syzkaller to fuzz the Serenity OS sycall boundary. #1025

Open bgianfo opened 4 years ago

bgianfo commented 4 years ago

Fuzzing the syscall boundary via a security based fuzzer like syzkaller would improve the security of Serenity substantially. https://github.com/google/syzkaller

For more information the talk on syzbot for the linux kernel is very good. https://www.youtube.com/watch?v=qrBVXxZDVQY

ADKaster commented 4 years ago

Looks like that tool is written in Go, so there needs to be at least a Go port before that can happen, unless i'm misreading the Makefile in their repo. Definitely worth doing if someone has the time/knowledge though!

ADKaster commented 3 years ago

In fact, I think last year I was misreading how syzkaller works. A closer look at the source and the FreeBSD docs page suggests that they only use go on the host machine, and that the executor program that's run on the target is generated C-looking C++ code?

https://github.com/google/syzkaller/blob/master/docs/freebsd/README.md

Would probably need some help from the syzkaller team to understand how to add serenity though.

Probably would involve some work here: https://github.com/google/syzkaller/blob/master/sys/targets/targets.go

and a new folder under sys with a list of syscalls for serenity, and a few new header files/changes to the executor folder?

bgianfo commented 3 years ago

Linking this related PR, where progress towards real syzkaller support is being made. See: #8943

bgianfo commented 3 years ago

I've stashed some of my work towards adding syzkaller support for SerenityOS here in case anyone wants to collaborate: https://github.com/bgianfo/syzkaller/tree/serenityos-support

HerrSpace commented 3 years ago

Ohai. I used last weekend to build a syzkaller serenity POC that kinda works. I pushed my syzkaller branch here: https://github.com/HerrSpace/syzkaller/commits/serenityos

It compiles, but everything in the last commit needs to be fixed before this can be upstreamed. In particular I'm interested in somebody helping me fix issues on the serenity side, allowing me to undo the hacks in the executor/ subdir of that last commit.

I'm currently building via make TARGETOS=serenity TARGETARCH=386 SOURCEDIR=/Users/space/repos/upstream/serenity/Kernel/

I use syzkaller.cfg:

{
    "target": "serenity/386",
    "http": "127.0.0.1:56741",
    "sshkey": "/Users/space/.ssh/id_old_priv",
    "workdir": "/Users/space/sk_serenity/",
    "kernel_obj": "/Users/space/repos/upstream/serenity/Build/i686/Kernel/",
    "kernel_src": "/Users/space/repos/upstream/serenity/Kernel/",
    "syzkaller": "/Users/space/repos/go/src/github.com/google/syzkaller",
    "procs": 2,
    "type": "qemu",
    "cover": false,
    "image": "/Users/space/repos/upstream/serenity/Build/i686/_disk_image",
    "vm": {
        "count": 2,
        "cpu": 2
    }
}

And run via syz-manager -config=/Users/space/serenity/syzkaller.cfg

I also pushed my serenity branch here. There are some convenience hacks here in the image rebuilding script, copying in the current executor build, ssh host keys, authorized_keys, fixing the dropbear port signiture... https://github.com/HerrSpace/serenity/commit/7fb97c7c0e3fc86787e87b25e30fdfb30bb1522f#diff-b21f1e4dd266b3c01933be6d857e563c527bd2e430c898a9b2f5c8fb012370d9R1

^ I also included a small c++ programm in Ports/dummy summing up some of the executor issues I've encountered so far. Might be easier to start hacking on that without needing to bootstrap syzkaller.

bgianfo commented 2 years ago

Linking #8943 since it was progress towards syzkaller.

HerrSpace commented 2 years ago

Since we talked about this yesterday one more important point about the state of syzkaller here:

If somebody fixed the serenity side and we rebase and eventually upstream everything we will still not have coverage guided fuzzing. As serenity isn't supported as a golang target we have to use syzkallers HostFuzzer mode. Normally you start syz-manager(go) on the host and syzkaller will take care of starting syz-fuzzer(go) and syz-executor(c++) inside the quest. In HostFuzzer mode there is no syz-fuzzer, enabling it to work on targets with no golang target support. However HostFuzzer is a giant hack and doesn't support a bunch of things. As a result there is no plumbing set up to extract coverage from the guest to the syz-manger.

There are a bunch of ways to fix this for serenity:

However we should focus on first cleaning up what we have before worrying about we worry about this. :^)

bgianfo commented 2 years ago

@HerrSpace FYI I spent some time on your dummy package tonight, it seems that std::min() now works as expected. I also sent out #11392 to implement _setjmp and _longjmp as needed by the syzkaller executor.

HerrSpace commented 2 years ago

I rebased my syzkaller and serenity branches and removed the _setjmp/_longjmp changes.

To build syz-executor I currently manually build it after the rest of syzkaller with this hack: /Users/space/repos/upstream/serenity/Kernel/../Toolchain/Local/i686/bin/i686-pc-serenity-gcc -o ./bin/serenity_386/syz-executor executor/executor.cc -std=c++20 -I /Users/space/repos/upstream/serenity/Kernel/../Build/i686/Root/usr/include/ -I /Users/space/repos/upstream/serenity/Kernel/../Userland/Libraries/ -O2 -pthread -Wall -Werror -Wparentheses -Wunused-const-variable -Wframe-larger-than=16384 -DGOOS_serenity=1 -DGOARCH_386=1 -DHOSTGOOS_darwin=1 -DGIT_REVISION=\"82209f56b7b05a04c107405601a9a45adac60f65+\" -Wno-error=deprecated-declarations -lsystem

^ I think i might need to declare the std lib somehow for instance to fix the std:: stuff, but couldn't figure out how ^^

HerrSpace commented 2 years ago

Dropbear Port Update: https://github.com/SerenityOS/serenity/pull/13827

HerrSpace commented 2 years ago

OpenSSH Port Update and OpenSSH server fix: https://github.com/SerenityOS/serenity/pull/13846

HerrSpace commented 2 years ago

I've rebased and cleaned up my work here for easier reproduction. I tested on a debian box.

Assuming you already have a current and compiled serenity checkout and you only want to work on the repro, then you can skip the Build serenity disk image for syzkaller for now.

Build serenity disk image for syzkaller

add key /root/.ssh/authorized_keys

if [ -z "$PUBKEY" ]; then echo "Please set the PUBKEY env var to your ssh pubkey" exit 1 fi mkdir -p mnt/root/.ssh chmod 700 mnt/root/.ssh/ echo "$PUBKEY" > mnt/root/.ssh/authorized_keys chmod 600 mnt/root/.ssh/authorized_keys chown -R 0:0 mnt/root/

the scp binary from the openssh port ends up here. scp invokes

another scp on the receiving end, expecting it to be in the path

echo "export PATH=/usr/local/bin/:$PATH" > mnt/root/.shellrc

enable openssh server

cat <> mnt/etc/SystemServer.ini

[SSHServer] Executable=/usr/local/sbin/sshd Arguments=-D KeepAlive=1 SystemModes=text,graphical EOF

not sure why these keys are only sometimes go+r

chmod 600 mnt/etc/ssh/*key

boot into text mode (when booted from grub image)

cat < "$SERENITY_SOURCE_DIR"/Meta/grub-mbr.cfg timeout=0 menuentry 'SerenityOS (text mode)' { root=hd0,1 multiboot /boot/Prekernel fbdev=off root=/dev/hda1 module /boot/Kernel } EOF

popd echo "done"

- install `grub-pc`
- `cd Build/i686` and ` PUBKEY="<your pubkey>" ninja grub-image`

### Build syzkaller
- clone [my syzkaller fork](https://github.com/HerrSpace/syzkaller/tree/serenityos)
- checkout the `serenityos` branch
- `make TARGETOS=serenity TARGETARCH=386 SOURCEDIR=~/repos/serenity/Kernel/`
- this should compile just fine

### The repro
- start undoing the changes in executor/executor.cc from the last commit on that branch (https://github.com/HerrSpace/syzkaller/commit/799fbf7fc9ce3c034e77cbd1a9b2fd98f05c8a50)
- one of the bugs I encountered:
  - https://github.com/SerenityOS/serenity/issues/13867 (workaround available)
  - https://github.com/SerenityOS/serenity/issues/13869
  - https://github.com/SerenityOS/serenity/issues/13879
### Actually running the whole thing
- create an empty workdir for szkaller to work with in the following file
- create a config file, fix paths, ...:

{ "target": "serenity/386", "http": "127.0.0.1:56741", "sshkey": "/home/space/.ssh/id_ed25519", "workdir": "/home/space/sk_serenity/", "kernel_obj": "/home/space/repos/serenity/Build/i686/Kernel/", "kernel_src": "/home/space/repos/serenity/Kernel/", "syzkaller": "/home/space/repos/syzkaller", "procs": 2, "type": "qemu", "cover": false, "image": "/home/space/repos/serenity/Build/i686/grub_disk_image", "vm": { "count": 2, "cpu": 2 } }


- Run `./bin/syz-manager -config=<path to your config file> -debug`
HerrSpace commented 2 years ago

Given the workaround in https://github.com/SerenityOS/serenity/issues/13867 and fix for https://github.com/SerenityOS/serenity/issues/13869 I was able to remove all but this tiny hack today :^) https://github.com/HerrSpace/syzkaller/commit/ec41070750099b11bdfa5d8eecbc3a618281d1cc

HerrSpace commented 2 years ago

Resolved all serenity side blockers for upstreaming. The port is not quite finished though, so I'll need to work more on it when I find time.