gammaray

gammaray is a system implementing disk-based introspection for virtual machines. It thus far works with VMs that have disks virtualized via QEMU; however, this limitation is only due to the scarcity of developer time. Conceptually, and practically, gammaray can perform introspection with any source of raw disk writes. The instructions below assume an Ubuntu 12.04 LTS host, although they should be similar for most distributions of Linux.

Installation Procedure

Follow the non-optional dependencies instructions
Follow the installation procedure for gammaray
Follow the example virtul disk creation instructions
Follow the instructions for running the gammaray pipeline
Check that things are working by using gray-fs and tail from outside the guest on a known text file. Add lines within the guest and ensure that they eventually become visible via the gray-fs mounted file system.
```
gray-fs /mnt/disk -d -s disk.raw
```
Replace /mnt/disk with whatever folder you'd like to use as the read-only view into the real-time guest file system. gray-fs doesn't properly parse command line options yet, so type the command almost exactly as it appears above (-s -d force single-threaded, debug mode).
- Example check command outside the VM guest
```
tail /mnt/disk/home/wolf/test.txt
```
- Example check command within the VM guest
```
while 1; do echo 'test' >> test.txt; sleep 3; done 
```

License

All source code, documentation, and related artifacts associated with the gammaray open source project are licensed under the Apache License, Version 2.0.

A copy of this license is reproduced in the LICENSE file, and the licenses of dependencies and included code are enumerated in the NOTICE file.

Dependencies

The following libraries are needed to build and execute gammaray (most developers can skip the Python libraries):

hiredis [BSD 3-clause] - the Redis client-side library
```
sudo apt-get install libhiredis-dev libhiredis0.10
```
and install a Redis server if you don't have one already:
```
sudo apt-get install redis-server
```
event [BSD 3-clause] - the libevent event-driven networking library
```
sudo apt-get install libevent-dev libevent-2.0.5 
```
bson [Optional, BSD 3-clause] - Python BSON library
redis-py [Optional, MIT] - Python hiredis wrapper
libfuse [GNU LGPL] - the filesystem in userspace library
```
sudo apt-get install libfuse-dev
```

The Python libraries are optional if you want to write or execute Python monitors consuming gammaray's publish-subscribe stream of file-level updates.

In addition to libhiredis and libevent, gammaray requires a slightly modified version of QEMU. Clone the QEMU repository, apply a patch for gammaray support, and then compile a gammaray-friendly QEMU.

Get the official QEMU source tree

git clone git://git.qemu-project.org/qemu.git

Checkout a specific tagged commit to apply our patch cleanly
```
git checkout v1.4.0
```
Apply the patch (located in src/patches) to your checked out QEMU tree. This patch is in the gammaray source tree binary_tracing_block_qemu_1-4-0.patch
```
git apply binary_tracing_block_qemu_1-4-0.patch
```

Configure QEMU, remember to change the prefix as needed

./configure \
--enable-system \
--disable-user \
--enable-kvm \
--enable-trace-backend=stderr \
--target-list='i386-softmmu i386-linux-user x86_64-linux-user x86_64-softmmu' \
--prefix=/home/wolf/qemu_bin

Make, and make install QEMU.
```
make -j2
make install
```
QEMU binaries with the patch compiled in should be within the prefix folder, specifically inside the bin subfolder.

Installing gammaray

Ensure all dependencies are installed already

git clone gammaray's source tree

git clone https://github.com/cmusatyalab/gammaray.git

Bootstrap your source tree
```
./bootstrap.sh
```
Run configure
```
./configure
```
If you want to create a statically linked version of everything, use the following configure and make options (reference: Re: Building all static):
```
./configure LDFLAGS="-static" LIBS='-lrt'
make LDFLAGS=-all-static
```
If you want to compile with a custom library, and statically link it in for development purposes, the following should work (replace paths):
```
./configure LDFLAGS='-L/home/wolf/Dropbox/Projects/hiredis -static' LIBS="-lrt" CPPFLAGS="-I/home/wolf/Dropbox/Projects/hiredis"
make LDFLAGS="-all-static -L/home/wolf/Dropbox/Projects/hiredis/"
```
Run make
```
make -j 2
```
All binaries will now be built and placed in the bin folder at the top-level directory of the project.
[Optional] Run make install (if you want)
```
sudo make install
```

Description of Components

gammaray is organized as a set of tools, described below:

gray-crawler - used to index a disk in preparation for introspection
- This tool is usually used offline, although it can be used online as well
gray-ndb-queuer - used to asynchronously queue writes for analysis
- This tool interfaces with a stream of writes, soon-to-be an NBD endpoint
gray-inferencer - used to analyze queued writes and perform introspection
- This tool loads and maintains metadata in-sync with a live disk
gray-fs - used to produce a FUSE file system view of the in-sync metadata
- This tool provides a read-only FS without mounting the real guest FS

gammaray Pipeline

The exact steps are enumerated below, but at a high level you must crawl the disk you wish to introspect, load metadata from that crawl for run-time introspection, and attach a copy of the write stream to that disk at run-time to the inferencing backend. Currently, this is coordinated via a named pipe and Redis. In the future, the named pipe is being replaced by NBD. None of the tools auto-daemonize as of this writing.

Crawl the disk that you wish to introspect using gray-crawler
```
gray-crawler disk.raw disk.bson
```
Setup a named pipe to receive raw disk writes to the gray-ndb-queuer
```
mkfifo disk.fifo
```

Run the gray-ndb-queuer and let it read from the named pipe

gray-ndb-queuer disk.bson disk.fifo 4 1>queuer.log 2>queuer.error.log &

Run gray-inferencer and wait for it to load metadata from gray-crawler
```
gray-inferencer disk.bson 4 disk_test_instance &
```
Run QEMU with this disk redirecting stderr output to the named pipe
```
/path/to/custom/qemu-system-x86_64 \
-enable-kvm \
-cpu kvm64 \
-smp cores=1,threads=1,sockets=1 \
-drive file=disk.raw,if=virtio,aio=native \
-m 1024 \
-display vnc=127.0.0.1:1 \
-trace events=events \
-redir tcp:2222::22 2> disk.fifo &
```
Remember to use the specially built QEMU when installing dependencies (ie replace the path in the above command). Also, ensure you have an events file that turns on tracing for the event of interest: bdrv_write. The events file's contents should have only that event name on a single line by itself (tracing multiple events will mangle the binary tracing needed for disk introspection).

Example Creation of gammaray-Supported Disk Layout

If you're sufficiently confident with the installer of your OS of choice, feel free to skip the steps below. Otherwise, it might be easier to setup a known good partition configuration with a host system and then install the guest OS into the pre-existing partition.

Use dd or another suitable command to preallocate the raw disk image
```
dd of=disk.raw seek=$((1024*1024*1024*5)) count=0 bs=1
```
Create a partition table
```
parted -s disk.raw mklabel msdos
```
Create a single primary partition taking up the entire image
```
parted -s disk.raw mkpart primary ext4 1 $((1024*5))
```
Make the partition visible to your host as a block device
```
sudo kpartx -av disk.raw
```
Format the partition as ext4 [replace loop0 with output from kpartx command]
```
sudo mkfs.ext4 /dev/mapper/loop0p1
```
Remove the visible partition and block device from your host
```
sudo kpartx -dv disk.raw
```
Boot the instance with install media (example: ubuntu-12.04.2-server-amd64.iso) and the new drive attached
```
qemu-system-x86_64 disk.raw -cpu kvm64 -cdrom ubuntu-12.04.2-server-amd64.iso
```
Using Manual Partitioning at the disk setup stage, select the first partition to be used as ext4 and mount point '/'
Then just finish partitioning and continue with the installation procedure

Currently Supported

File Systems
- ext4
- NTFS
VMMs
- KVM/QEMU
Number of Disks
- Single attached disk
Number of File Systems
- Single introspected file system
Disk Formats
- raw

cmusatyalab / gammaray

readme