This is the main source tree of HyperTrace, a CADETS-derived implementation of DTrace that supports tracing of virtual machines in real time. HyperTrace works much like present-day DTrace does, with a few minor user-visible changes.
Currently, HyperTrace only supports one mode of operation where the guest
instruments itself and then uses a hypercall instruction (vmcall
or vmmcall
)
to call back a DTrace probe to the host. Different modes of operation are
planned, but not currently supported.
High level changes:
dtraced
which is responsible for transactional
transmission of assembled D programs between host and any number of guests./etc/rc.d
script for dtraced
.target
. Instead of using provider:module:function:name
, you can now use
target:provider:module:function:name
.uload
, usetx
, uuload
, typecast
).dtrace(1)
is now concurrent and asynchronous, which should be invisible to
the user unless HyperTrace mode is enabled.dtrace(1)
command line interface:
-E
, specifying that an ELF file should be sent
to dtraced
and that dtrace
is operating in HyperTrace mode.-Y
, specifying that an ELF file given to the
dtrace
command line utility should be executed.-y
, specifying that an ELF file should be
further processed (applying relocations, etc.) and a new ELF file should be
produced.DTrace options:
resolvers
. Currently only on the command line. It takes
comma-separated arguments, and currently supports hostname
and version
,
signifying that the target
tuple entry should be resolved as either a
machine hostname or version of the operating system.hypertrace
. Specify that we are operating in HyperTrace mode.
Users should not need to set this manually.dtraced
transactional daemon:
/var/ddtrace
./var/ddtrace/sub.sock
that processes can subscribe to./var/run/dtraced.pid
).overlord
mode (usually host machine) or minion
mode (usually
guest machines). If you have better naming suggestions, please let me know!/var/ddtrace/base
.At some point, there will be an ansible
playbook for use on any machine.
Presently, the playbooks we have are private and specific to our testing
infrastructure. Once we've cleaned it up and have one ready for public
consumption, it will be linked here. Until then, these are the steps necessary
to set up HyperTrace manually:
First, build the source code:
make buildworld
make buildkernel
and follow the usual FreeBSD installation procedure from source.
make installkernel && reboot
Once you boot back in:
make installworld && reboot
With that, you should have HyperTrace installed. However, it is not yet ready
for full use. Enable dtraced
in /etc/rc.conf
.
For host machine:
dtraced_enable="YES"
dtraced_type="overlord"
For guest machine:
dtraced_enable="YES"
dtraced_type="minion"
And then run
service dtraced start
You now have a fully functional HyperTrace installation. However, HyperTrace on its own is not terribly useful. You need some VMs. We suggest that for now, you just build a VM from the same source and follow the setup for guests described above using your preferred method.
Once you have a full HyperTrace setup, you are ready to run things! Try these
example scripts to start with (make sure dtraced
is running on both host and
your guests).
Assuming ${TARGET}
is set to the hostname or version of the OS you want to
trace, or a valid glob:
dtrace -E -n "$TARGET:syscall:::entry { @[vmname, execname] = count(); }"
dtrace -E -n "$TARGET:syscall:::entry { @[curthread->td_ucred->cr_ruid, curthread->td_cred->cr_rgid] = count(); }"
dtrace -E -n "$TARGET:syscall:::entry { self->ts = timestamp; } $TARGET:syscall:::entry/self->ts/ { @[vmname, probefunc] = quantize(timestamp - self->ts); self->ts = 0; }"
Each of the above scripts demonstrates a couple of interesting properties. Firstly, we show a simple aggregation smoke-test. The second script shows that HyperTrace can correctly poke into guest memory with the right offsets for guest types, rather than using those on the host. Finally, the last script shows that HyperTrace can deal with collision of thread IDs in the kernel without being confused about namespaces across different guests (and the host). Furthermore, the timestamps gathered in the last script are in fact gathered on the host, so they don't suffer any of the traditional issues when it comes to time on virtual machines.
libdtrace (cddl/contrib/opensolaris/lib/libdtrace/common
) changes:
New files:
dt_elf.c, dt_elf.h
-- Implements conversion from a DTrace program to ELF.dt_typefile.c, dt_typefile.h, _dt_typefile.h
-- Similar to
dt_module.{c,h}
, but intended for use with HyperTrace, as the API differs
a little bit.dt_cfg.{c,h}
-- Control flow graph implemntation for DIF.dt_basic_block.c, dt_basic_block.h, _dt_basic_block.h
-- Basic block
implementation for DIF.dt_ifgnode.c, dt_ifgnode.h, _dt_ifgnode_h
-- helpers for IFG.dt_typing*
-- Implementation of a type inference & checking system on top
of DIF. Similar to D and C type systems.dt_resolver.{c,h}
-- A simple implementation to compare the 5th tuple entry
(standing for target) to various names of the current machine, such as an IP
address, hostname, FreeBSD version, etc.dt_prog_link.{c,h}
-- Linking implementation (applying relocations,
patching up return types, etc.).dt_linker_subr.{c,h}
-- Linking helpers.dt_analysis.{c,h}
-- Various helpers.dt_benchmark.{c,h}
-- DTrace benchmarking API.dt_hashmap.{c,h}
-- Hash map implementation for DTrace.There are plenty of changes in existing libdtrace files as well.
dtraced (cddl/contrib/opensolaris/cmd/dtraced
):
dtraced.c
-- main
function, initialization, main thread.dtraced_chld.{c,h}
-- child management.dtraced_*job.{c,h}
-- processing functions of various jobs.dtraced_connection.{c,h}, _dtraced_connection.h
-- management of dtraced
socket connections.dtraced_directory.{c,h}
-- directory-related functionality.dtraced_dttransport.{c,h}
-- dtraced communication with dttransport.dtraced_errmsg.{c,h}
-- logging functions.dtraced.h
-- main includes.dtraced_lock.{c,h}
-- locking helpers.dtraced_misc.{c,h}
-- various functions that don't fit anywhere else.dtraced_state.{c,h}
-- state management.dttransport (sys/dev/dttransport
):
dttransport.{c,h}
-- simple kernel module to forward DTrace-related
events from the virtio frontend to userspace.dtvirt (sys/cddl/dev/dtvirt
):
dtvirt.{c,h}
-- shim layer to bhyve from dtrace.ko
.virtio-dtrace:
sys/dev/virtio/dtrace/virtio_dtrace.{c,h}
-- virtio dtrace frontend.usr.sbin/bhyve/pci_virtio_dtrace.c
-- virtio dtrace backend.hypertrace (sys/cddl/dev/hypertrace
):
hypertrace.{c,h}
-- a kernel module for safely managing HyperTrace state.hypertrace_map.c
-- probe map helpers.packet tagging:
sys/kern/subr_mbufid.c
-- Core implementation of packet tagging.sys/sys/_mbufid.h
-- structs/defines.sys/sys/mbufid.h
-- mbuf tagging header.sys/kern/subr_msgid.c
-- light-weight alternative to UUIDs (used for tags).sys/sys/_msgid.h
-- structs/defines.sys/sys/msgid.h
-- msgid header.Packet tagging includes many changes in various existing files revolving around networking, such as tap, tun, bpf, various mbuf copy routines and virtio-net.
dtracedctl (cddl/contrib/opensolaris/cmd/dtracedctl
):
dtracedctl.{c,h}
-- A simple interface to dtraced
.hypercall interface:
sys/amd64/amd64/bhyve_hypercall.S
-- implementation of hypercalls.sys/amd64/include/bhyve_hypercall.h
-- header for hypercalls.This functionality also involves heavy changes to the bhyve kernel module
(vmm.ko
).
Nested page table walk:
sys/cddl/dev/dtrace/amd64/dtrace_vm_subr.c
-- Implements a nested page
table walk suitable for use inside the DTrace probe context.