Hypervisor crash on Guest reboot

[v-p-b]

I recently encountered several crashes at the hypervisor level by foolishly executing the following steps:

• Attach DRAKVUF to Guest
• Reboot guest (manually or due to a crash)
• Exit DRAKVUF with ^C

Unfortunately the most recent crash resulted in some kind of FS corruption so I can't attach more logs or do retests right now. Also I'm running a custom fork of DRAKVUF but I don't think my changes cause the problem (although I can't rule this out).

My guess is that things go wrong when DRAKVUF handles SIGINT, because everything seem to work normal until the signal is sent: Guest is rebooting/running, DRAKVUF is logging, although a lot of "Write memaccess" messages appear.

My main question is if there is a way to properly stop DRAKVUF in case a Guest reboot happens?

[tklengyel]

Are you creating VM clones too, or simply running DRAKVUF? Either way, a guest reboot should definitely not crash Xen, but I've seen mem_sharing do that too. In such cases unfortunately your best bet is to make a debug build of Xen (Config.mk -> debug := y) and create a serial port for Xen to put the debug output to. If you have a vPro processor with AMT, you can use the serial-over-lan feature for that:

GRUB_CMDLINE_XEN_DEFAULT="loglvl=all guest_loglvl=all console_to_ring com1=115200,8n1,amt console=com1"

I'll check if I can reproduce this on my side too.

[tklengyel]

So I just tested this on my rig and Xen didn't crash. DRAKVUF did segfault at the end which I'll look into but it's not really critical. I tried both with cloning and without and Xen didn't crash.. But at the same time I'm running Xen 4.7 at the moment so YMMV. A crash log would definitely help figuring this out.

[tklengyel]

As for catching a domain shutdown/restart event, there are two possibilities: 1) catch the in-guest kernel function responsible for restart/shutdown and use that information as a signal for DRAKVUF to stop as well. 2) use Xenstore watches on @releaseDomain events to determine when Xen is stopping the domain (see http://xenbits.xen.org/docs/4.3-testing/misc/xenstore.txt). I have a tool that uses Xenstore watches which you may be able to tweak for this (https://github.com/tklengyel/drakvuf-deployer/blob/master/server/server.c).

[v-p-b]

Thank you for looking into this! I'll try to recover my logs and update the ticket (probably next week).

[v-p-b]

Since this is probably a problem in my fork and I can't find any useful logs (syslog was unable to flush its last msg to disk, etc.) I'm closing this issue for now.