System Call Scan on Ubuntu 18.04 Guest

[tushariitr1]

I am trying to perform System Call Scan using Drakvuf on Ubuntu 18.04 guest. I created the rekall profile for the linux guest as mentioned on the drakvuf page. When I starts the scan it generates the System Calls but after 30 seconds, both the VM and the hypervisor hangs and breaks. I repeated the above process around 4 times, but getting the same results.

system configuration: Xen : 4.12 Drakvuf : Latest version Libvmi : Latest version

[tklengyel]

You'll need to provide more information, such as the debug logs of both DRAKVUF and Xen, and also your command line boot arguments you used to boot Xen.

[tushariitr1]

Debug Log of Xen for Ubuntu 18.04 guest. qemu-system-i386: -usbdevice tablet: '-usbdevice' is deprecated, please use '-device usb-...' instead qemu-system-i386: -serial pty: char device redirected to /dev/pts/7 (label serial0) qemu-system-i386: terminating on signal 1 from pid 3918 (/usr/local/sbin/libvirtd)

Command line boot arguments you used to boot Xen: GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=16384M,max:16384M dom0_max_vcpus=8 force-ept=1 ept=ad=0 hap_1gb=false hap_2mb=false altp2m=1"

[tklengyel]

Sorry but that's not the Xen debug log, you'll need to attach a serial cable and boot Xen with the debug output enabled. You probably will also have to recompile Xen with the debug options enabled. See https://wiki.xenproject.org/wiki/Xen_Serial_Console and https://wiki.xenproject.org/wiki/Debugging_Xen and https://github.com/tklengyel/drakvuf/wiki/Debugging-DRAKVUF

[tushariitr1]

Thanks. I will look into it and get back to you soon.

[Vladinuto]

Good afternoon, I'm working along with @tushariitr1 on this issue and I'm trying to collect the debug logs from our server. So far, I've enabled debug mode on Xen, and recompiled drakuf with debug options enabled.

My question is, will Xen still be able to write the logs when it oops into a Virtual Serial Port ? Our hardware does not have a serial port available, but on the Xen documentation it says that a Virtual Serial Over Lan (SOL) can be used to extract logs. If so, any recommendations/packages/programs that might help on this?

Thanks!

-CR.

[tklengyel]

SOL hasn't worked for me in a couple years, so I would recommend getting a physical serial cable attached. If the motherboard has no serial pins you could try a PCIe serial card.

[Vladinuto]

Hello,

Please find the Xen logs attached. Any hints/helps would be appreciated.

Thanks! -CR Reboot Hyp breaks but no data after break.TXT

[tklengyel]

Your logs don't contain anything useful other than that you are running on 4.12.2-pre. You should always use the version of Xen that's tagged in the git submodule, which is currently 4.13.

[tklengyel]

Also, perhaps compiling Xen with the debug options enabled and using loglvl=all guest_loglvl=all would tell us more. But from the looks of it it just locks up on your machine so I wouldn't be surprised if there wouldn't be any more debug output. Does the system reboot (crash) or just sits there hanging?

[Vladinuto]

I think we do have the loglvl=all guest_loglvl=all enabled as a command line argument (line 5 on the .txt). The hypervisor just stops responding and we have to manually restart the server.

[tklengyel]

Ah yea, you are right, you already have that set. So in that case what this sounds like is a deadlock within Xen with some of its internal locking mechanism. We've ran into deadlocks like that before with Xen's PML feature but you have ept=ad=0 set which disables it completely, so this seems to be something else. Unfortunately these are extremely hard bugs to find. I find it odd that we haven't ran into it so far so I wouldn't be surprised if this was somehow related to the specific hardware you have. Can you reproduce the issue on other hardware? Also, can you upgrade to Xen 4.13 and see if it goes away?

[Vladinuto]

We've tried with different VMs configurations on our current setup and this is the pattern that we've seen: Linux 2 CPUs: Worked around 80% of the time. Linux 2+ CPUs: Always breaks. We are looking into updating one of our servers to 4.13 and come back to report the results.

[Vladinuto]

This is the output of a 4 CPU VM with only syscall plugin enabled and debug enabled for Drakvuf. Drakvuf Debug.log

[tklengyel]

I can't reproduce the problem on my end, I've ran the standard CI tests with 4 vCPU debian-stretch VM and also just letting drakvuf run on the domain for an extended period, I don't see any crashes.

[Vladinuto]

Hi @tklengyel Thanks for the help so far. We've tried various settings but the issue still persists. Could you share the Xen, Drakvuf & LibVMI versions, as well as hypervisor's hardware configuration you used for the testing?

Thanks again!

-CR

[tklengyel]

The latest master branch and the latest versions of the submodules. Setup as described on the website. Hardware is an i3-7100.

[Vladinuto]

Good evening,

As an update, we updated our servers to the release version of Xen 4.13.0, and we are seeing the same behavior with Linux virtual machines.

The server crashes and needs to be manually restarted. As we are working remotely, its a little difficult to manually restart the servers on every crash but the same behavior is on 4.13 as well.

Regards,

-CR

[tklengyel]

Thanks for the update. Did you also try updating your BIOS and CPU microcode? This is most likely a hardware issue. You could also try to post to the xen-devel mailinglist to see if anyone has experience with your particular hardware, maybe its a known bug.

[Vladinuto]

We have not tried updating bios/Micro code but we'll look into it and see if anything changes.

Quick note, we create VMs using files and not LVMs. Would this make any difference?

Thanks!

[tklengyel]

No, that shouldn't make a difference. Having Xen deadlock on you is a much deeper issue then that.

[Vladinuto]

Hi Tamas, We have used the latest drakvuf, as well as Xen 4.15 but the same issue persists. The hypervisor restarts after a couple of seconds of starting drakvuf with only the syscall plugin enabled.

Hypervisor: Xen 4.15, Libvirt 5.4.0, Latest drakvuf Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz Latest bios/Micro code

VM: Ubuntu 18.04 , 4 CPU , 4GB Ram

Drakvuf works without any issues when using a Windows 10 VM for any length of time tho.

Any ideas / thoughts on how to go about fixing/debugging/etc this type of issue?

Thanks in advance!