search

tklengyel / drakvuf

DRAKVUF Black-box Binary Analysis
https://drakvuf.com
Other
1.04k stars 249 forks source link

VNC screen not refreshed while DRAKVUF is running #238

Closed rstocktox closed 5 years ago

rstocktox commented 7 years ago

Hi,

Testing a fresh install of the current DRAKVUF version on an HP Z40 desktop server (16GB RAM, 8 cores, Debian Jessie) found that whatever plugin I ran the VNC screen is not refreshed until DRAKVUF stops running.

I'm currently using TightVNC as my VNC client from my Windows laptop and it really works like a charm connecting to the same DomU in the same desktop server (without using DRAKVUF) or even connecting to another server with XEN v4.7 and old DRAKVUF versions.

Seems strange to me and, sincerely, I don't see clearly where the problem lies, as said before DRAKVUF + Xen 4.7 didn't exhibit this behaviour on another server.

I tried and changed lot of things: enabling/disabling logs on the XEN kernel command line, increasing/decreasing DomU memory, using 1 or 2 vcpus for the DomU, increasing shadow memory DomU config option, enabling/disabling DRAKVUF debug, compiling and enabling just 1 DRAKVUF plugin... but unfortunately nothing changed

I think it's worth to mention that although the screen not been refreshed the DomU seems to work/run as usual (tested enabling the file delete plugin and deleting some files in a "blindly" way).

Would you be so kind to shed some light on this issue?

Generic system info:

dev@xen-devel:~/drakvuf$ cat /etc/debian_version
8.6

dev@xen-devel:~/drakvuf$ uname -a
Linux xen-devel 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux

dev@xen-devel:~/drakvuf$ sudo xen-detect
Running in PV context on Xen v4.8.

dev@xen-devel:~/drakvuf$ src/drakvuf
DRAKVUF v0.4-4095577

xl info output before starting the DomU:

dev@xen-devel:~/drakvuf$ sudo xl info
host                   : xen-devel
release                : 3.16.0-4-amd64
version                : #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
machine                : x86_64
nr_cpus                : 8
max_cpu_id             : 7
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 3408
hw_caps                : b7ebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 15784
free_memory            : 11474
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 8
xen_extra              : .0
xen_version            : 4.8.0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder vga=gfx-1024x768x16 consoleblank=0 dom0_mem=4096M,max:4096M dom0_max_vcpus=2 dom0_vcpus_pin=true hap_1gb=false hap_2mb=false altp2m=1 loglvl=all guest_loglvl=all flask_enforcing=1
cc_compiler            : gcc (Debian 4.9.2-10) 4.9.2
cc_compile_by          : dev
cc_compile_domain      :
cc_compile_date        : Wed Jan 11 11:43:08 CET 2017
build_id               : 407d7cdcfecb42ee39d536f94590cb6b925535f4
xend_config_format     : 4

DomU configuration file (Windows 7 SP1 x64 guest):

dev@xen-devel:~/drakvuf$ more ~/win7-sp1-x64-bios.cfg
name = "win7-sp1-x64-bios"

arch = 'x86_64'

builder ='hvm'
memory  = 3000
maxmem  = 3000

vcpus   = 2
maxcpus = 2

hap  = 1
acpi = 1
altp2mhvm = 1

shadow_memory = 16

seclabel='drakvuf:vm_r:drakvuf_domU_t'

on_poweroff = 'destroy'
on_reboot   = 'destroy'
on_crash    = 'destroy'

tsc_mode="native"

vif = [ 'type=ioemu,model=e1000,bridge=xenbr1.200,script=vif-openvswitch,backend=0,mac=00:04:5B:48:62:60' ]
disk = [ 'phy:/dev/analyzers/win7-sp1-x64-bios,hda,w', 'file:/opt/iso/win7-prof-n-sp1-x64-en.iso,hdc:cdrom,r' ]

boot="cd"

vnc        = 1
vnclisten  = '0.0.0.0'
vncpasswd  = ''
vncunused  = 0
vncdisplay = 2

serial = 'pty'
usb       = 1
usbdevice = 'tablet'
vga       = 'stdvga'
videoram  = 16

xl create output:

dev@xen-devel:~/drakvuf$ sudo xl -vv create ~/win7-sp1-x64-bios.cfg
Parsing config from /home/dev/win7-sp1-x64-bios.cfg
got a tsc mode string: "native"
domainbuilder: detail: xc_dom_allocate: cmdline="(null)", features="(null)"
domainbuilder: detail: xc_dom_kernel_file: filename="/usr/local/lib/xen/boot/hvmloader"
domainbuilder: detail: xc_dom_malloc_filemap    : 463 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.8, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x7bd64
xc: detail: ELF: memory: 0x100000 -> 0x17bd64
domainbuilder: detail: xc_dom_mem_init: mem 2984 MB, pages 0xba800 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0xba800 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: xc_dom_malloc            : 5968 kB
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000003d3
xc: detail:   1GB PAGES: 0x0000000000000001
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x7c at 0x7f7297cde000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 -> 0x17c000  (pfn 0x100 + 0x7c pages)
xc: detail: ELF: phdr 0 at 0x7f7297c62000 -> 0x7f7297cd41c0
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x17c+0x40 at 0x7f7297c9e000
domainbuilder: detail: xc_dom_alloc_segment:   System Firmware module : 0x17c000 -> 0x1bc000  (pfn 0x17c + 0x40 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x1bc+0x1 at 0x7f7297e21000
domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x1bc000 -> 0x1bd000  (pfn 0x1bc + 0x1 pages)
domainbuilder: detail: alloc_pgtables_hvm: doing nothing
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x1bd000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: clear_page: pfn 0xfefff, mfn 0xfefff
domainbuilder: detail: clear_page: pfn 0xfeffc, mfn 0xfeffc
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 5975 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 463 kB
domainbuilder: detail:       domU mmap          : 756 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff001
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_qmp.c:287:qmp_handle_error_response: received an error message from QMP server: Could not set password

Output of xl info after DomU started:

dev@xen-devel:~/drakvuf$ sudo xl info
host                   : xen-devel
release                : 3.16.0-4-amd64
version                : #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03)
machine                : x86_64
nr_cpus                : 8
max_cpu_id             : 7
nr_nodes               : 1
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 3408
hw_caps                : b7ebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps              : hvm hvm_directio
total_memory           : 15784
free_memory            : 8457
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 8
xen_extra              : .0
xen_version            : 4.8.0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          :
xen_commandline        : placeholder vga=gfx-1024x768x16 consoleblank=0 dom0_mem=4096M,max:4096M dom0_max_vcpus=2 dom0_vcpus_pin=true hap_1gb=false hap_2mb=false altp2m=1 loglvl=all guest_loglvl=all flask_enforcing=1
cc_compiler            : gcc (Debian 4.9.2-10) 4.9.2
cc_compile_by          : dev
cc_compile_domain      :
cc_compile_date        : Wed Jan 11 11:43:08 CET 2017
build_id               : 407d7cdcfecb42ee39d536f94590cb6b925535f4
xend_config_format     : 4

As I said before DRAKVUF was compiled with debug enabled and just the exmon plugin (tested with the filedelete and the syscalls plugins also and achieving the same result):

[...]
-------------------------------------------------------------------------------
DRAKVUF is configured as follows. Please verify that this configuration
matches your expectations.

Host system type: x86_64-unknown-linux-gnu
Build system type: x86_64-unknown-linux-gnu
Installation prefix: /usr/local
-------------------------------------------------------------------------------
DRAKVUF Plugins
Syscalls:   no
Poolmon:    no
Filetracer: no
Filedelete: no
Objmon:     no
Exmon:      yes
SSDTmon:    no
CPUIDmon:   no
Debugmon:   no
-------------------------------------------------------------------------------

DRAKVUF debugging output:

dev@xen-devel:~/drakvuf$ sudo src/drakvuf -t 10 -d win7-sp1-x64-bios -r ~/rekall-profiles/win7-sp1-x64-bios.json -v
DRAKVUF v0.4-4095577
Starting DRAKVUF initialization
Init VMI on domID 2 -> win7-sp1-x64-bios
Reservation increased? 0 with new gfn: 0x3b9bca
Xen altp2m view created with idx: 1 idr: 2
Windows kernel base address is 0xfffff8000265d000
Rekall profile: '_KPCR' has no 'PrcbData' member
Failed to find offset for _KPCR:PrcbData
libdrakvuf initialized
DRAKVUF initializated
Starting plugins
Starting plugin syscalls
Starting plugin syscalls finished
Starting plugin poolmon
Starting plugin poolmon finished
Starting plugin filetracer
Starting plugin filetracer finished
Starting plugin filedelete
Starting plugin filedelete finished
Starting plugin objmon
Starting plugin objmon finished
Starting plugin exmon
Rekall profile: '_KTRAP_FRAME' has no 'Eip' member
Rekall profile: '_KTRAP_FRAME' has no 'Eax' member
Rekall profile: '_KTRAP_FRAME' has no 'Ebx' member
Rekall profile: '_KTRAP_FRAME' has no 'Ecx' member
Rekall profile: '_KTRAP_FRAME' has no 'Edx' member
Rekall profile: '_KTRAP_FRAME' has no 'Edi' member
Rekall profile: '_KTRAP_FRAME' has no 'Esi' member
Rekall profile: '_KTRAP_FRAME' has no 'Ebp' member
Rekall profile: '_KTRAP_FRAME' has no 'HardwareEsp' member
        ntoskrnl.exe @ 0xfffff8000265d000
Reservation increased? 0 with new gfn: 0x3b9bc8
Copied trapped page to new location
Activating remapped gfns in the altp2m views!
                Trap added @ PA 0x2718c8c RPA 0x3b9bc8c8c Page 10008 for KiDispatchException.
Starting plugin exmon finished
Starting plugin ssdtmon
Starting plugin ssdtmon finished
Starting plugin debugmon
Starting plugin debugmon finished
Starting plugin cpuidmon
Starting plugin cpuidmon finished
Beginning DRAKVUF loop
Started DRAKVUF loop
DRAKVUF loop finished
Finished DRAKVUF loop
starting close_vmi_drakvuf
close_vmi_drakvuf finished

That desktop server is built for the only purpose of testing/developing so feel free to ask for any kind of information you need or source code modifying.

Best regards

PD: Modified sentences

tklengyel commented 7 years ago

I've noticed this as well, although I've only seen artifacts on the screen, never the whole screen freezing. This is due using altp2m in Xen, but I haven't been able to track down the underlying reason. I guess swapping around the memory underneath the VM affects the framebuffer somehow. I would recommend using RDP if you really need to access the VM while DRAKVUF is running on it.

rstocktox commented 7 years ago

Thank you so much, Tamas

threatinteltest commented 7 years ago

I have only noticed this a couple of times. Using RDP into the VM, I have not seen it since.

trapframe commented 6 years ago

I am experiencing the same issue on my desktop, as soon as i start Drakvuf the VNC view freeze although the guest is still running fine (checked with RDP) and Drakvuf outputing events. I can 100% repro on 2 hosts OS (same computer, dual boot) one is Kubuntu 18.04 and the other is Mint 19. Have the problem with Xen 4.9 but also with 4.11 using the latest libvmi or the one from Drakvuf (submodule).

On another computer (laptop) i never had this issue, this one is running Ubuntu 16.04 but i am not convinced this is related to it since several people with different hosts have experienced the problem.

The other main difference is that the laptop is booting in legacy mode while the desktop showing the problem boots in UEFI. I will try to replicate the problem on the laptop using the same OS / software configuration than on the desktop.

aoshiken commented 6 years ago

Is not related to UEFI in any way, noticed the same behaviour booting in legacy mode

trapframe commented 6 years ago

doesn't look tied to VNC as i switched the VM config , disabled VNC and enabled SDL => same issue :( SDL viewer is frozen. Running in VNC mode, I also noticed that if you disconnect / reconnect vncviewer then the screen is refresh and you can see that the mouse click & keyboard entries emitted during the frozen state were indeed registered ! so the mouse / keyboard event are transmitted to the guest OS , only the display is not refresh until you reconnect again.

tklengyel commented 6 years ago

This is only related to Xen and altp2m specifically. We don't yet understand exactly why the behavior is not consistent though. Relevant discussion from xen-devel: https://www.mail-archive.com/xen-devel@lists.xenproject.org/msg11760.html

tklengyel commented 5 years ago

Relevant discussion and patches: https://lists.xenproject.org/archives/html/xen-devel/2018-10/msg01425.html https://github.com/razvan-cojocaru/xen/tree/altp2m-logdirty-take1

icedevml commented 5 years ago

@tklengyel Hello, the issue with screen not being refreshed still exists for me. Is there any patch for Xen that could be applied right now? The abovementioned branch is not there anymore.

tklengyel commented 5 years ago

Just look at the comment before yours.

icedevml commented 5 years ago

Ah, okay, the patches are available after clicking on each of the "follow ups". I was not familiar with how to navigate the mailing list archive.

Direct links: https://lists.xenproject.org/archives/html/xen-devel/2018-10/msg01427.html https://lists.xenproject.org/archives/html/xen-devel/2018-10/msg01426.html

tklengyel commented 5 years ago

You can also just grab the latest Xen 4.12 release candidate, these patches are already part of it.

asabellico commented 5 years ago

Dear Tamas, did you already tested drakvuf (and its current libvmi version) with Xen 4.12? I did a fast test some days ago and had some error condition at the beginning. I will further investigate on that, and want to know if you successfully tested it without changing the libvmi version.

Thank you, Alessandro

On Mon, Feb 25, 2019 at 3:54 AM Tamas K Lengyel notifications@github.com wrote:

You can also just grab the latest Xen 4.12 release candidate, these patches are already part of it.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tklengyel/drakvuf/issues/238#issuecomment-466854427, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYiLwouDUntsbm5eK91EF7FfMBId_1Rks5vQ1BTgaJpZM4LkfF7 .

tklengyel commented 5 years ago

Yes, it works just fine AFAICT

icedevml commented 5 years ago

I've updated Xen to tags/4.12.0-rc3 version from git://xenbits.xen.org/xen.git. The whole machine (including host system) hangs just few seconds after I run DRAKVUF and VNC.

tklengyel commented 5 years ago

You should try to attach a serial cable and enable all debug loglevels for Xen. When you get a crashlog on the serial post it to the xen-devel mailinglist.

tklengyel commented 5 years ago

I also just tested 4.12.0-rc3 on my system and it works just fine.

icedevml commented 5 years ago

Update: I've set up Debian Stretch host with 4.12.0-rc4 on another server and it works fine. The previous one still doesn't work, for now let's assume that it's some misconfiguration on my side.