Proper use of injector

[JordanBoulan]

Hi,

I am trying to figure out how to properly use the injector. Does it work with DLL's? I don't see any manual mapping or fixing of relocs etc in injector code but I am still familiarizing myself.

I created both an .exe and a .dll to try both of which open a simple message box.

I opened a notepad in the vm for injection and run:

sudo injector -r /home/jade/win10_2.json -d 3 -i 6756 -e /home/jade/tst/tst.exe (or .dll)

DRAKVUF injector v0.7-git20200519164913+aa3625d-1 Copyright (C) 2014-2020 Tamas K Lengyel
Injector starting /home/jade/tst/tst.exe through PID 6756 TID: 0

The console just hangs forever and nothing happens. I tried specifying different injection methods as well. I have a feeling I am missing something simple. If I get this working id be willing to write a wiki page or tutorial on using it.

[tklengyel]

-e /home/jade/tst/tst.exe

This is a common misconception about the injector. It does not inject an exe or dll from outside the VM into the VM. It injects a function-call to execute something that's already in the VM.

[icedevml]

The hang itself also looks suspicious - normally you should receive an error within up to few seconds, if the process creation has failed. Could you compile adding --enable-debug to the ./configure command and then start injector with -v switch? You should receive more logs then.

Also, PID passed to the -i switch must be a PID of existing process in guest, e.g. explorer.exe

For -e I would encourage to try -e 'c:\windows\system32\notepad.exe' for a good start.

[JordanBoulan]

Thanks. I was wondering if the file needed to be in the vm already. I thought i looked at where the -e file argument was taken and saw an fopen so I assumed in needed to be in linux. I must have looked at something wrong.

Thanks, I will try as suggested. I was using the pid of a notepad i opened.

New Questions:

Is it setup to work with dll's? I imagine injecting a loadlibrary call instead of createproc one could make it work with a .dll? Or maybe a dll already works with shellcode method?

Finally, I assume it is possible to inject code from outside the vm if you manually mapped the file into the vm's memory and then hijacked a thread for it?

[icedevml]

Is it setup to work with dll's? I imagine injecting a loadlibrary call instead of createproc one could make it work with a .dll?

Could be, but you would have to implement it manually. You can also just run the DLL using rundll32.

Finally, I assume it is possible to inject code from outside the vm if you manually mapped the file into the vm's memory and then hijacked a thread for it?

Check drakvuf doppelganging, this is a special win10-only technique which allows to copy the binary into the guest vm.

[JordanBoulan]

It is still hanging with...

sudo injector -r /home/jade/win10_2.json -d 1 -i 4980 -e 'C:\Windows\system32\notepad.exe' 
DRAKVUF injector v0.7-git20200519164913+aa3625d-1 Copyright (C) 2014-2020 Tamas K Lengyel
Injector starting C:\Windows\system32\notepad.exe through PID 4980 TID: 0

.....

[INJECT] TIME:1590260522.511119 STATUS:PrematureBreak

I tried enabling debugging and reinstalling. The output isn't any different. -v says unrecognized option. I tried an all lower-case vm path as you suggested and the one i posted. The premature break message happens after force closing with Crtl+C.

Guess I need to go in and check whats going on by putting in my own debug messages? Like you said it is strange it just seems to hang and do nothing

[icedevml]

If -v is unrecognized option then you don't have DRAKVUF compiled with debug printouts:

In DRAKVUF directory:

./configure --enable-debug
make

[JordanBoulan]

I tried that multiple times but was still getting unrecognized. had to run make clean first.

[JordanBoulan]

getting lots of output now. I attached the log file. It seems like it is having trouble finding some offsets? and then just gets stuck in a CB loop

out.log

I notice:

1590358060.124331 Failed to find address for symbol KiInitialPCR
1590358060.124343 Failed to find offset for _EPROCESS:Wow64Process
1590358060.124360 Failed to find offset for VadRoot:BalancedRoot
1590358060.124369 Failed to find offset for _MMVAD:LeftChild
1590358060.124378 Failed to find offset for _MMVAD:RightChild
1590358060.124386 Failed to find offset for _MMVAD:StartingVpn
1590358060.124395 Failed to find offset for _MMVAD:EndingVpn
1590358060.124403 Failed to find offset for _MMVAD:u
1590358060.124424 Failed to find offset for _KPCR:PrcbData
1590358060.124454 Failed to find offsets for array of structure names and subsymbols.
1590358060.124471 Failed to find offsets for of bitfield: _MMVAD_FLAGS:MemCommit.
1590358060.124482 Failed to find offsets for of bitfield: (null):(null).
1590358060.124490 Failed to find offsets for of bitfield: _MMVAD_FLAGS:CommitCharge.

and

INT3 Callback @ 0x7ffd14ddc6f4. CR3 0x1d5b6a001.
INT3 received but '(null)' PID (0) doesn't match target process (4300)

I just reran volatility and am still getting failed to find offsets. Why Would that be?

[icedevml]

This is the relevant part of the log:

1590358060.128011 CR3 cb on vCPU 0: 0x1d6c6b002
CR3 changed to 0x1d6c6b002. PID: 4300 PPID: 1180 TID: 4000
1590358060.128220 Breakpoint VA 0x7ffd14ddc6f4 -> PA 0x1f67a36f4
1590358060.128249 Physmap populated? 0
1590358060.128292 Copied trapped page to new location
1590358060.128307 Activating remapped gfns in the altp2m views!
1590358060.128382       Trap added @ PA 0x1f67a36f4 RPA 0x2220096f4 Page 2058147 for entry.
Got return address 0x7ffd14ddc6f4 from trapframe and it's now trapped!
1590358060.128441 CR3 cb on vCPU 1: 0x1c3bca002
1590358060.128472 CR3 cb on vCPU 0: 0x1d5b6a001
1590358060.128493 CR3 cb on vCPU 3: 0x1c08be001
1590358060.128508 CR3 cb on vCPU 2: 0x18397d002
INT3 Callback @ 0x7ffd14ddc6f4. CR3 0x1d5b6a001.
INT3 received but '(null)' PID (0) doesn't match target process (4300)
1590358060.128592 Switching altp2m and to singlestep on vcpu 0
1590358060.128602 CR3 cb on vCPU 2: 0x1aa002
1590358060.128672 CR3 cb on vCPU 3: 0x1c0dff002
1590358060.128700 reset trap on vCPU 0, switching altp2m 0->1

and then the injector is entering some erroneous state and just looping forever.

I would highlight this:

CR3 changed to 0x1d6c6b002. PID: 4300 PPID: 1180 TID: 4000

and this:

INT3 Callback @ 0x7ffd14ddc6f4. CR3 0x1d5b6a001.

For some reason there is a CR3 mismatch: 0x1d6c6b002 != 0x1d5b6a001

https://github.com/tklengyel/drakvuf/blob/57cec3cabfbfc1dc5db288469526a7df3326c733/src/libinjector/win_injector.c#L656-L659

This might be some problem with the recent Windows 10 builds (it it some KPTI problem again?). I will try to reproduce it tomorrow.

Btw. could you point out what Windows 10 build are you using? Is it the recent build number 2004?

[icedevml]

Also, these Failed to find address/offset seem to be unrelated to the problem. It's just a problem with locating some structures but these seem rather unrelated.

[icedevml]

I'm 73.14% sure that this is somehow related to the Kernel Page Table Isolation feature. Comparing these two CR3's, there is a 0x1d6c6b002 - 0x1d5b6a001 = 0x1101001.

Looking at this difference:

1000100000001 000000000001
              ^^^^^^^^^^^^ PCID incremented by 1

looks suspicious :P

Hmm also checks out with this: https://github.com/wbenny/hvpp/blob/master/src/hvpp/hvpp/lib/win32/cr3_guard.cpp#L6-L7 and this mention of these constants: https://msrc-blog.microsoft.com/2018/03/23/kva-shadow-mitigating-meltdown-on-windows/

Yep, so it is very likely that the injector broke due to it's not aware of KPTI (yet).

[tklengyel]

What the real issue seems to be is:

INT3 received but '(null)' PID (0) doesn't match target process (4300)

For some reason when that breakpoint triggers libdrakvuf wasn't able to gather those pieces of information.

[icedevml]

The first part with CR3=0x1d6c6b002 has succeeded, notice PCID = CR3 & 0xFFF = 0x002 which is PCID_KERNEL. In the second part with CR3=0x1d5b6a001 it failed and this has PCID = CR3 & 0xFFF = 0x001 which is PCID_USER. So basically until it was in the kernel mode it worked and it stopped working after it approached user mode.

The explanation may be that the routine for determining the current process PID is walking through _KPRCB, _KTHREAD and _EPROCESS structures and most probably it was not possible to access these information using usermode DTB value, because the KVA shadowing mechanism is employed.

What is worrying is that this problem probably applies to all components of DRAKVUF which do interact with usermode...

I will try to confirm that today and also I will think about some patch.

[icedevml]

Unable to reproduce this on Win10_1909_English_x64, sha256 of the install media: 01bf1eb643f7e50d0438f4f74fb91468d35cde2c82b07abc1390d47fc6a356be.

It injects succesfully and there is no problem.

1590424905.218632 CR3 cb on vCPU 0: 0x3529f002
CR3 changed to 0x3529f002. PID: 3136 PPID: 3112 TID: 1292
...
1590424667.208035 CR3 cb on vCPU 1: 0x87b9e001
INT3 Callback @ 0x7ffef7cd9a84. CR3 0x87b9e001.
RAX: 0x1
Injected PID: 2576. TID: 3312
Waiting for injected process
1590424667.209167 Switching altp2m and to singlestep on vcpu 1
1590424667.209893 reset trap on vCPU 1, switching altp2m 0->1
1590424667.210213 CR3 cb on vCPU 1: 0x3529f002
1590424667.210525 CR3 cb on vCPU 1: 0x87b9e001
1590424667.210575 CR3 cb on vCPU 0: 0xacd43002
Process start detected 2576 -> 0xacd43002
INT3 Callback @ 0x7ffef7cd9a84. CR3 0x87b9e001.
RAX: 0x1
Resumed
1590424667.210931 Removing breakpoint trap from 0x10027a84.
1590424667.211146 Removed memtrap for GFN 0x10027 in altp2m view 1
1590424667.211191 Removed memtrap for GFN 0xff00f in altp2m view 1
1590424667.211201 CR3 cb on vCPU 0: 0x1aa002
1590424667.211426 DRAKVUF loop finished
[INJECT] TIME:1590424667.211630 STATUS:SUCCESS PID:3136 FILE:"C:\windows\system32\notepad.exe" ARGUMENTS:"" INJECTED_PID:2576 INJECTED_TID:3312
Finished with injection. Ret: 2.
Injector freed
Process startup success

OK so seems like KPTI is not actually a problem here and my previous theory was wrong.

[JordanBoulan]

So weird, I am also on 1909 build 18363. However, my vm installed some "auto" updates shortly after I installed 1909. As a result i had to rerun volatility. Could that update be related? My sha256 of iso: 01bf1eb643f7e50d0438f4f74fb91468d35cde2c82b07abc1390d47fc6a356be

It appears the same as yours. So strange. I will try a fresh install today.

I'm also passing through a gpu, seems unrelated but thought i would mention.

[icedevml]

I don't have any updates installed on my test instance, and if you had to rerun volatility then it looks like your kernel was updated, so this might be related.

I will try with updates.

[JordanBoulan]

With a fresh install i get:

jade@jadedeb:~$ sudo injector -r /home/jade/win10.json -d 4 -i 7272 -e 'C:\windows\system32\notepad.exe' -v  
DRAKVUF injector v0.7-git20200519164913+aa3625d-1 Copyright (C) 2014-2020 Tamas K Lengyel
1590428385.136743 drakvuf_init: Rekall WoW64 profile not used
1590428385.136926 drakvuf_event_fd_add fd=18
1590428385.136938 size of list=1
1590428385.136944 regenerating event_fds and fd_info_lookup...
1590428385.136950 new event_fd i=0 for fd=18
1590428385.136958 new fd_info_lookup i=0 for fd=18
1590428385.136965 drakvuf_init: adding event_fd done
1590428385.137177 init_vmi on domID 4 -> win10
1590428385.137772 init_vmi: initializing vmi done
1590428385.137957 init_vmi: initializing vmi paging done
1590428385.294054 init_vmi: initializing vmi OS done
1590428385.294160 Max GPFN: 0x2105b8
1590428385.294188 Max mem set? 0
1590428385.294211 Physmap populated? 0
1590428385.294251 Altp2m enabled? 1
1590428385.294264 Altp2m view X created with ID 1
1590428385.294280 Altp2m view R created with ID 2
1590428385.294350 init_vmi finished
1590428385.294363 Windows kernel base address is 0xfffff8037a400000
1590428385.294371 Failed to find address for symbol KiInitialPCR
1590428385.294383 Failed to find offset for _EPROCESS:Wow64Process
1590428385.294399 Failed to find offset for VadRoot:BalancedRoot
1590428385.294407 Failed to find offset for _MMVAD:LeftChild
1590428385.294415 Failed to find offset for _MMVAD:RightChild
1590428385.294421 Failed to find offset for _MMVAD:StartingVpn
1590428385.294429 Failed to find offset for _MMVAD:EndingVpn
1590428385.294439 Failed to find offset for _MMVAD:u
1590428385.294460 Failed to find offset for _KPCR:PrcbData
1590428385.294488 Failed to find offsets for array of structure names and subsymbols.
1590428385.294498 Failed to find offsets for of bitfield: _MMVAD_FLAGS:MemCommit.
1590428385.294508 Failed to find offsets for of bitfield: (null):(null).
1590428385.294518 Failed to find offsets for of bitfield: _MMVAD_FLAGS:CommitCharge.
1590428385.294528 libdrakvuf initialized
Injector starting C:\windows\system32\notepad.exe through PID 7272 TID: 0
Target PID 7272 to start 'C:\windows\system32\notepad.exe'
1590428385.296415 Found PEB @ 0xad4000. LDR @ 0x7ffd889053c0. INLOADORDER @ 0x9f27b0.
1590428385.296464 Found module OneDrive.exe
1590428385.296517 Found module ntdll.dll
1590428385.296532 Found module wow64.dll
1590428385.296557 Found module wow64win.dll
1590428385.296573 Found module wow64cpu.dll
Failed to get address of kernel32.dll!ResumeThread
Unable to initialize injector functions
Injector freed
Process startup failed
1590428385.296602 close_vmi starting
1590428385.328932 close_vmi finished

Now it looks like an error finding the resumethread offset. I just did a fresh install with our iso after first boot i ran volatility and made a json. Ran vmi-process-list. selected onedrive at random. Then ran injector. All on the first login to the vm... It worked for you?

[JordanBoulan]

Wow. Now after restarting the vm the guid is the same (no updates) but I'm getting the same error as before of null CR3. All i did was restart the vm, didnt reboot xen. I ran volatility twice just to be sure.

out2.log

Mind posting the xl.cfg you used? Also my windows isn't activated but I can activate if relevant.

Overall, I'm so confused. I just did a fresh install with the same iso as you.

[tklengyel]

It's best not to give the VM any network access until you can login and block auto updates from happening.

[icedevml]

Since we have the same ISO, let's also confirm kernel GUIDs:

Windows Kernel found @ 0x2400000
        Version: 64-bit Windows 10
        PE GUID: fc9570f2ab6000
        PDB GUID: e0093f3aef15d58168b753c9488a40431
        Kernel filename: ntkrnlmp.pdb
        Multi-processor without PAE

this is what I've got out of ISO 01bf1eb643f7e50d0438f4f74fb91468d35cde2c82b07abc1390d47fc6a356be when installing with the network cut out. Do you have the same kernel GUID?

Here is my config. If you just use the example provided on drakvuf.com - that should work fine. I would only add that with Windows 10 also worth giving it at least 3 GB RAM (I've seen some issues with swapping when you have less memory).

arch = 'x86_64'
name = "vm-1"
maxmem = 3048
memory = 3048
vcpus = 2
maxvcpus = 2
builder = "hvm"
boot = "cd"
hap = 1
acpi = 1
on_poweroff = "destroy"
on_reboot = "destroy"
on_crash = "destroy"
vnc=1
vnclisten="0.0.0.0"
vncdisplay=1
vncpasswd="pass"
usb = 1
usbdevice = "tablet"
altp2m = 2
shadow_memory = 16
audio=1
soundhw='hda'
vif = [ 'type=ioemu,model=e1000,bridge=drak1' ]
disk = [ "tap:qcow2:/var/lib/drakrun//volumes/vm-1.img,xvda,w", "file:/home/mq/Win10_1909_English_x64.iso,hdc:cdrom,r" ]

[JordanBoulan]

Same GUID. I tried reinstalling xen and repulled everything. Now when i run xl create i just get aborted without any messages. Something must have gone wrong? Tried running with -d but unhelpful. Using same config as before. It was a fresh install of linux.

On debain virt-manager can't even connect to the xen socket. tried installing virt-manager out of curiosity. The debian xen repo creates a bunch of scripts that might not be included in non-packaged install im not sure

Anyway, I'm reverting back to my old installation. Not sure what went wrong. Anybody use debian?

[icedevml]

I also use Debian but I haven't experienced any problems with Xen in that matter previously.

The highest probability is that the problem is related to DRAKVUF, LibVMI or the VM itself. I would not touch Xen at the moment, the first thing is rather to debug the higher level.

[icedevml]

As Tamas said, it would be good to inspect this log line:

INT3 received but '(null)' PID (0) doesn't match target process (4300)

for some reason it failed to determine what is the current process at this stage.

[JordanBoulan]

I'm trying to compile the latest version of again and am getting this strange error now.

  LINK    qemu-edid
  LINK    scsi/qemu-pr-helper
  LINK    qemu-bridge-helper
In file included from /home/jade/drakvuf/xen/tools/qemu-xen-dir/include/qemu/timer.h:4,
                 from /home/jade/drakvuf/xen/tools/qemu-xen-dir/include/qemu/timed-average.h:29,
                 from /home/jade/drakvuf/xen/tools/qemu-xen-dir/include/block/accounting.h:28,
                 from /home/jade/drakvuf/xen/tools/qemu-xen-dir/include/block/block_int.h:27,
                 from /home/jade/drakvuf/xen/tools/qemu-xen-dir/block/file-posix.c:30:
/usr/include/linux/swab.h: In function ‘__swab’:
/home/jade/drakvuf/xen/tools/qemu-xen-dir/include/qemu/bitops.h:20:34: error: "sizeof" is not defined, evaluates to 0 [-Werror=undef]
 #define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
                                  ^~~~~~
/home/jade/drakvuf/xen/tools/qemu-xen-dir/include/qemu/bitops.h:20:41: error: missing binary operator before token "("
 #define BITS_PER_LONG           (sizeof (unsigned long) * BITS_PER_BYTE)
                                         ^
cc1: all warnings being treated as errors
make: *** [/home/jade/drakvuf/xen/tools/qemu-xen-dir/rules.mak:69: block/file-posix.o] Error 1
make: *** Waiting for unfinished jobs....
make: Leaving directory '/home/jade/drakvuf/xen/tools/qemu-xen-build'
make[3]: *** [Makefile:219: subdir-all-qemu-xen-dir] Error 2

edit: updating to sid fixed it.

[tklengyel]

I'm on buster and it works just fine, usually its a problem with your workdirectory being unclean. Running git clean -xdf usually resolves it but in the Xen folder you may also have to manually delete the tools/qemu-xen-remote and tools/qemu-xen-traditional-remote folders.

[tklengyel]

Also, if you were on Xen 4.11 that would explain why it wasn't working properly. You always need to use the version of Xen that's tagged in the submodule. Anything else isn't going to work and may result in strange issues like what you encountered. This is usually due to some bug in Xen that I fix in later versions that don't get backported.

[icedevml]

Maybe some sanity check in DRAKVUF would the job? E.g. checking for Xen version and if it's too low - displaying a warning? Or even checking Xen build commit ID (if possible).

[tklengyel]

That probably would help avoiding issues such as this, yes.

[JordanBoulan]

I got 4.13 installed. But I can't run any vm's, Every time i start a vm about 30 sec into boot start getting watchdog cpu stuck messages and then the whole comp freezes and i have to restart. No response from linux and even my keyboard and mouse turn off (lights).

Xen appears to install ok. The vm boots and i can get to the windows installer first menu but soon after that the whole comp will freeze. I reverted back and reinstalled 3 times and am having the same issue, Nothing in /var/log/.

My process following directions on website: install debian update to sid install dependencies compile install reboot run vm

The reason i tried 4.11 is because that is what debian offers in their packages. Is there some setup script in the packages I need not mentioned on the website? It worked when installed that way. However, I have installed manually before and had it working. So I'm not sure what is going on. Not even sure how to debug because there is nothing in /var/log and the whole comp just freezes and i have to reboot. Is this some bug in the latest version?

Not sure why I seem to get so many issues.

[tklengyel]

Perhaps try to just install it from https://github.com/tklengyel/drakvuf-builds/releases?

[JordanBoulan]

I have a gpu passthrough patch (modified hypervisor vendor id from "XenVmm" for nvidia) I am using. Can I patch xen after installing it somehow?

[tklengyel]

It supports live updating but I never tried it so I can't really tell you whether that's going to work.

[tklengyel]

Also, if you just need to get the guest to not see XenVmm during CPUID you don't really need to patch Xen for that. You can just modify the cpuid config option in the xl config file.

[JordanBoulan]

Are you sure? traps.c has them hardcoded with #defines does the code get redirected to a different file with cpuid config options?

[tklengyel]

It's been a while, so I don't recall exactly. But you can certainly override any cpuid stuff using vm_events, so pretty much using drakvuf's CPUIDMON you can return whatever you want as vendor string: https://github.com/tklengyel/drakvuf/blob/master/src/plugins/cpuidmon/cpuidmon.cpp#L195

[tklengyel]

Default:

root@HVM-domU:/home/x# cpuid -l 0x40000000
CPU 0:
   hypervisor_id = "XenVMMXenVMM"

With -s cpuidmon mode:

root@HVM-domU:/home/x# cpuid -l 0x40000000
CPU 0:
   hypervisor_id = "            "

[JordanBoulan]

Can you provide an example of how to set in config? sorry.

You need to use the xend version and set each bit manually i assume? I don't see option for it in xl config string options. Can you provide an example?

Or should i just use cpuidmon?

[tklengyel]

Would be something like

cpuid =  ["‭1073741824‬:eax=000000000000000000000000000000000,ebx=000000000000000000000000000000000,ecx=000000000000000000000000000000000,edx=00000000000000000000000000000000"]

[tklengyel]

It's worth a shot to see if you can override it from the config file, then you don't need an active agent catching and overwriting the cpuids every time. But if the config option doesn't work, the active agent route is an option, you probably only need it while the system boots anyway, I doubt the nvidia driver would constantly check the cpuid during runtime.

[JordanBoulan]

I finally got 4.13 installed and working. Interesting bug now...

The injector works when i use notepad and explorer.exe. However it fails when i try to start notepad from a different process (OneDrive.exe) with message:

jade@jadedeb:~$ sudo injector -r /home/jade/win10.json -d 2 -i 6944 -e 'c:\windows\system32\notepad.exe' -v    
DRAKVUF injector v0.7-git20200528073757+e961705-1 Copyright (C) 2014-2020 Tamas K Lengyel
1590698786.146650 drakvuf_init: Rekall WoW64 profile not used
1590698786.146809 drakvuf_event_fd_add fd=18
1590698786.146822 size of list=1
1590698786.146826 regenerating event_fds and fd_info_lookup...
1590698786.146829 new event_fd i=0 for fd=18
1590698786.146833 new fd_info_lookup i=0 for fd=18
1590698786.146836 drakvuf_init: adding event_fd done
1590698786.147050 init_vmi on domID 2 -> win10
1590698786.147543 init_vmi: initializing vmi done
1590698786.147616 init_vmi: initializing vmi paging done
1590698786.269778 init_vmi: initializing vmi OS done
1590698786.269817 Max GPFN: 0x210411
1590698786.269849 Max mem set? 0
1590698786.269878 Physmap populated? 0
1590698786.270008 Altp2m enabled? 1
1590698786.270019 Altp2m view X created with ID 1
1590698786.270040 Altp2m view R created with ID 2
1590698786.270103 init_vmi finished
1590698786.270114 Windows kernel base address is 0xfffff80568600000
1590698786.270132 Failed to find address for symbol KiInitialPCR
1590698786.270152 Failed to find offset for _EPROCESS:Wow64Process
1590698786.270175 Failed to find offset for VadRoot:BalancedRoot
1590698786.270192 Failed to find offset for _MMVAD:LeftChild
1590698786.270208 Failed to find offset for _MMVAD:RightChild
1590698786.270224 Failed to find offset for _MMVAD:StartingVpn
1590698786.270240 Failed to find offset for _MMVAD:EndingVpn
1590698786.270256 Failed to find offset for _MMVAD:u
1590698786.270282 Failed to find offset for _KPCR:PrcbData
1590698786.270318 Failed to find offsets for array of structure names and subsymbols.
1590698786.270327 Failed to find offsets for of bitfield: _MMVAD_FLAGS:MemCommit.
1590698786.270346 Failed to find offsets for of bitfield: (null):(null).
1590698786.270363 Failed to find offsets for of bitfield: _MMVAD_FLAGS:CommitCharge.
1590698786.270380 libdrakvuf initialized
Injector starting c:\windows\system32\notepad.exe through PID 6944 TID: 0
Target PID 6944 to start 'c:\windows\system32\notepad.exe'
1590698786.271514 Found PEB @ 0x730000. LDR @ 0x7ff91ebd94c0. INLOADORDER @ 0x9925b0.
1590698786.271554 Found module OneDrive.exe
1590698786.271581 Found module ntdll.dll
1590698786.271592 Found module wow64.dll
1590698786.271600 Found module wow64win.dll
1590698786.271619 Found module wow64cpu.dll
Failed to get address of kernel32.dll!ResumeThread
Unable to initialize injector functions
Injector freed
Process startup failed
1590698786.271648 close_vmi starting
1590698786.301934 close_vmi finished

Why would it be unable to get the resume thread offset for onedrive but it works fine with explorer? How do I fix this offset error?

Also, I'm on the new .iso version "May 2020 Update" which was just released a day or two ago.

Edit: was also able to hijack a notepad.exe and make another notepad process. without testing every process it seems to be working at least for some processes.

[tklengyel]

It just means that the process you chose doesn't use kernel32.dll and because of that it's not mapped. You can't really "fix" it, you need to pick a process that does use kernel32.dll.

[JordanBoulan]

Thanks, makes sense. Is there further documentation on the altp2m functions and the "xc" library in general? (xc_function_name()). I'm learning how to use the libraries by looking at your code but can't seem to find the libraries documented anywhere. You also link like Lxencrtl etc but i can't seem to find anything besides the high-level docs.

Thanks.

[tklengyel]

You won't find documentation on it, there is none. You can read https://xenproject.org/2016/04/13/stealthy-monitoring-with-xen-altp2m and https://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01319.html for some context.

[JordanBoulan]

I have a couple more questions then I will close this. Thanks, as always, for your fast responses.

First, I'm getting random full system freezes (xen and all) with windows vm's anywhere from 5-20 min after booting. i've tried changing settings around in my cfg a bunch and can't seem to pinpoint any one setting causing it. How do i go about debugging this? Is it simply unstable because it is a new version?

second, my pafish looks really good. It passing the first rdtsc() check but fails the one where it does an rdtsc() and forces a vm exit. Is this handled in any plugin? how do i go about fixing it?

[tklengyel]

Depending on what hardware you are running on you might be running into the same issue discussed here: https://lists.xenproject.org/archives/html/xen-devel/2020-05/msg01985.html

RDTSC is not monitored by any plugins and there is currently no vm_event defined by Xen to trap and manipulate it when it goes to the hypervisor. It can certainly be added, but it's a full-stack process, adding the feature to Xen, then to LibVMI and then to DRAKVUF.

[tklengyel]

Also, FYI, with DRAKVUF we never intended to hide the fact that the VM is a VM. That's impossible to do and there always be checks that will detect that you are running in a VM. With Hyper-V now being on by default on most systems its a pointless check anyway, everything is a VM now ¯\_(ツ)_/¯

[JordanBoulan]

Very true, I agree with you there

For the deadlock issue? How did he get his kernel output? I can't input or even connect via ssh after the freeze. I'd like to see if it is the same issue

maj 27 23:17:02 debian kernel: rcu: INFO: rcu_sched self-detected stall on CPU maj 27 23:17:02 debian kernel: rcu: 0-....: (5250 ticks this GP) idle=cee/1/0x4000000000000002 softirq=11964/11964 fqs=2515 maj 27 23:17:02 debian kernel: rcu: (t=5251 jiffies g=27237 q=799) maj 27 23:17:02 debian kernel: NMI backtrace for cpu 0 maj 27 23:17:02 debian kernel: CPU: 0 PID: 643 Comm: z_rd_int_1 Tainted: P OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2 maj 27 23:17:02 debian kernel: Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.1.8 04/30/2019 maj 27 23:17:02 debian kernel: Call Trace: maj 27 23:17:02 debian kernel: maj 27 23:17:02 debian kernel: dump_stack+0x5c/0x80 maj 27 23:17:02 debian kernel: nmi_cpu_backtrace.cold.4+0x13/0x50 maj 27 23:17:02 debian kernel: ? lapic_can_unplug_cpu.cold.29+0x3b/0x3b maj 27 23:17:02 debian kernel: nmi_trigger_cpumask_backtrace+0xf9/0xfb maj 27 23:17:02 debian kernel: rcu_dump_cpu_stacks+0x9b/0xcb maj 27 23:17:02 debian kernel: rcu_check_callbacks.cold.81+0x1db/0x335 maj 27 23:17:02 debian kernel: ? tick_sched_do_timer+0x60/0x60 maj 27 23:17:02 debian kernel: update_process_times+0x28/0x60 maj 27 23:17:02 debian kernel: tick_sched_handle+0x22/0x60

[tklengyel]

You need a serial cable to the machine and configure Xen to print to that and Linux to print to the Xen console. https://wiki.xenproject.org/wiki/Xen_Serial_Console

[icedevml]

@JordanBoulan I had journalctl -xef constantly running on the SSH session with the server. This time I was lucky enough to get this output before everything froze for good. You should also see some interesting printouts on the server's display.

[JordanBoulan]

has the deadlock issue been fixed? Any updates?