Userspace physical memory virtualization

[mor619dx]

Hey,

Edit: I see you updated the project, I will check and update here again.

I've taken the latest release of this project and tried to compile it with visual studio 2015, Trying to compile with release x64 will give the following compile errors:

....\exit.c(1680): error C2065: 'curr_handler': undeclared identifier ....\exit.c(1684): error C2065: 'curr_handler': undeclared identifier ....\exit.c(1685): error C2065: 'curr_handler': undeclared identifier

Which happen because

if defined(NESTED_VMX) || defined(DBG)

static u16 curr_handler = 0;

not happening.

Compiling in debug x64 will work, however trying to run it result in bsod. Stack:

This crash is on physical computer, Windows 10 x64 Version 10.0.14393, all virtualization and stuff are on.

Also i have a question: I want to use the vmfunc to change eptp when a selected program is running, so basically I need to check context switches and to run vmfunc whenever the context switch is toward giving run time to my process.

I have all the code about the process creation and such but not really sure how to "hook" the context switches, I could for example exit when cr3 load (which happens when program context switch) but it's not really bullet proof since any kernel driver can just change the cr3 register to the program cr3 (in which this case I don't want to call the vmfunc to switch the eptp).

I was wondered if you have any idea how to do this? (preferably someway with not much overhead since context switches are happen very frequent)

[asamy]

I am not getting the same bug with the debug build. It works just fine for me, have you tried latest? Same for the release build, it was fixed earlier. Can you fix your kernel symbols so we can get a proper trace? (Perhaps if you're using the VS project from upstream, then you need to adjust the windows version setting in Driver Settings to W10)

It highly depend on what you're trying to do exactly, e.g. are you trying to fake out RW/RWX pages or what? It can be a pain sometimes.

About the context switch there are multiple ways:

• Hook KiSwapContext (specifically SwapContext_PatchLdtBypass which does the cr3 load) or;
• Use cr3 load exiting

Regardless, when the context switch happens, there will be 2 cases:

• The current process ID won't match the one which cr3 is for (this is the "attach" process case.)
• The current process ID does indeed match the one which cr3 is for ("real" context switch.)

Of course this can still be hijacked but it will require some effort, the best way would be through EPT directly, and nothing else. You'd need to force CoW pages to get mapped to your process exclusively if you're going to use CRT, otherwise you should just fake out your own process pages only.

NB: If you're going to do it the EPT-only way, then you're gonna have a tons of violations if you're also going to fake out RW pages, incase of a rep prefix or similar, or even regular read/writes through executable pages, but this barely gives any overhead if using guest's IDT directly, page faults happen at pretty much same rate (maybe even higher).

You might also want to hook the #PF handler, it's up to you what you want to do, in that case, I wouldn't advise hooking it through IDT, rather hook MmAccessFault directly.

You might even want to integrate somewhat APIC self IPI (or even hook some sort of timer, e.g. APIC timer or HPET) and use that to your advantage.

[mor619dx]

I couldn't really find KiSwapContext function or SwapContext_LdtBypass (I see people research it on the internet but can't find it in ntoskrnl.exe).

Basically I'm using PsSetCreateProcessNotifyRoutine getting specific process I want (currently by name), iterate over it's base module va, getting the pa and lock them using MmProbeAndLockPages and creating ept table with each of the pages of the process have no RW.

I also create another ept table with no restriction at all,

I want to create some protection over the binary that's running (all of it sections - code, data etc..) but without damaging the performance by handle the all the RW violations and make sure they from the right caller (so if the binary itself try to change values in the pages it's ok, if someone else try I will get notify), So i thought I could something like this: When the process start to execute -> switch the eptp to the non violating -> no ept violation happening. When the process stop to execute -> switch the eptp to violating -> so when code running in the system tries to temper with the process pages I will get ept violation and so performance should be good most of the time (when the process running), and maybe some violations when someone try to temper the process.

Performance is important in this one so I don't want to have many ept violations in normal execution.

[asamy]

SwapContext_PatchLdtBypass:

About the performance, if you just do the EPT handling solely, you will barely have any drops (in the application that is). I wonder how are you even detecting if the EPT violation is coming from your application? Are you checking the RIP the violation came from?

How do you plan on distinguishing readwrite pages and executable pages? Can you describe your EPTP usage?

[mor619dx]

I haven't implemented yet the eptp switches since I wanted to think about how well this method will work.

However I did the process pages ept violation (on RW violations), and for a process running 80% on a single vcpu (i7 6700 4ghz) I got 1 million violations in a matter of seconds and it seems to effect the performance quite a lot. (I tested process base module which is 16mb in size only). In depth check I saw the .text section have some violations because of the functions jump table, and i'm guessing globals / data read / string etc are also hit the violations.

I tried using RIP but figured our very fast that it's contains the va (which can also be va of any process) and since my checks are working on pa it's not gonna work, I still need to think of a way how to get the info from where the ept violation came from.

What version of ntoskrnl.exe / ntkrnlmp.exe are you seeing this function? I checked file version 10.0.14393.0 / 10.0.14393.447 and can't really find this function anywhere - I do see the KiSwapContext but not SwapContext_PatchLdt (or any SwapContext*)

[asamy]

SwapContext is called from KiSwapContext, I am not sure which kernel I was using when I took the screenshot, probably 8.1. I can't load the 10 kernel right now as IDA is failing to get the symbols for some reason. It might have been changed to be named something else (maybe no symbols available for it at all), you could just search for this sequence of bytes: 0F 22 DA (aka mov cr3, rdx).

About the performance drop, it could be due to the debug prints as well, but a million of violations sounds too much.

VA's don't get shared to other processes unless they are special ones, e.g. SHARED_USER_DATA or similar. CoW pages get mapped to same physical page, however, different VAs (ntdll.dll, kernel32.dll, etc.), so you'll probably have trouble with CRT verifying your own process address space. Moreover, you can also check current thread or just check the physical pages, you have them locked anyway.

Another way would also to use VMFUNC inside your userspace app, there is no CPL checks, but you'd need to somehow ensure you won't get moved (or preempted) out of the current processor without restoring the EPTP which makes it really ghetto, not to mention that some syscalls will eventually end up doing a context switch, so it really isn't that viable at all. Take note that the VMFUNC usage inside user space also raises some awareness if some other process does it and tries to read yours, but this can be massive ;p

[mor619dx]

In windows 10 the swap context fuction seemed to have changed but it is pretty similar overall.

Finding when we enter our process runtime is not difficult but I wonder how hard is it to find when we perform context switch and running another code - I will need to do some more research on that.

In the updated version it doesn't seem to crash, Also I noticed some random performance issues that after some investigation seemed to be related to using ept in vmware (when running on physical machine this problems not happening).

Thank you for the helpful ideas!

[asamy]

Like I said, on cr3 switch, you'll have 2 cases, see my first comment.