Accessing usermode memory via hypercall from MemoryAccess

[nicholasdkunes]

My end goal is to create a shared buffer between my usermode Hypervisor controller, call it ring3control, and the Hypervisor.

This shared buffer exists, and I hypercall to MiniVisor and give it the virtual address of the buffer, the size of the buffer. I then use MemoryAccess with the context of the current processor the hypercall was excecuted with to write 0xCC to my buffer.

In usermode, ring3control sees 0xCC as the first byte in it's buffer.

So all good, yes. Well, sort-of.

I'm not super familiar with the inner workings of MemoryAccess yet but from what I understand every processor has a link to the MemoryAccess context that includes the pages of memory.

My next test attempted to copy memory from notepad.exe (an x64 program on Windows) into my shared buffer. I hypercall to MiniVisor and pass the base address of notepad.exe to MiniVisor, and use MemoryAccess to read the first 32 bytes of notepad.exe, and write them using MemoryAccess to my buffer. The hypercall returns.

I don't see the bytes written to my buffer in this case.

I guess my question is, is MemoryAccess capable of doing cross-process memory access? Or can it not? Can you explain why? I'm not quite sure I understand it because I figured this test would work.

Thanks Satoshi.

[nicholasdkunes]

I'll start experimenting again today. This is what I understand is going on.

Memory Access uses the CR3 from the CPU that hypercalled to the HV to access the pages. We find the page that the physical address is on, and map it to our reserved page for read/write access. We then modify the memory or just read it into a buffer in the HV. This is what fails for me when hypercalling from one application, and trying to read from another (notepad.exe x64).

I even force notepad.exe to be on the same core as my usermode process by strapping the affinity, and I see that the thread is executing on the same CPU. Still nothing.

By the way, this is for my own research and I'm trying to develop a basic introspection engine similar to HVMI (: So any help would be appreciated.

My end goal is to implement features such as ddimon, memorymon, etc. To virtualize the system and watch for code execution exploits, access to memory that shouldn't be accessed, etc.

I've been working on MiniVisor for around 3 months now daily. So still learning, but getting there!

[nicholasdkunes]

Also, if you have time. Could you explain what you mean when you say VMM exit is effectively at HIGH_LEVEL? I don't see why I cannot call ExAllocatePool as long as I'm using Non Paged Pool in VMM exit, if I make sure the exit is called from PASSIVE_LEVEL, shouldn't winapi work? Why are we at high level? Interrupts are disabled there shouldn't be any issue.

I got the address of ExAllocatePool via traversing ntoskrnl.exe headers from HV, and call it in VMM exit, works just fine for me... No bugs seen (yet?)

[nicholasdkunes]

An update, if I try to read the base address of notepad.exe x64, I can't. If I try to read say the base address of one of its modules, shell32.dll, I am able to read it. Why is this? Access protection? Why does that matter when we are reading pages that are allocated by the HV and used by the OS?

[nicholasdkunes]

GetPhysicalAddressForGuest() is returning MV_INVALID_PHYSICAL_ADDRESS for the virtual address of notepad.exe x64 base address. In the code, this should only occur if the virtual address is not mapped into the guest address space, which it definitely is.

[tandasat]

Your understanding of the implementation is correct. The reason for getting MV_INVALID_PHYSICAL_ADDRESS could be that the base address of the notepad.exe is paged out, and also that CR3 that function uses is different from that of notepad.exe (ie, it would be that of your control process).

Because you are hypercalling from the other process (which has a difference CR3 from notepad's), the function translates VA to PA using a different paging structures. Then the base address of notepad.exe (say, 0x400000) is not mapped or paged out for your process, and the function fails. It successes with the DLL likely because that is mapped in both your process and notepad.exe at the same virtual address (and backed by the same physical pages).

You could extend the hypercall to take CR3 (or PID though it would be nasty) and use it instead of GUEST_CR3.

On the IRQL thing, here is an excellent explanation. https://github.com/tandasat/HyperPlatform/issues/3#issuecomment-230494046 IIRC, ExAllocatePool() could indeed do IPI. Even if it does not now, you do not have control over its implementation and taking a risk of the issues that is very hard to diagnose.

[nicholasdkunes]

You could extend the hypercall to take CR3 (or PID though it would be nasty) and use it instead of GUEST_CR3.

Can you elaborate on this? Like I said, I'm still learning. Your explanation of notepad.exe being paged out, and the translation occurring with different paging structures makes sense to me. Would the HOST_CR3 have the correct paging structure for the entire virtual address space of the OS?

My end goal is for my user mode process to be able to read the entire virtual address space of any ring3 process via MiniVisor.

On the IRQL thing, here is an excellent explanation. tandasat/HyperPlatform#3 (comment) IIRC, ExAllocatePool() could indeed do IPI. Even if it does not now, you do not have control over its implementation and taking a risk of the issues that is very hard to diagnose.

Oh wow. Alex explains that perfectly. I'll take my winapi calls out of VMM transition, and put them into hooks on winapi functions. For example, if I hook a winapi function with MiniVisor which I currently do, I should be able to call winapi functions from that context, yes? It should be safe, as long as I track my IRQL level and follow the documentation of the functions.

Cool, thank you. @tandasat you are always super helpful.

[nicholasdkunes]

As an extension, how could you accomplish this via PID like you mentioned? I'd prefer to do it properly with the correct CR3, but understanding both ways is better in the long run I suppose :)

[nicholasdkunes]

Intel volume 2A states:

Moves the contents of a control register (CR0, CR2, CR3, CR4, or CR8) to a general-purpose register or the contents of a general purpose register to a control register. The operand size for these instructions is always 32 bits in non-64-bit modes, regardless of the operand-size attribute. (See “Control Registers” in Chapter 2 of the Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, for a detailed description of the flags and fields in the control registers.) This instruction can be executed only when the current privilege level is 0

So, if the context of the VMCall is not a priv 0 (kernel), then how would one usermode process be able to retrieve the CR3 of another usermode process to access that processes page tables?

[nicholasdkunes]

I've settled on the idea of injecting kernel code, which is the worst way to do this in my opinion, so please let me know if you have a better way.

Currently, I'm implementing some basic assembly that I can set the guest RIP to and set the RSP to build the guest stack. The assembly then jumps into a C function in the HV. I trigger this based on the first instance of a kernel MSR after the target process to be introspected is launched, and jump into kernel context, where I can effectively access the CR3 for that process.

This seems messy, unintuitive, and strung out for such a simple concept.

Save me.

[tandasat]

Yup, if you need CR3, you would need kernel-mode code to resolve that. Unless you are taking about UEFI version of MiniVisor, it does not change things a lot since you are already able to load arbitrary kernel-mode drivers, it just adds another driver to interact with the hypervisor.

Taking PID is an terrible idea. You would configure the hypervisor to VM-exit on MOV-to-CR3, and each time CR3 changes, build and update CR3-to-PID mapping by inspecting the EPROCESS structures from the VM-exit handler. This is horrible because it requires tight dependency between hypervisor and NTOS (... though MiniVisor is not designed with isolation in mind anyway, so maybe "meh").

[nicholasdkunes]

@tandasat Of course I'm using MiniVisor UEFI, where would the fun be without it being awfully difficult!

So no, no direct kernel access, I have to inject kernel code into the OS, which I just got working... Hijacking a ntoskrnl thread and hypercalling out from it to check the descriptor to ensure ring 0 privilege.

How would you personally inspect EPROCESS structure from VM-exit handler on a vm-exit for mov-to-cr3? Does the context of that vm-exit provide the EPROCESS structure? If not, I don't see how we could call winapi safely in VM-exit to retrieve EPROCESS structure...

[nicholasdkunes]

Ok :) I have the CR3 of the target process. I will try to read it's headers. Hopefully this works. Very proud of myself so far this is a big achievement for me because today you helped me learn about processor control registers and the inner workings of my CPU.

And I'm doing it all myself too, which feels good! Thanks Satoshi, I'll update soon if it works...

[nicholasdkunes]

I was unable to read. My computer faulted. I'll keep looking... Although I know I have the correct CR3 as I verified with with WinDBG

[tandasat]

Did not realize you were playing with the the UEFI version, and am happy too that you have been able to learn something using my project.

How would you personally inspect EPROCESS structure from VM-exit handler on a vm-exit for mov-to-cr3?

Good question. No EPROCESS, so I was thinking that the guest FS (BASE) might point to ETHREAD and reading those structures with the provided CR3. I would think injecting KM code and doing VMCALL would be cleaner. As a thumb of rule, stay away from the idea of calling NT functions from the host.

(...) have to inject kernel code into the OS, which I just got working... Hijacking a ntoskrnl thread and hypercalling out from it (...)

Could you be able to outline how you did it? Previously I did a similar thing but wondering if you came up with other approach.

[nicholasdkunes]

Could you be able to outline how you did it? Previously I did a similar thing but wondering if you came up with other approach.

Wow, would have saved me a lot of time if that gist was in the master branch. No worries, I think it's better for people to implement this on their own so they understand how it works! Your implementation is essentially the same as mine.

Here's a basic outline of how I do it:

1. Trigger code one time, after OS is booted, on MSR IA32_LSTAR (or other MSRs that are reliably in kernel context) to capture the kernel context doing a syscall. 
2. Normally, people would use this to hook the syscall, but I'm not interested in that, just using it to inject kernel code snippets when I need to execute in r0 context. 
3. Since we are at HIGH_LEVEL like you stated, we want to be able to do winapi functions so we need to get back down to preferably <= APC_LEVEL so change guest RIP to custom assembly entry, and rebuild RSP for guest, preserve previous RSP for when we jump back into kernel exec after we are done injecting. Assembly entry's only job is to jump to C context. 
4. Now execution is in C function in HV, but it is not at HIGH_LEVEL, we are running at r0 context. 
5. Where our code differs is you use the original guest RIP to effectively pattern scan for ntoskrnl base, whereas I just provide the HV with the ntoskrnl base from usermode controller via hypercall. I may implement your solution instead.
6. I use ntoskrnl base to traverse the NT functions, and get address to ~20 functions I use regularly to do kernel operations, so I have a winapi toolset in the HV. 
7. I execute my kernel code that I'd like to execute, in r0 context.
8. I jump back to original guest RIP, and swap back in original guest RSP, and we continue execution from where we left off. 

Good question. No EPROCESS, so I was thinking that the guest FS (BASE) might point to ETHREAD and reading those structures with the provided CR3. I would think injecting KM code and doing VMCALL would be cleaner. As a thumb of rule, stay away from the idea of calling NT functions from the host.

Agreed. I'm sure we could find an elegant way to get the EPROCESS, but like you said, intertwining the host HV and the guest OS will make things messy overtime. Injecting kernel code is modular, and you can create different methods for different operating systems without having to have OS specific code in the HV.

--

I do have one question for you though as I go on this adventure still, because I faulted my PC testing something and I think I know why it happened, but I'm not super sure. If we are in r0 context like I told you, via kernel code injection, running our C function that runs in r0 context, and we pass an address that is a host address (like a static data in the .bss/.data section of the HV) to a winapi function, the winapi function does not have access to that address, and cannot use it, correct?

I imagine if I want to transfer data from host to guest r0 context, it would be more difficult...

[tandasat]

Thank you for sharing your trick. It is cool as it allows you are at certain process's context with that approach.

I might not be understanding your question, but I think you are asking if r0 C code running in the process context can touch any of virtual pages that the hypervisor is mapped. If so, not trivially. The HV is mapped in virtual addresses somewhere where other UEFI runtime drivers are mapped too, but there is no easy reference to it from the Windows world. That region is marked as "reserved" by the Windows OS loader, and so, the Windows kernel has no knowledge in the layout of the region (ie, where our HV is mapped in the region).

To pass data, I think you want to somehow map a page where the host writes data, and then, pass down its physical address to ring-0, which would map the address to virtual address and read/write contents to it.

[nicholasdkunes]

Thank you for sharing your trick. It is cool as it allows you are at certain process's context with that approach.

Yes, and you can actually expand that to do some janky, but (even more stable surprisingly) kernel code execution. Once you trigger on an MSR, you can hook ntoskrnl functions like you did (if you do it before PG init).

After seeing your code, I modified my kernel code injector to hook ExAllocatePoolWithTag since it is called pretty much constantly. As long as you export KeGetCurrentIrql() and setup a hypercall back to the VM to test if you are in kernel context, you can ensure you are at <= APC_LEVEL and safely call any winapi functions.

If you ever need to inject code, just notify the HV to execute the required code on the next execution of ExAllocatePoolWithTag that is <= APC_LEVEL and in kernel context.

Fun stuff :)

Regardless, @tandasat I was able to solve the greater problem and I'm using kernel code injection techniques like the one specified above to read/write memory to processes using the winapi stack RtlCopyMemory after I attach the usermode's address space to the kernel thread we hijack. It's an interesting way, but I learned a lot along the way, and I'm glad we got to discuss.

No worries, I'll be back surely with another issue soon enough, plenty to learn about this. Appreciate the help, closing this ticket.

[nicholasdkunes]

        /* are we in r0 context and at PASSIVE_LEVEL */
        UINT64 hyper_ret = AsmVmxCall(MV_VMCALL_INTRO_IS_R0_CTX, NULL, NULL, NULL);

        if (hyper_ret == FALSE || g_GuestAgent.KeGetCurrentIrql() != PASSIVE_LEVEL) {
            return;
        }

For those who read this in the future,

Above is an example of how I test to see if the call to ExAllocatePoolWithTag is 1) In ring0 context and 2) At PASSIVE_LEVEL

Pretty simple and elegant in my opinion.

[nicholasdkunes]

Of course, that code uses Satoshi's GuestAgent gist that he provided me with earlier. It manually exports KeGetCurrentIrql() and I implemented a custom hypercall to test kernel context, but that is rather trivial. Note that you must test IRQL in the context of the ExAllocatePoolWithTag() call and not via VMExit, as that IRQL level is at HIGH_LEVEL thanks to information provided earlier in this thread.