Unable to read VA of ntoskrnl module.

[tttteatree]

@ufrisk

I have a problem with trying to read, for example, the PE header of win32kfull.sys inside ntoskrnl.exe. I can grab the base address successfully, but reads fail. I tried to set the context of the read (via PID) to both System (ntoskrnl.exe) and usermode (for example explorer.exe) but neither worked.

It also appears to be impossible to translate this VA to a PA, and then read the physical address.

And the reason I am having to do this anyway, is because the VMMDLL_ProcessGetEAT function does not work on win32kfull.sys. It returns a count of 0, so the 2nd part of the function fails.

I don't think I'm doing something wrong, because even the example using VMMDLL_ProcessGetEAT does not work, which makes me think I am missing something.

[ufrisk]

Yes, this applies to the win32k* modules. They are not mapped into the memory space of PID 4 (you'll see the same missing behavior in windbg when trying to read that memory).

win32k is only mapped into the address space of user mode processes (most but not all of them) such as explorer.exe.

Early on in the design process I made a decision to exclude the kernel parts of memory from the analysis of the user mode processes. This reduces noise to the end user a lot and also greatly increases overall performance. The reasoning is that if you do wish to analyze kernel memory you can do it in PID4.

I would need to somehow make some changes to the external API to accommodate this need of being able to analyze the kernel parts of some user mode processes; without making it a general setting (due to performance and increased non-essential info to the user).

If you're willing to recompile you can alter this behavior in the vmm.h!VMM_PROCESS.fUserOnly flag. Also due to unfortunate happenings out of my control I'm not able to publish any updates for some time. I hope (but I don't know how realistic it is) to have these issues resolved by end of June.

[tttteatree]

@ufrisk

I hope you are well!

I managed to bypass this issue utilizing fUserOnly as you said, setting it to false for all cases, it's hacky but it gets the job done for my specific usage right now.

I've ran into another problem, I am allocating memory inside win32kfull.sys using ExAllocatePool, but I am unable to write to this memory using pcileech/MemProcFS. Since the shellcode to call for the allocation is executed inside win32kfull.sys, the memory should be there. Therefor, I should pass to VMMDLL_MemWrite the PID of either 4 (System/ntosknrl.exe) or the PID of explorer.exe or another session-space process, right? Neither of those writes succeeed, and again I am unable to translate from virtual to physical for the address returned by ExAllocatePool.

Do you have any advice?

[ufrisk]

or just set it on some individual process; like csrss.exe that is always there; I think it's likely that I'll do something like this in the next update; but I haven't decided what to do quite yet.

anyway, it can be either that the memory is just not allocated against a physical page; i.e. that you need to write some non-zero byte to it for the windows memory manager to actually allocate a physical page against your allocation. Or it can be due to MemProcFs caching of page tables.

There are some config options for the caching, or just wait some time for it to clear (if it's the issue). You can also check out vmmproc.c!VmmProcCacheUpdaterThread and more specifically VmmCacheClear(VMM_CACHE_TAG_PAGING); and VmmCacheClear(VMM_CACHE_TAG_PHYS);

I now see that I also should have added CACHE_FLUSH options to the API that is not there. I'll do this in the next update.

[tttteatree]

Would this not apply only to reads, rather than writes? I already pass VMMDLL_FLAG_NOCACHE on reads, but the write equivalent has no flags parameter and I didn't think caching effected it.

[ufrisk]

writes are generally not affected by caching, but page table lookups are; which if required for the physical to virtual translation is.

I don't know if this page table caching is your issue; it may reside with the Windows memory manager if you just allocated the memory block without writing anything into it...

Try alter the Write call so that you add VmmCacheClear(VMM_CACHE_TAG_TLB); VmmCacheClear(VMM_CACHE_TAG_PAGING); before doing any virtual2physical translations to test it.

[tttteatree]

You're awesome, clearing cache fixed it and writes go through now as expected. I hope I didn't bother you too much.

[ufrisk]

Awesome, and options for cache flushes are added to the external API in my dev copy now.

we'll see how I'll do with the win32k and the user-mode flag, that is a bit more tricky to fix in a good way that makes sense API-wise.

please let me know if you run into more issues; otherwise good luck :)

[ufrisk]

You may try VMMDLL_Refresh() as well.

[tttteatree]

Since VmmCacheClear wasn't available quickly via dll, I quickly appended (again a bad hack) VmmCacheClear(VMM_CACHE_TAG_PAGING) into VMMDLL_Refresh() and used that after doing my allocations.

Since I've made the "necessary" changes to accomplish what I need, I can now see how it may need to be implemented properly (cleanly and with user accessible flags), perhaps it's something I could work on and submit to you.

I accomplished my goal, these 2 issues were the only things holding me back. Now I could also re-add (as it's out of date) driver mapping and remote code execution to PCILeech if you were interested.

[ufrisk]

Nice, no hurry for me to fix this then :)

I'm working on fixing the pcileech kernel module injects for windows 2004 release now if that's what you meant with remote code execution?

About the driver mappings; i.e. loading unsigned drivers, that would be so very awesome to have it re-enabled. But my troubles around this project is around copyright legal stuff though so I'll not be able to accept contributions to the core pcileech code right now at least. A separate module (.kmd) would be totally awesome though. We'll see what comes out the legal stuff; but I expect it to take some more time unfortunately.

[tttteatree]

Sweet, well in my case remote code execution (at kernel level) and loading unsigned drivers happened to be one and the same, since I use a remote code execution method to achieve manual mapping the unsigned driver. Perhaps a .kmd is the way to go about it.

I am sure that if I uploaded the meat of what I am doing, you would understand how to architect certain things (such as the flags) to go along with the current design of pcileech & friends. Maybe a repo for "offensive" modules / capabilities is in order too!

[ufrisk]

Yes, I try to keep offensive capabilities away from the LeechCore project (memory capture) and MemProcFS (memory analysis) and keep it to PCILeech. Reason is that I don't want MemProcFS to be an offensive project detected by AV's and such. It's primarily a "blue team" memory analysis/forensic project. MemProcFS already supports plugins, and I have a separate repository for it, with some offensive ones - pypykatz. Having a driver load as a MemProcFS plugin doesn't make much sense though.

Next version of both PCILeech and LeechCore will contain separate plugins in separate respositories :) The LeechCore ones will as an example include the reverse engineered FTDI driver for Linux (which will make that driver stand-alone and usable in other peoples projects too); and also standalone "devices" in the form of .so for linux and .dll for Windows. The FPGA support will continue to be a built-in core feature though.

PCILeech will also have support for external command modules in the form of .so and .dll; which will have access to an initialized LeechCore/MemProcFS environment. Having your driver load as a module here would be totally awesome :) and I imagine super easy to implement if you already have it working. Having a convenient unsigned driver load working in the recent win10 once again would be super sweet :)

There is still some time to go before I'll be able to release this relatively major re-design/update though. Maybe in a month time or so even if my legal discussions go my way - which is far from certain. Also, I have to figure out how to handle the win32k edge case in a nice non-hacky way in the MemProcFS API.

[tttteatree]

@ufrisk

Coming back at you because I ran into more issues that I think could be related.

I'm searching for codecaves inside kernel32, or user32, for example, and noticed that some areas of memory are invalid (zeroed out). At first I thought that it was the read failing. Since for the purposes of codecaves, I'm searching for areas of zeroed memory with execution privileges, my tool is mistaking these failed / invalid reads for codecaves.

Here is an example of what I am describing:

Of course, after a certain amount of bytes, the read is valid again. There's no consistency as far as I can tell, it's as though some of the pages have protection from DMA.

I tried chunking the reads, thinking it was related to reading large pages, but it had no effect, even dropping down to reading in 8 byte chunks.

[ufrisk]

Hyper-V may protect/hide some memory from reading. It's not doing that with ntoskrnl.exe though. If HVCI is enabled kernel executable pages are write protected,, but not read protected as far as I know, but I have not checked this in the most recent Windows.

failed reads are sometimes zero-padded depending on which read function you use; also be careful to check number of read bytes and the "cb" field in each individual page if using the readscatter function.

reason is most probably instability with regards to the device, or too aggressive default settings for your computer; check out https://github.com/ufrisk/LeechCore/wiki/Device_FPGA for various settings. An example would be to start MemProcFS/vmm.dll with MemProcFS -device fpga://pciegen=1,tmread=1500,tmwrite=1500,readsize=0x10000,readretry=1 to use super conservative and slow settings (not really recommended but for testing).

Also about code caves; there is never a code cave which is one full zero page. Code caves as far as I know is always in a page with some valid data at the low byte range and zeroes at the higher byte range; so it should be possible to detect errors from code caves.

[tttteatree]

VT-d is disabled and so is Hyper-V and VBS. I did try with very conservative settings and no change. Also note these are usermode modules. I'm stumped, I've never had any sort of instability before.

[ufrisk]

For both user-mode and kernel-mode modules this may also happen due to paging (if the page have been paged out to the page file). This happens quite frequently expecially on lower memory systems. That should never be the case with ntoskrnl.exe though.

If it's not due to paging then I don't know why it's happening. It would be hugely helpful if you are able to replicate the issue somehow and possibly debug why it's happening. If there is a bug I'd very much like to fix it; but it's not something I've experienced without finding out the explanation for it.

Meanwhile, for code cave detection, if the page is all null bytes it's not a code cave, it's an error.

[tttteatree]

I'm just about to give up on this :(

I never expected to have the majority of my problems in usermode. I have no problems reading and writing kernel space memory, but working with user32/kernel32 to get usermode execution to then gain kernelmode execution is proving impossible. There's just so much instability that I've never encountered before and I can't explain it or debug it, for example today rather than working on the features itself, I am battling with user32.dll writes failing (due to virt2phys failing) out of nowhere, reboot and it's fixed, but at some point writes might stop working again, only to begin working again at a later time. I run 32GB memory, I disabled my page file to see if that helped at all, but nada.

[ufrisk]

1. Memory mapped as "image" in Windows (.exe/.dll/.sys) is shared amongst all processes. MemProcFS does a virtual->physical translation and then writes to the physical page. This means that if you write to user32.dll in explorer.exe you'll in effect write to user32.dll in all processes at the same time. You need to be aware of this and account for this in your shellcode, check the PID before doing anything for example.

2. Windows when it loads a DLL in to a new process it doesn't really load anything into that process memory space. Whenever a thread in the process tries to read (or execute) some memory in a newly loaded dll there is nothing there. A page fault occurs. The Windows memory manager then sees that this page that the page fault occured in belongs to a shared dll, and it looks in that dll's "prototype page" and maps the specific page into the memory space of the process and then resumes process thread execution. The process thread doing the read never sees anything.

---

MemProcFS when reading it first tries to translate virtual2physical, if that fails it tries to read the underlying "protype page", or if the memory manager compressed that memory page to save memory it reads it from the compressed memory store, or from the page file (if available).

MemProcFS when doing writes only support writing to pure physical memory right now. If virtual2physical translation fails the write fill fail. I could and probably should add writing to backing "prototype pages" as well to make things easier.

I won't be able to add writing to compressed memory or completely paged out memory though. So the problem will remain somewhat; but in your case I strongly suspect that the most failed virtual2physical translations is due to not yet mapped "prototype pages" - so fixing this should solve most of the issues. I'll add this, but it may take some time, I need to await my legal discussions unfortunately :/

[tttteatree]

Yep I know about #1, I would guess then that it's #2. I will search for a way to gain kernel execution without usermode execution. Thanks for the details, and good luck with the legal issues.

[ufrisk]

and I'll add a feature enhancement allowing to write to "prototype pages" as well in the next update, whenever that may be :)

[ufrisk]

I have added support for writing to some paged out pages, such as prototype pages. This should greatly increase your chances of success. You don't have to do anything to activate this support; it will try to write to such pages whenever it's required so support should be fully transparent.

Please note that I'm not going to be able to resolve all these issues; I'll never going to be able to support writing to compressed memory as an example. But the great majority should now be taken care of.

Closing issue.