NtGetNextThread / NtGetNextProcess

[reliasn]

ScyllaHide's ntdll.h and .lib's are the most complete that I know of. After searching extensively, I found that most ntdll libraries would rely on a bunch of calls to GetProcAddress in order to initialize the function pointers while ScyllaHide is able to call the Nt* functions directly, thanks to the provided .lib's.

I'm currently trying to implement a "custom" NtSuspendProcess where the function would also be able to skip the current thread:

#define OBJ_KERNEL_EXCLUSIVE           0x00010000L
#define OBJ_VALID_PRIVATE_ATTRIBUTES   0x00010000L
#define OBJ_ALL_VALID_ATTRIBUTES (OBJ_VALID_PRIVATE_ATTRIBUTES | OBJ_VALID_ATTRIBUTES)

void SuspendThreads(HANDLE procHandle, bool suspend, bool skipSelf){
    DWORD curThreadId = GetCurrentThreadId();
    HANDLE hThread = NULL;
    while(1) {
        NTSTATUS ntStat = NtGetNextThread(procHandle, hThread, STANDARD_RIGHTS_ALL, OBJ_ALL_VALID_ATTRIBUTES, 0, &hThread);
        if (!NT_SUCCESS(ntStat)) {
            break;
        }
        if (skipSelf && curThreadId == GetThreadId(hThread)) {
            continue;
        }
        if (suspend) {
            SuspendThread(hThread);
        } else {
            ResumeThread(hThread);
        }
    }
}

I really need to use NtGetNextThread because the CreateToolHelp32Snapshot + Thread32First/Next combination is just too slow for what I'm doing. From what I read, this function is available on Windows Vista onwards.

Would it be possible to add NtGetNextThread and NtGetNextProcess to the static libraries? Since they are already declared in ntdll.h, I thought it would be nice to have them included in the libraries as well.

At the moment, the following linker error is thrown:

[Mattiwatti]

First, I think you meant to pass (OBJ_VALID_ATTRIBUTES & OBJ_KERNEL_EXCLUSIVE) to NtGetNextThread, otherwise you will get very few results ;) In fact, you probably want to pass 0 (no attributes at all) instead, otherwise you are asking for the handle to have OBJ_INHERIT | OBJ_PERMANENT | OBJ_EXCLUSIVE | ..., which you probably don't actually want.

Similarly, I would request the minimum necessary access rights (I don't know what STANDARD_RIGHTS_ALL translates to, but it's probably either too much or not enough). In this case that would be THREAD_QUERY_LIMITED_INFORMATION | THREAD_SUSPEND_RESUME.

Anyway, to address the question: the reason ntdll.lib/ntdllp.lib don't export these functions is because ScyllaHide has a minimum target version of XP. That means any ntdll.dll functions not present in XP aren't in the import libraries. The import libraries are also forward compatible up to Windows 10, so the total set of exports is restricted to the subset of ntdll.dll exports that have existed since XP or before without ever being removed in a future version.

NtGetNextProcess and NtGetNextThread are an odd pair: ntdll.h requires a minimum target version of Server 2003 for them, which I probably got from the Process Hacker SDK, or else the fact that the Server 2003 kernel source code contains implementations of both. However that would mean the x64 version would have the exports, since XP x64 is actually based on the Server 2003 kernel. However I checked the ntdll files for both Server 2003 x86 and XP x64, and neither exports these functions. So this version requirement seems to be a mistake; it should actually be Vista or higher. Maybe someone wrote these functions for Server 2003 and then forgot to add them to the syscall list. (Side note: for some reason the kernel mode ZwGetNextProcess actually wasn't exported until Windows 10, and ZwGetNextThread is only exported in Windows 10 RS3!)

So, no, NtGetNextProcess/NtGetNextThread won't be added to the import libraries as long as XP is still supported. However, the good news is that you don't have to be a wizard to generate your own import libraries! For example, check out dumplib which is what I used to create the import libraries for ScyllaHide/x64dbg. If you're not super concerned about guaranteed compatibility with every Windows version, you can just use the output from dumpbin /EXPORTS %SYSTEMROOT%\System32\ntdll.dll as input to get an import library containing every function your system's ntdll.dll exports. If you only want to add a few exports, I'd recommend using the exports from ScyllaHide's .lib files as a base instead since I already did the work to make them forwards and backwards compatible with XP through 10.

If the above is too much work, there is also the option of using the import libraries directly from the Windows SDK. The Windows 10 SDK installs a heavily censored ntdll.lib. ntdllp.lib is only available from the WDK. There are some issues with these .lib files though: (1) they are targeted to match the Windows version of the SDK specifically, (2) many exports are missing, and (3) if you want to remove exports (not uncommon if you're mixing e.g. ntdllp.lib and the CRT and getting linker collisions), you have to restort to hex editing the lib file to get rid of unwanted names.

[reliasn]

Wow... that's a complete answer!

I will apply your suggestions to how I'm calling NtGetNextThread and will definitely check out your dumplib. I am familiar with dumpbin.exe and lib.exe but I've never customized static libraries like that. All this sounds a lot of work, but maybe in the end I find it easier than how I'm thinking now 😅

Out of curiosity: why didn't you take the GetProcAddress approach? For example, you would have ntdll.cpp with many declarations like this:

HMODULE ntdllMdl = GetModuleHandle("ntdll.dll");

t_NtGetNextThread NtGetNextThreadFn = t_NtGetNextThread(GetProcAddress(ntdllMdl, "NtGetNextThread"));
NTSTATUS NTAPI NtGetNextThread(HANDLE ProcessHandle, HANDLE ThreadHandle, ACCESS_MASK DesiredAccess, ULONG HandleAttributes, 
                               ULONG Flags, PHANDLE NewThreadHandle) 
{
    if (NtGetNextThreadFn)
        return NtGetNextThreadFn(ProcessHandle, ThreadHandle, DesiredAccess, HandleAttributes, Flags, NewThreadHandle);
    SetLastError(ERROR_PROC_NOT_FOUND);
    return STATUS_PROCEDURE_NOT_FOUND;
}

Don't you think it's easier to maintain the library for both platforms this way?

Thank you so much for taking the time to explain and for making such a complete NT library! Even though my question was related to a "3rdparty", the anti-anti-debugging techniques implemented in ScyllaHide have also been very useful for what I'm doing.

[Mattiwatti]

Why not use GetProcAddress? Interesting question, and so many answers, perspectives, philosophies... to choose from... where to begin? :)

• Since you mention ease of maintenance, I'll start with that. I actually think it is the opposite (in favour of static imports) by a large margin:
  • To do the same thing as your code, I only have to add one line to the .def file for the import library: NtGetNextThread. (admittedly, there are x86 and x64 versions, so 1 line times two I suppose). I'm not a great fan of line count wanking, but you have to admit that 2 lines is a fair amount less than the ~8-10 I see in your function.
  • Let's say you want to generalise this approach to make all exports of a given DLL available. The most extreme example I can think of for which I make my own import libraries is ntoskrnl.exe, which has nearly 3000 exports. How do you propose to extend your method to do this? I can think of a few ways: macros, templates (in reality, probably macros and templates), and automatic code generation. All of those will result in a guaranteed unmaintainable mess. My approach is actually almost exactly like code generation, except making use of existing tools to generate the code and moving the exports themselves to a separate definitions file. This is the only file you need to version or backup, the rest of the toolchain can be regenerated at will. And even the .def file can be regenerated from the .lib if needed.
  • Your function calls SetLastError and returns STATUS_PROCEDURE_NOT_FOUND if the import can't be located, which is good practice, but not the end of the story. What happens in the calling function? Maybe it will check for errors and show a message to the user. Or maybe it will simply ignore the error. Depending on the size of the project you might have a lot of control over this, or none at all. In the latter case, you'll probably eventually wind up deciding to move error handling to a centralised location and deal with it yourself so the caller can't mess it up. But then... oh. One of the consumers of your header/lib is a console app and the other a GUI app. Do you use MessageBoxW or fprintf(stderr)? You are now working on something that has nothing to do with the original purpose of your header. This spiral will continue downward forever until you become a heroin addict, and one day you will finally hit rock bottom and decide to download Boost. Know what the static import version looks like? It's this: God himself could not have devised a better error message. It's perfection in every way imaginable. It's short and to the point, and explains what's wrong in a way my grandmother could understand. And you don't have to write a single line of code for it! The ntdll loader will see to it that users will install the correct DLL before a single line of your code will ever be executed. This means that you are guaranteed by the loader that once your application is running, every DLL in the import directory will be loaded in your process's address space, and every function in the DLL will be available to call. This is an extremely nice property to have and somewhat underrated IMO (probably because we're so used to it).
• Performance. The Ldr component of NT is extremely well optimized, in fact ridiculously so. I would estimate (without evidence to back it up) that static imports are faster than the GetProcAddress equivalent of the same application by at least two orders of magnitude or more. The reason I don't have evidence for this is because I can't seem to create a PE file containing enough imports to rule out margin of error in the measurement. Most likely I'd hit some form of filesystem or PE format limitation before that point. OK, so I don't really care whether my imports are snapped in 100 microseconds or 1 ms. But you sure as hell will notice if you try to GetProcAddress 3000 functions :) Yes, you probably aren't going to do that. But that is in fact essentially what the ntdll loader does for you at startup, so it's a reasonable comparison.
• Code size (part 1). I guess you could call this another way of saying performance, because smaller code is faster code (strictly on the CPU). Smaller code also makes it easier for the compiler to reason about your code and apply further optimizations.
• Code size (part 2). Given the choice, I'd prefer not having a 'wrapper' (for anything) in my project sources. Wrappers add extra layers to debug, extra layers to disassemble, extra layers to keep in your head when you're building a mental model of a problem, and they add additional sources of bugs.
• Separation of function declarations and definitions. Your code is a function definition, and (as you said) would go in a .cpp file. I guess I won't have to ask how others are going to know about your functions ;) So now you have a .cpp and a .h file to maintain. Compare this to ScyllaHide/x64dbg's ntdll.h which contains only declarations. This is how third party code should be interfaced with: it can be described, but not implemented.
• Dependency analysis. Given a file that properly declares the modules and functions it uses (meaning in its import table) it is possible to generate a complete dependency tree of the entire application. You probably know Dependency Walker, but if not: click. It has a terrible, arcane interface (much like WinDbg) but if you know how to use it properly, it is an extremely powerful tool. It's diagnosed issues for me that would never have occurred to me as even possible. But static analysis requires static import declarations.

OK, that's enough for now I think ;) There's probably a few other things I forgot to mention, but those are the main reasons why I prefer static imports to a GetProcAddress style approach. That doesn't mean GetProcAddress is useless however, but you should limit its usage to cases where being able to call a function would be nice, but isn't critical. Personally, I have the luck of being able to work on pretty much whatever I want. Since I'm a simple man, my philosophy is: if it's optional, it's probably not interesting. If it's not interesting, I'll go work on something that is. Hence I can't remember the last time I actually used GetProcAddress, ha.

Re: dumplib: yes, it sounds like a lot of work, but it really isn't since you only have to do it once. But I'm guessing you're running a 64 bit OS? If so, start by ignoring all of the source code in the repository and just copy/paste the shitty hack batch file from the README into a new file. Run dumplib.bat C:\Windows\System32\ntdll.dll ntdll.dll This will produce a .lib file, but more importantly the .def file which you can now customise completely for your needs. When you're done, simply run lib again (exact command is in the batch file) and you get the new import library. 32 bit files are more of a PITA due to extern "C" name mangling; you'll want to use dumplib or similar to generate x86 import libs.