search

Reloaded-Project / Reloaded.Hooks

Advanced native function hooks for x86, x64. Welcome to the next level!
GNU Lesser General Public License v3.0
213 stars 33 forks source link

Can this be a valid EasyHook replacement? #7

Closed netcorefan1 closed 2 years ago

netcorefan1 commented 3 years ago

Hello, first of all let me congratulate for this wonderful project!

I was wondering if this library could be a valid EasyHook replacement. If yes, are there any other limitations (except kernel mode hooking not supported in RH) or we can do everything we do in EH?

I like EH, but recently I abandoned .Net Framework completely in favour of .Net 5 and discovered that EH is not compatible. I tried to upgrade the library to .Net 5 and the injection works, but I don't get the remote event in the loader. It uses the old remoting API which I had to replace with named pipes. I get the remote event only if I target .Net Standard, but then I can't use AssemblyLoadContext and needed Net core dependencies for the implementation (if you have some suggestion, please let me know, thanks!). it is now a couple of days that I am stuck with this and I discovered your library by pure accidence. Given that EH development is stuck from very long time (except some recent updates containing a rude .Net standard version only compatible with local hooks), I am asking myself if I should switch.

I am surprised that your library is very well hidden. I scanned Github and the web to find alternatives and your RH never appeared. I only found CoreHook which is great, but then I quickly discovered that the project has been abandoned and had to return back to EH. Have you intention to maintain this project over the time? The last thing I want is to modify my code base and then suddenly discover a repo archived as read only. Thanks

Sewer56 commented 3 years ago

I was wondering if this library could be a valid EasyHook replacement. If yes, are there any other limitations (except kernel mode hooking not supported in RH) or we can do everything we do in EH?

I've never worked in Kernel Mode (or much with other libraries for that matter) so I can't answer that part. However, one thing of note is Reloaded.Hooks provides only does hooking, unlike other libraries which also provide you with the code to inject your program into a foreign process.

In order to get your code executing inside a native program in the first place (if that's a goal), you can use Reloaded-II or if you would prefer something more custom, use Reloaded.Core.Bootstrap (and the linked articles within) as an inspiration.

I am surprised that your library is very well hidden. I scanned Github and the web to find alternatives and your RH never appeared.

I don't advertise, simple as that. For better or worse I'd rather spend my time developing something than letting people know about it. It's kind of futile in a way, you get super jealous of people doing the simplest of things and feel generally unappreciated but it is how it is.

Have you intention to maintain this project over the time? The last thing I want is to modify my code base and then suddenly discover a repo archived as read only.

My hobby is to tamper with games and as such I use the library on almost a daily basis. Whenever there is a need to add a new feature, a bug report or a new useful .NET or language feature arrives (e.g. UnmanagedCallersOnly in .NET 5 and Function Pointers in C#9) the library gets updated; simple as that.

It's not as frequent these days as I don't run into scenarios where I need to update often but when there is a good reason, there is an update; so yes, by all means the project is actively maintained.

netcorefan1 commented 3 years ago

Hello @Sewer56 and many thanks for your answer. Sorry for my delay, but your library really deserved to be tried and I preferred to wait a bit more in order to return with results of my findings. I also never had the need for Kernel mode hook, but one day I may need. Do you think would be possible at least a smoothless integration? Yeahh, Reloaded.Hooks only manages local hooks and this is something I discovered after running my first hook. I liked how it works and this brought me to do more extensive experiments and this is where I started to have problems.

I fully understand why you don't advertise your project and I can easily feel your frustration. People got used to take advantage of these kind of open source projects without expressing a minimum of gratitude. Not only any gratitude, but sometimes they open issues with questions like "What this project is used for?". It seems a mockery than anything else. They expect the developer to explain everything without doing any effort on their own. However I fell that your projects really deserve to be known from the community to make it even better with bug reports, pull request, new features, issues etc. But the right persons!

I am glad to see that this project will be maintained. If you need to update, then I also need to update...therefore I believe that I don't have to care.

Ok. Let me go into details of what are my findings, otherwise you will take me for someone of that people :-) I have attached a sample project. It's a solution that shows how it is performed a remote hook with EasyHook and how I (tried) to replicate the same using reloaded-project. I must admit I had very hard time in making reloaded working even with what I believed was tasks free from trials and errors (and I'm still having issues with finding an easy way to call the right assemblies without going into complicated hard coding). What the sample does is this: it launches notepad3 in a completely hidden state. It does this by launching the process in a suspended state (to give the hook the time to act), then it enable "ShowWindow" hook, resume the process and once the call is intercepted it present a message box to the user asking for confirmation before displaying the window. Before someone think that I am trying to make malware...this isn't. Sometimes I need to execute processes for which I don't have control in an automated way without having their user interface disturbing my eyes and the flow of my main application. With EasyHook the sample works perfectly, but with reloaded I am having several troubles. I used Reloaded.Injector and DllExport since these were the ones you recommended in your extra note at the end of What do you use this library for? issue. If I only could return back in time I would have tried with the two alternatives you mentioned and save time. These are the problems I am facing:

  1. When I suspend the process, reloaded-injector fails (stating that has been able to write only part of the memory). I saw online that one possible solution could be LdrLoadDll, but since the injection is managed from reloaded, I am not sure on how this would be possible without modify the code of the injector. in addition to that, I always assume that it is not working due to my fault.
  2. Without suspending the process, the injector still fails to call the exported function if the function contain a call to any external referenced assembly (in this specific case Vanara which I use to have the pinvoke signature at hands). It doesn't matter if I directly reference the injected DLL project or where I keep the referenced assemblies. The function call fails in any case as soon as I call a method which belongs to referenced assemblies. I am unable to find a solution to this despite all of my efforts.
  3. If the injected DLL target .Net 5 (or any other .Net core) the injection fail. The maximum I can obtain is targeting Net Standard, but this brings some concerns. How to handle assembly unloading without having access to AssemblyLoadContext or better, let do this to the injector ?? It will still uses the old reflection AppDomain creation under the old .Net Framework? The good news is that, at least, I can reference from .Net 5 which is not possible with EH.
  4. In a working injection, the reloaded-injector is responsible for creating 6 exceptions in PeNet.dll (see VS output). Exceptions are supposed to never occur and I assume we should take a look.
  5. Communication with reloaded injector from client happens through a manual marshalling process. I find much better and easier to use named pipes solutions out there. I suppose the entry point remote event is still needed and can't be replaced whatever the injector is. I am only wondering of the overhead of adding named pipes instead of just relying on dll exported functions along with the limitations (can I easy consume events, async functions etc like I would do with named pipes) ?? EasyHook is removing the IPC functionality and leave the implementation to the user. This is something that I really appreciate.

Now, it appears to be that I must focus my efforts into the right injector, but as stated previously, if something does not work, I always assume it is my fault. If it isn't and it's just a limitation of reloaded-injector, then these are what could be the possible alternatives:

I lost lot of time in experiments, therefore this time I need to be very careful in choosing the right tool otherwise I will risk to enter into a very long tunnel.

It's soon to talk since I still have to manage to complete a working sample with reloaded hook, but I feel to say people reading this thread that it is not a replacement for EasyHook. Without having some background on EasyHook, an user will not be able to use reloaded other than local hooks and even for a sample basic project like the one I posted. There's much more involved. EasyHook is not a plug-n-play...you still have to proceed with trials and errors, but you have all the tools at hands and with enough patience the final goal will be reached. Even myself I would have never considered to replace EasyHook if it wasn't for the fact that I am forced to reference it from the old .Net Framework, a died technology I don't want to depend in any way. At the same time I am glad about this unwanted accident because it allowed me to discover this wonderful project from this guy. I see new and greater possibilities If I do the switch and invest time in learning, but I really need to be careful because the next road must be right one. So, in a situation like this there's not a clear winner. It all depends from the user needs. and experience. The attached sample should help to understand people to compare EasyHook and Reloaded hooks. Just don't do the mistake to think that EasyHook is easy because it works. To make it work for an user that try for the first time he will have to go in many trials, error and very long web searches.

EasyHook_Vs_ReloadedHookks.zip

Sewer56 commented 3 years ago

It's around 4:50am in the morning for me right now; and as such I'm a bit sleepy but let me at the very least drop a few comments.

Edit: I've spent more time writing this than I thought I would; oof.

Yeahh, Reloaded.Hooks only manages local hooks Yup, as I mentioned in the post above.

Actually injecting the code itself in the first place isn't normally a feature of hooking libraries; it's just been a thing in other .NET libraries because it's a non-trivial operation.

I used Reloaded.Injector and DllExport since these were the ones you recommended in your extra note at the end of What do you use this library for? issue.

For the record, it wasn't a recommendation; just an example/pointer. That post is a bit dated, from a time where DllExport didn't even have any support for Core and things were still running on Framework for me. Core 3.0 was barely in its first previews hehehe.

Personally I wouldn't use DllExport due to lacking Core support and the fact I've had it break on me before in the past with older versions when upgrading from VS2017 to VS2019 (fixed in later versions though).

When I suspend the process, reloaded-injector fails The injector works if the process is suspended (threads sleeping), however it fails if the process has been started after being created in a suspended state.

It's been a while but if I recall correctly it was due to the process not yet having the modules loaded if started suspended, meaning that I couldn't get the handle/address of Kernel32.

Problem: Getting address of Kernel32 for x86 process in x64 program.

I originally built this injector for Reloaded-II but sadly... for this very reason... never got to use it in practice. I ended up writing a tiny very basic DLL injector directly into R-II instead.

Source: 1, 2

I ended up using a "trick" where I launch a separate 32-bit application from the 64-bit process to get the address of function I wanted (LoadLibrary) and passed it to current application using Memory Mapped Files.

I did consider implementing this "trick" into this library but I felt like including an EXE in the build output would have raised a lot of flags for other people (aside being just generally weird and making initialization slow).

Without suspending the process, the injector still fails to call the exported function if the function contain a call to any external referenced assembly.

I can explain what's going on here!

The default AssemblyLoadContext into which your program is being loaded into does not know where to probe for libraries/DLLs in; or more specifically, it doesn't know it should check in the same location as the DLL.

The easiest way to resolve this issue is probably to overwrite the legacy AppDomain.AssemblyResolve event which is still respected by .NET Core's AssemblyLoadContexts.

In the old Reloaded (not Reloaded II), when things were on .NET Framework, I used to do this, so if you really want to try going this direction, feel free to use this class as inspiration 2

Important Note: I have no experience using DllExport with Core. When I started moving things to Core it simply wasn't available, so I had to go for other options.

It's entirely possible that DllExport loads a DLL into the main AssemblyLoadContext, which is bad as it could potentially clash with other injected .NET Core DLLs. (Think conflicts in dependency versions!)

Relevant MSDN Documentation

On Framework, EasyHook normally solves this assembly loading and isolation problem for you automatically. DllExport doesn't however.

If the injected DLL target .Net 5 (or any other .Net core) the injection fail.

DllExport's Problem.

How to handle assembly unloading without having access to AssemblyLoadContext

The .NET Core runtime and default AssemblyLoadContext cannot be unloaded. Just a heads up.

Additional created AssemblyLoadContext(s) can be unloaded starting with .NET Core 3 however.

In a working injection, the reloaded-injector is responsible for creating 6 exceptions in PeNet.dll (see VS output)

I'm not really bothered about any internal errors of a 3rd party library; that's on the library authors; as long as things work.

I might even replace it with my own minimalistic PE parser from Reloaded-II if I get motivated since it's much faster, smaller and more performant while only parsing the very basics I need.

Communication with reloaded injector from client happens through a manual marshalling process. I find much better and easier to use named pipes solutions out there.

Yeah I wouldn't suggest communication through executing remote functions. Not specifically for the way things are passed/marshalled but because creating a thread in a remote process every single time is really wasteful.

I've personally never really used named pipes but I made a small library for asynchronous handling of messages sent by a client named Reloaded.Messaging, which runs ontop of LiteNetLib.

You can get going with a simple server that handles serialization and asynchronous messages for you with really little code.

I use it in Reloaded-II to communicate with the injected DLL and load/unload mods in real time.

What could be the possible alternatives.

Some important notes.

Reloaded-II is a standalone program though so you're probably looking for one of the two options above it.

// Is tehere a way to make members non-static as the original sample???
private static IHook<ShowWindowDelegate> _createShowWindowHook;

You're creating the hook in a static method which can only access static members. You can just do the same thing in a regular class instance.

e.g.

public class CreateShowWindowHooker
{
    private IHook<ShowWindowDelegate> _createShowWindowHook;

    public CreateShowWindowHooker(long showWindowPointer) => _createShowWindowHook = ReloadedHooks.Instance.CreateHook<ShowWindowDelegate>(ShowWindowCallback, (long)showWindowPointer).Activate();

    bool ShowWindowCallback(HWND hWnd, ShowWindowCommand nCmdShow)
    {
        if (!string.Equals(GetWindowName(hWnd.DangerousGetHandle()), "Notepad3", StringComparison.OrdinalIgnoreCase)) 
            return _createShowWindowHook.OriginalFunction(hWnd, nCmdShow);

        MessageBox((IntPtr)0, "You're trying to open Notepad! Press OK to continue...", "Awoooo 56709!!", 0);
        return _createShowWindowHook.OriginalFunction(hWnd, nCmdShow);;
    }
}
_continueHooking = false; // unlike easyhook, it seems we need to close the hook from the client...

As per above (scroll a bit); it should be noted that the .NET Runtime and default AssemblyLoadContext cannot be unloaded.

Hooks created by Reloaded.Hooks can however be disabled with a simple function call e.g.

_createShowWindowHook.Disable();

You can enable and disable them as many times as you like. If you're curious how it works under the hood, a high level diagram (hint) can be found at https://github.com/Reloaded-Project/Reloaded.Hooks/issues/2 .

Unlike many other libraries, native and managed, Reloaded.Hooks' hooks are designed in such a way that it is safe to unload your code (or AssemblyLoadContext for that matter) after disabling hooks.

Unloading after disabling hooks is technically a memory leak as the Wrappers still exist in native memory however it's generally only <30 bytes per hook (on x86) in practice.

This should be everything, sleeptime!

netcorefan1 commented 3 years ago

Many thanks for your support. You're really an amazing guy.

It's around 4:50am in the morning for me right now; and as such I'm a bit sleepy but let me at the very least drop a few comments.

Please accept my apologies if I misunderstood something of what you said (it was late even here when I responded you and I was nearly in black-out with all of that code inside my brain!) like for example the purpose of Reloaded.Core.Bootstrap. I had to investigate and study in order to understand and respond to your comments because a great number of things were new to my knowledge. I tried the sample and I really like how it works. It would be amazing if we can call the functions in this way. I have only been able to successfully execute the sample outside VS. As soon as I try to debug I get "Reloaded.Core.Bootstrap.Example.exe has triggered a breakpoint" in say_hello, but I don't want to abuse your time in asking how to fix this especially because I am sure I am missing something obvious and it's my responsibility to find out what it is. I just wanted let you know in case that this is something unexpected when trying to debug the project as is.

Edit: I've spent more time writing this than I thought I would; oof.

Please take the time you need because I really don't want to abuse your time. The only reason for which my comments are so long is not because I want your comments for everything, but only to let you have a clear and detailed idea of what I am doing. You have already given to me the right directions especially in this response and it's my responsibility all the rest. If I ask your help is only when I tried everything on my own without success.

So, this should be the starting point for running in .Net core without any old framework dependency. To be honest, I don't have any idea on how to integrate this with EasyHook without going into heavy modification of its managed and unmanaged parts. Even the author has some concerns on how to manage this and I suppose this is the reason for which the .Net Standard is the only target framework taken in consideration for the upgrade. Now it is very clear why you decided to use a different approach and split components apart. It's great to have an all-in-one solution like EH, but when components are so tied then things can become very hard to manage. I spent days trying to make EH working and now I know that I simply can't.

So... the right road is to return to Reloaded sample project since it is the only place where I can see the light in this tunnel...

For the record, it wasn't a recommendation; just an example/pointer. That post is a bit dated, from a time where DllExport didn't even have any support for Core and things were still running on Framework for me. Core 3.0 was barely in its first previews hehehe.

Personally I wouldn't use DllExport due to lacking Core support and the fact I've had it break on me before in the past with older versions when upgrading from VS2017 to VS2019 (fixed in later versions though).

I am also not happy to depend from Dllexport since it adds complexity to the project even if they managed to automate the whole process. I believed it was the only way to do things without going into complicated hard coding.

If the injected DLL target .Net 5 (or any other .Net core) the injection fail.

DllExport's Problem.

Then, the problem is not on the injector side. So, if I'm not wrong their .Net core support is some sort of "fake" support and there are the serious dangers you mentioned. Oh my God!

When I suspend the process, reloaded-injector fails The injector works if the process is suspended (threads sleeping), however it fails if the process has been started after being created in a suspended state.

It's been a while but if I recall correctly it was due to the process not yet having the modules loaded if started suspended, meaning that I couldn't get the handle/address of Kernel32.

Problem: Getting address of Kernel32 for x86 process in x64 program.

I originally built this injector for Reloaded-II but sadly... for this very reason... never got to use it in practice. I ended up writing a tiny very basic DLL injector directly into R-II instead.

Source: 1, 2

I ended up using a "trick" where I launch a separate 32-bit application from the 64-bit process to get the address of function I wanted (LoadLibrary) and passed it to current application using Memory Mapped Files.

I did consider implementing this "trick" into this library but I felt like including an EXE in the build output would have raised a lot of flags for other people (aside being just generally weird and making initialization slow).

I am not sure that I have understood well, but I believe the reason for which reloaded-injector fails to write memory in a process created suspended (with error "only a part of memory has been written") should be due to the fact that it uses CreateRemoteThread api. This thread describes the reasons for the failure. Basically kernel32.dll is not loaded yet (only ntdll.dll is loaded), so it cannot use createprocess to call LoadLibrary (in the kernel32.dll). One solution is to use LdrLoadDll and this is how EasyHook managed to make it working. I am not sure on how integrate this solution with reloaded-injector. That's why I started to look at alternative injectors like, in primis, GH Injector. It is also more features rich and I suppose this should minimize similar injection problems in the future. The trick you linked in the two sources may be another approach, but I don't see why recurring to this hack while there are already working solutions.

Without suspending the process, the injector still fails to call the exported function if the function contain a call to any external referenced assembly.

I can explain what's going on here!

The default AssemblyLoadContext into which your program is being loaded into does not know where to probe for libraries/DLLs in; or more specifically, it doesn't know it should check in the same location as the DLL.

The easiest way to resolve this issue is probably to overwrite the legacy AppDomain.AssemblyResolve event which is still respected by .NET Core's AssemblyLoadContexts.

In the old Reloaded (not Reloaded II), when things were on .NET Framework, I used to do this, so if you really want to try going this direction, feel free to use this class as inspiration 2 On Framework, EasyHook normally solves this assembly loading and isolation problem for you automatically. DllExport doesn't however.

I tried to play with this without success and I didn't spent further time to make it work because I realized that I am forced to target .Net Framework and basically I am going to recreate the same EasyHook loader. Once the .Net Framework is loaded I am afraid that I will go into the same issue I have gone with EH.

Important Note: I have no experience using DllExport with Core. When I started moving things to Core it simply wasn't available, so I had to go for other options.

It's entirely possible that DllExport loads a DLL into the main AssemblyLoadContext, which is bad as it could potentially clash with other injected .NET Core DLLs. (Think conflicts in dependency versions!)

This is scaring !!!

How to handle assembly unloading without having access to AssemblyLoadContext

The .NET Core runtime and default AssemblyLoadContext cannot be unloaded. Just a heads up.

Additional created AssemblyLoadContext(s) can be unloaded starting with .NET Core 3 however.

As per above (scroll a bit); it should be noted that the .NET Runtime and default AssemblyLoadContext cannot be unloaded.

Hooks created by Reloaded.Hooks can however be disabled with a simple function call e.g.

_createShowWindowHook.Disable();

You can enable and disable them as many times as you like. If you're curious how it works under the hood, a high level diagram (hint) can be found at #2 .

Unlike many other libraries, native and managed, Reloaded.Hooks' hooks are designed in such a way that it is safe to unload your code (or AssemblyLoadContext for that matter) after disabling hooks.

This is for sure another valid reason that makes Reloaded.Hooks even more valuable. Final users (me included) are not aware of such details and has the false belief that calling Dispose or any other release function is completely safe.

Unloading after disabling hooks is technically a memory leak as the Wrappers still exist in native memory however it's generally only <30 bytes per hook (on x86) in practice.

This means that a complete cleanup process (which basically means returning the application in its default untouched state) is only possible with .Net framework? This means that the default AssemblyLoadContext will stay loaded for the entire life of the targeted application? I must admit that I was not prepared for this and I never investigated because I had the assumption that since .Net Core is supposed to be the future it was already prepared for such scenarios and in a better way than .Net Framework already does. Instead I just learned that is not even supported! I have to suppose this also applies to .Net 5. If I have understood well, .Net Framework is able to remove even those 30-60 bytes/hook while .Net core can't (and don't because would mean creating a leak). Seems to be that I can safely load and unload the hooks into a custom AssemblyLoadContext, but the main CoreCLR context will remain loaded for the full life of the application. As long as the only traces left after closing an hook are those 30/60 bytes per hook, I really don't care, but my concern is that the whole net core runtime will remain loaded into the target app and this is not light at all. I am sure that I am missing something very obvious. There must be some sort of sharing mechanism. I suppose that if there is one app that uses the CoreCLR, then the CCLR is loaded in the system. If a second app needs the CCLR, I suppose it will share the same CCLR loaded from the first app. I can't believe that each app will use their own CCLR "instances" because a system with limited resources will go down soon. I must have for sure misunderstood the point, but the inability of the .net core to unload the default context make me confused. Another reason that increase my concerns is that time ago I reported the inability to unload WPF runtimes from the custom context and after their investigations, it turned out that it was not a bug, but just how is designed to work and it is unclear if in the future they will implement such feature or not. This is expected even for winforms (and I suppose any other official framework runtime). If I remember well, they told me that there was the same sharing mechanism I just mentioned, but I don't know if this also applies to the CCLR and in the case of hooks.

In a working injection, the reloaded-injector is responsible for creating 6 exceptions in PeNet.dll (see VS output)

I'm not really bothered about any internal errors of a 3rd party library; that's on the library authors; as long as things work.

I might even replace it with my own minimalistic PE parser from Reloaded-II if I get motivated since it's much faster, smaller and more performant while only parsing the very basics I need.

You're right, I should adopt the same behaviour when things work instead of getting bothered for everything. It would save me a lot of time!

Communication with reloaded injector from client happens through a manual marshalling process. I find much better and easier to use named pipes solutions out there.

Yeah I wouldn't suggest communication through executing remote functions. Not specifically for the way things are passed/marshalled but because creating a thread in a remote process every single time is really wasteful.

I've personally never really used named pipes but I made a small library for asynchronous handling of messages sent by a client named Reloaded.Messaging, which runs ontop of LiteNetLib.

You can get going with a simple server that handles serialization and asynchronous messages for you with really little code.

I use it in Reloaded-II to communicate with the injected DLL and load/unload mods in real time.

You understood well my concerns. So, in the specific case of reloaded-injector (and I suppose in general too), the best way would be to use the remote thread only once as an entry point and then run a dedicated IPC server for all the rest of the communication. Thanks for links.

What could be the possible alternatives.

Some important notes.

  • Choice of DLL Injector doesn't really matter; you shouldn't really concern yourself there.

If you says that the choice of DLL Injector doesn't really matter, then I am sure it really doesn't matter. However, I am still asking myself how I could handle the suspension problem I faced with reloaded-injector without dirty hacks or modification to its source code. This also applies to other kind of injection scenarios which I may have to deal with. I know that the all-in-one solutions has a price to pay (and I know well because that price has been the cause for this thread), but I would prefer at least to not split the injector into multiple solutions. I am doing again the same mistake? eh eh... Anyway, now the injector is the last of my problems. I must first successfully complete the next point below...

  • The recommended approach for .NET Core/.NET 5 is to load the runtime manually from a native program as detailed here and then ask the runtime to load your DLL. It will be loaded into a custom load context, nicely isolated from the default one.

    • The recommended alternative to DllExport for Core is DNNE. I haven't had a chance to use it myself but basically it auto-generates a native library that loads the runtime under the hood (as mentioned above) and passes over and exports the same functions in the native library, forwarding them to .NET.
    • Reloaded.Core.Bootstrap which I mentioned is basically a library/example for loading the runtime as per the MSDN approach above. The reason I brought it up is in case you wanted something more custom than Reloaded, or the "Do it yourself" way.
    • Reloaded-II is probably straight up by far the easiest way to get your .NET code running. All you do is build a single AnyCPU library and call it a day. The default template even has a shared instance of Reloaded.Hooks present supplied from another mod which is updated alongside the library (your code will always use the newest version of Reloaded.Hooks!).

    That documentation link not working was an oopsie on my end. Seems you stumbled on the GameBanana page where I didn't update the description since I renamed the Docs folder to docs because MkDocs (documentation generator) which I added recently is case sensitive and that also broke the URL ahahaha.

Thanks for your recommendation. As an EasyHook user I would have never believed there was so much stuff to take care about under the hood. Only now I start to realize what are the pieces of the puzzle and I still have to understand how to put all of them together. My nightmare started some weeks ago when I decided to make a wrapper around EasyHook. Think to it like a general hooker utility that each app can call on demand to enable desired hooks. It simply wasn't smart to create a different hooker for each app requirement (may be I was doing something similar to Reloaded II?). At half of the job I realized that .Net core apps refused to reference the wrapper and even if I managed to make the whole thing to compile trough interfaces, .Net standard and other tricks, during runtime I got several generic exceptions (after consulting the issues section of dotnet core repo I had to finally accept that these two runtimes can't work together in such scenarios). I wanted to insist and started to play with EH source code. I have gone further with a certain degree of success, but then I have been forced to abort until I discovered by accident your project and this is where I found new hopes. To be honest I am tempted all the times to just leave everything as is (like you see in the EasyHook project sample), launch the hooker as a .Net Framework executable and then call the functions using an IPC solution. If I only was aware of this from the beginning! Now it's late, all the time I lost imposes me to continue until I come up with a solution and see this as a chance to learn more.

So, let see how I can convert the sample project into a full .Net core solution. I think that starting with DNNE should be the best way because (if I am not wrong) it allow to manage the whole process directly from C#. I have considered starting with Reloaded.Core.Bootstrap, but it seems that I have to play with native code.

  • Reloaded-II is probably straight up by far the easiest way to get your .NET code running. All you do is build a single AnyCPU library and call it a day. The default template even has a shared instance of Reloaded.Hooks present supplied from another mod which is updated alongside the library (your code will always use the newest version of Reloaded.Hooks!).

This is highly attractive and I am really tempted to start with this. However you also said...

Reloaded-II is a standalone program though so you're probably looking for one of the two options above it.

...and I have no idea of the implications (may be running the gui mod and my app at the same time?). I also have to take into account the injection process and avoid breakages as happened with the suspended process. You suggested that I could be better off with one of the first two options and this is enough for me.

// Is tehere a way to make members non-static as the original sample???
private static IHook<ShowWindowDelegate> _createShowWindowHook;

You're creating the hook in a static method which can only access static members. You can just do the same thing in a regular class instance.

e.g.

Thanks for the sample code, I have implemented the changes.

My next challenge... Basically, the changes I should do in the sample project involve creating a new library which references DNNE and the exported methods. Seems easy to say, but I already know it wouldn't! Once the exported methods are called, then I should proceed to call the injector from that methods and the injector will take care to load the injected DLL. Not sure on how to join all of these pieces, but let try with a piece at a time!

Sorry for the long comment. Again, do not lose time to respond to every single line, but I would appreciate if you could correct me when you see that I am doing wrong things because it will save me lot of time and headaches. I will update the thread as soon as I have enough relevant news. Thanks

Sewer56 commented 3 years ago

This means that a complete cleanup process (which basically means returning the application in its default untouched state) is only possible with .Net framework?

I specifically said you cannot unload the default AssemblyLoadContext (similar to Framework AppDomain) and runtime. I'm not 100% sure on this but I believe that with Framework you cannot unload the runtime or the default AppDomain either.

EH puts your injected library into a new AppDomain and kills that AppDomain when the library is unloaded. With Core 3/.NET 5 it's the same but with AssemblyLoadContext(s). Your Assembly goes into a separate AssemblyLoadContext (when using native hosting method/DNNE/Bootstrap and that AssemblyLoadContext can be killed to unload your Assembly).

I really don't care, but my concern is that the whole net core runtime will remain loaded into the target app and this is not light at all.

Pretty sure this will be the case regardless of whether you roll for Framework or Core. If you need to go lightweight, native code's probably the only way. If you're hooking a native application then it's likely the Visual C++ runtime will be loaded already which will make it even lighter on memory.

CoreRT/NativeAOT might be an option but it's really experimental and hacky; you'll likely have a bad time.

If I have understood well, .Net Framework is able to remove even those 30-60 bytes/hook while .Net core can't (and don't because would mean creating a leak).

No, this leftover memory is specific to my library implementation and has to-do with how I handle disabling/enabling hooks. This diagram from earlier is a good illustration of what's going on.

It specifically has to do with the fact that one of Reloaded.Hooks' goals is maximizing compatibility.

The common way to disable a hook seen in many tutorials and libraries is to just restore the original overwritten instructions that were made when the hook was installed, however, this has a serious flaw. If you install two hooks on top of each other, and disable the first one, the second hook is disabled too.

To give a real world example: If you, for example hook DirectX for whatever reason, then an external overlay comes along (e.g. Steam Overlay) and hooks after you; you wouldn't want unhooking your function to kill the Steam overlay.

Note: I specified less than 30 bytes, not sure where the 30-60 number came from. That number reaches around 30 when the Calling Convention of the hooked code and your code don't match. In practice when hooking Windows APIs and they match (Stdcall on x86) it's lower.

here must be some sort of sharing mechanism. I suppose that if there is one app that uses the CoreCLR, then the CCLR is loaded in the system. If a second app needs the CCLR, I suppose it will share the same CCLR loaded from the first app. I can't believe that each app will use their own CCLR "instances" because a system with limited resources will go down soon.

There is a solution, employed at the operating system level. Consider studying about topics such as Virtual Memory and Paging. Here's a Random Article I Like

A proper answer would be really complicated but an important takeaway is that memory occupied by DLLs are shared across processes.

Machine code compiled by the JIT will of course be local to the current process and not shared but the runtime is compiled with an optimization called ReadyToRun which includes machine code alongside IL code, and, because it's part of the DLL it will be shared.

I can't give you any hard numbers on memory usage though but if I were to have a guess 16-24MB of Physical RAM after unloading because of the runtime's managed heap and whatever else may be left. .NET itself does have some initial overhead but it actually becomes efficient once you start loading lots of stuff.

Here's Some Numbers for Reloaded though Memory is a complicated topic ahahaha.

Another reason that increase my concerns is that time ago I reported the inability to unload WPF runtimes from the custom context and after their investigations, it turned out that it was not a bug, but just how is designed to work and it is unclear if in the future they will implement such feature or not.

Assuming you mean WPF libraries that are part of the runtime, I'm going to have a guess about what's going on based on my experiments with tampering with AssemblyLoadContext(s) in Reloaded-II.

When loading WPF libraries by name inside your own AssemblyLoadContext; the ALC is unable to find the library. Therefore, the runtime attempts to load the library in the Default ALC; which succeeds because it knows where WPF (Microsoft.WindowsDesktop.App) is located. Because the library is loaded in the Default ALC, it therefore cannot be unloaded because the Default ALC cannot be unloaded.

Relevant MSDN Documentation

Note: I have managed to load WPF into a custom context before; however it's something I wouldn't recommend. You have to parse the RuntimeConfig and manually hunt for the install location of the Microsoft.WindowsDesktop.App to accomplish it; then load from file path inside your own ALC.

Here's a commit where I removed loading WPF & ASP.NET into separate ALCs from an experimental branch of Reloaded-II. I'll only ever re-consider adding it when it's necessary and take the minor memory savings for now with the default behaviour :P.

...and I have no idea of the implications (may be running the gui mod and my app at the same time?). Yeah, Reloaded-II is a standalone program. Specifically, it's a mod loader for modifying native applications and games using .NET.

I mentioned it because it's a really easy way to get your .NET code running inside a native process; so you could use it for testing the hook code in the DLL you plan to inject.

Sewer56 commented 3 years ago

Tsonic_win_3Di2rRAkvD P4G_WyTdmMZda8 image

Source Code.zip

Minimalist example of what you're trying to achieve as a Reloaded-II mod. No external dependencies, compiles into a tiny package. I made this in around 10-15 minutes.

netcorefan1 commented 3 years ago

Hello @Sewer56, sorry for my delay, but I didn't wanted to post until I could come back with relevant progresses. After following your suggestions I have gone really far and I have an example where I can finally see all the pieces working together. Thanks @Sewer56 !!

I must admit I had very hard time especially with x86-x64 architectures. At the beginning I hoped that AnyCPU would be enough for the reloaded injector, but it wasn't. The functions failed and I had been forced to hardcode the loader csproj in order to build automatically x64 and x86 binaries at once (I had to learn how to deal with this in csproj and msbuild, something that I never had to do in the past). DNNE also gave problems in x86 and you can see what I have been forced to do in the NetCoreLoader.csproj (along with the required conditional compilation). Basically, in x86, if the functions does not have the DNNE Cdecl attribute, the function call of your injector fails. DNNE also returns a very cryptic error message when the x86 version of the sdk is not installed which made me lose lot of time in searching for a solution while the solution was so easy. Apart from other initial required configurations which takes some to take confidence it works well. In the solution you will find a commented DependencyResolver.cs. It is some code which I have grabbed from an abandoned project called CoreHook. I tried it and does the same thing of my easier implementation. I am not sure on its usefulness. I left here just for reference.

There are several concerns I commented in the sample, but my priority was to get the engine on and I left them in standby. The whole chain seems to work (at least partially), but what I am really worried about is the resources usages:

I can't give you any hard numbers on memory usage though but if I were to have a guess 16-24MB of Physical RAM after unloading because of the runtime's managed heap and whatever else may be left. .NET itself does have some initial overhead but it actually becomes efficient once you start loading lots of stuff.

The new sample runs both EH and RH and shows memory usage:

MemoryUsage

You can see that EH adds around 1.5 Mb of memory (I don't know why memory usage is the same even after calling LocalHook.Release). In RH, memory usage remains similar to EH until injection, but once it starts hooking it suddenly reach more than 31 Mb which remains pretty much the same once the hook is closed. There must be some flaw somewhere and below are some of what I think could contribute to this...

No, this leftover memory is specific to my library implementation and has to-do with how I handle disabling/enabling hooks. This diagram from earlier is a good illustration of what's going on.

It specifically has to do with the fact that one of Reloaded.Hooks' goals is maximizing compatibility.

I totally agree with your design. However, shouldn't be better some sort of switch available to the user in order to override this behaviour? I suspect that these references may be what cause for the below assemblies loaded from RH to not unload (see NetCoreLoader.StopHooking.Unload): System.Runtime.InteropServices.RuntimeInformation.dll System.Linq.dll System.Collections.dll netstandard.dll (may be you can remove definitively any .Net Framework connection in the new releases?) System.Collections.Concurrent.dll Microsoft.Win32.Primitives.dll System.IO.FileSystem.dll System.Memory.dll System.Runtime.CompilerServices.Unsafe.dll System.Threading.dll I am not sure on memory impact, but it's still stuff that should not be left.

Note: I specified less than 30 bytes, not sure where the 30-60 number came from. That number reaches around 30 when the > Calling Convention of the hooked code and your code don't match. In practice when hooking Windows APIs and they match (Stdcall on x86) it's lower.

Unlike EH, RH forced me to decorate ShowWindowDelegate with a calling convention attribute. I am not sure if this, along with DNNE requirements on calling conventions, can play a role.

Here's Some Numbers for Reloaded though Memory is a complicated topic ahahaha.

Yes, I see! Fortunately I don't have to deal with games hooking, otherwise I would become crazy!

EH puts your injected library into a new AppDomain and kills that AppDomain when the library is unloaded. With Core 3/.NET 5 it's the same but with AssemblyLoadContext(s). Your Assembly goes into a separate AssemblyLoadContext (when using native hosting method/DNNE/Bootstrap and that AssemblyLoadContext can be killed to unload your Assembly).

I suppose that I should expect at least the same memory usage of EH and that those crazy 31 Mb on a 5-6 Mb Notepad x86 app are due to something wrong with my code.

I really don't care, but my concern is that the whole net core runtime will remain loaded into the target app and this is not light at all.

Pretty sure this will be the case regardless of whether you roll for Framework or Core. If you need to go lightweight, native code's probably the only way. If you're hooking a native application then it's likely the Visual C++ runtime will be loaded already which will make it even lighter on memory.

CoreRT/NativeAOT might be an option but it's really experimental and hacky; you'll likely have a bad time.

This means that, in the specific attached sample, there's no way to also unload NetCoreLoader and its DNNE CoreCLR once I am done with hooking. It will remain into the application domain for all the lifetime of the app. I have looked at the sample code, but wait a moment! It is identical to my NetCoreLoader !!! If I am not wrong, it will convert my NetCoreLoader into an unmanaged version which does not requires the CoreCLR to be loaded like DNNE. Once I have that converted unmanaged assembly it's unclear on how to connect to the injector. Should I expect my functions to be called just like happens now? And should I expect Reloaded.Hooks to work as does now? Or I should pinvoke something? I suppose that what I learned on DNNE should give me some advantages and make the process easier, but you said that I'll likely have a bad time and this is scaring me. Can I have some hopes to have less bad time considering that I already had bad time with DNNE? Ah ah ah So, the idea would to use DNNE when hooking managed apps and this one when hooking native apps.

there must be some sort of sharing mechanism. I suppose that if there is one app that uses the CoreCLR, then the CCLR is loaded in the system. If a second app needs the CCLR, I suppose it will share the same CCLR loaded from the first app. I can't believe that each app will use their own CCLR "instances" because a system with limited resources will go down soon.

There is a solution, employed at the operating system level. Consider studying about topics such as Virtual Memory and Paging. Here's a Random Article I Like

It seems that this does not even happens automatically and requires entering into more deep stuff. Thanks for sharing the article. There are really so many complicated things that one should be aware of. However, as a general rule, I learnt that when task manager shows low memory usage there's nothing to worrying about. When memory showed is high, may be it's still does not make difference, but we need to check with tools (vmmap and similars, actually I'm using TaskExplorer...I really like how it works). The sample uses the PerformanceCounter package from Microsoft which I saw is the most recommended one for debugging.

A proper answer would be really complicated but an important takeaway is that memory occupied by DLLs are shared across processes.

Oh yes, very complicated. Today we have so many apps running simultaneously and in the background, therefore this should help to decrease memory usage. Problem is that I would like to know how much memory I am saving when running x number of applications in order to have an idea and do better optimization, but I am afraid this is not an easy job.

Machine code compiled by the JIT will of course be local to the current process and not shared but the runtime is compiled with an optimization called ReadyToRun which includes machine code alongside IL code, and, because it's part of the DLL it will be shared.

Oh yes, In fact I can see that the memory occupied by my Loader and Hooker is nearly nothing. However what they reference is much more. Start up time is something that I have not taken in consideration for now. it's too soon to give any judge with a rude prototype like mine. I hope at least to not experience delay which would require further investigations.

Assuming you mean WPF libraries that are part of the runtime, I'm going to have a guess about what's going on based on my experiments with tampering with AssemblyLoadContext(s) in Reloaded-II.

When loading WPF libraries by name inside your own AssemblyLoadContext; the ALC is unable to find the library. Therefore, the runtime attempts to load the library in the Default ALC; which succeeds because it knows where WPF (Microsoft.WindowsDesktop.App) is located. Because the library is loaded in the Default ALC, it therefore cannot be unloaded because the Default ALC cannot be unloaded.

Relevant MSDN Documentation

Yes, I mean WPF libraries. Mhhh...if I remember well in my case the ALC found the libraries without problems. I means, the resolver got them and loaded in the custom ALC (WindowsBase.dll, PresentationCore.dll, PresentationFoundation.dll and probably their related dependencies. The core stuff clearly said that there's no support for unload framework assemblies either as a whole or per-assembly and even dynamic loading. I am not sure on the exact reason you failed to load into the custom context, but you can stay sure that once they enter in your custom context, you will be unable to kick them away. And the core team have no intention to implement such capability because is very hard to make it work as expected. If you have a console application or whatever and need to occasionally display a wpf library or app for which you have the source code, then the best way I can think of is to recur to IPC for data exchange and then close the connection once done. There will be no trace left and the IPC implementation is very easy.

Note: I have managed to load WPF into a custom context before; however it's something I wouldn't recommend. You have to parse the RuntimeConfig and manually hunt for the install location of the Microsoft.WindowsDesktop.App to accomplish it; then load from file path inside your own ALC.

Here's a commit where I removed loading WPF & ASP.NET into separate ALCs from an experimental branch of Reloaded-II. I'll only ever re-consider adding it when it's necessary and take the minor memory savings for now with the default behaviour :P.

Mhh...very interesting! In the next few months I would like to resume a few projects where I need some WPF on demand. One of them is just a tray icon which stay on the task bar for all the time, but occasionally needs to display custom user interface, progress bar etc. I will remember to check your experimental branch when it will come the time. My plan was to exchange data through IPC, but would be amazing if I could access the whole thing.

...and I have no idea of the implications (may be running the gui mod and my app at the same time?). Yeah, Reloaded-II is a standalone program. Specifically, it's a mod loader for modifying native applications and games using .NET. I mentioned it because it's a really easy way to get your .NET code running inside a native process; so you could use it for testing the hook code in the DLL you plan to inject. Thanks for the sample. You got really a nice idea. I tried to make it work, but without success. I supposes that I must have a game running and the one that is more familiar to me is Sonic (I liked so much Sonic Adventures on Dreamcast and when I was young I spent entire days playing Sonic on Megadrive, should be Sega Genesis in your country!). Problem is that out there there are hundreds of different versions and I have no idea what is the right one for Reloaded-II.

@Sewer56 I don't know where all of this will bring me and If I will ever come with a final working project. If this happens what you think to make it part of your respository? May be a new entry like Reloaded.RemoteHooks ? After all we put efforts on this and leaving it as something to just for my personal use would be a shame. What you think? For sure this will be a very unique library, not just like all the others.

So...for now my next step is to see what I can do with CoreRT/NativeAOT. Is there any other alternative? Just to see if there is something better. This one comes from Microsoft and this is for sure a valid reason to stick with it without looking elsewhere.

I hope that luck will assist me!

EasyHook_Vs_ReloadedHooks.zip

netcorefan1 commented 3 years ago

Just some news and new discoveries to integrate with my previous response... I wanted to see what was going on inside NetCoreLoader and NetCoreHooker and this is what showed the PerformanceCounter:

NetCoreLoader.Loader (before calling InitializeContext): 16Mb
NetCoreHooker.Hooker.StartShowWindowHook (before calling CreateShowWindowHooker): 17 Mb
NetCoreHooker.Hooker.StartShowWindowHook (after calling CreateShowWindowHooker): 37 Mb (uahhh!)
Inside ShowWindowCallback: 36 Mb
Loader.StopHooking (before calling Unload): 37 Mb
Loader.StopHooking (after calling Unload and the forced GC): 19 Mb (it's unclear why in the screenshot I previously posted memory usage remains pretty much the same while here I can notice a consistent reduction)

And I have just discovered this as soon as the RH instance is created:

mscoree.dll  | Microsoft .NET Runtime Execution Engine | C:\Windows\SysWOW64\mscoree.dll                              
mscoreei.dll | Microsoft .NET Runtime Execution Engine | C:\Windows\Microsoft.NET\Framework\v4.0.30319\mscoreei.dll

Basically, to me seems to be that we're running .Net Framework and .Net core at the same time. I am not sure if this could be due to the list of assemblies previously posted which I have not been able to unload, netstandard in primis which should be the main bridge to .Net Framework. I also have some suspect on PeNet.dll. It doesn't show up like the other assemblies, but it is causing execution delay and sometimes more than 10 consecutive exceptions:

Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll
Exception thrown: 'System.Exception' in PeNet.dll

I have not been able to discover where you reference PeNet.dll, but looking on Nuget I noticed some dependencies on netstandard2.1 and I am not sure if this library is responsible for loading .Net Framework runtime.

I will have to completely revise the NetCoreLoader.Loader. Everything I call here will stay loaded in the default context for the whole life of the application. I will have to see what I can take out and if possible, even reflection (I don't have any idea on how). This should be no more than a loader and I am wondering if could work if I do something like:

This should allow to save on memory and on the number of assemblies loaded into the target application. I don't need any further communication once the loader has done its launching process because NetCoreHook will communicate with the client through IPC. This should also simplify CoreRT/NativeAOT If the code to write is the same as I suspect (but let me see how that lib works since it could be completely different of what I believe). My main concern is that I am not sure if DNNE supports a similar scenario, but even an hack would be fine as long as it is reliable... Let see...

P.S. I forgot to tell you that in the new sample, RH cause Notepad to crash finally, but I didn't searched further (I just didn't thought to write a perfect closure and wanted something quick to test). Also, the interception is semi-working. You have to manually activate Notepad focus and/or manually move mouse cursor over its window in order to raise the ShowWindow event. In EasyHook it works as expected, but only because I can launch the process suspended, activate the hook and then resume the process and catch its first ShowWindow event. This is not a problem of RH, but it relates to the injector which does not support such kind of injection scenario. I'll think to this at the final stage.

netcorefan1 commented 3 years ago

...OK! I got NetCoreLoader compiled as native dll using CoreRT/NativeAOT. The injection succeeds, but fails to find the functions. The readme says "Exported methods cannot be called from regular managed C# code, an exception will be thrown.". Seems that I have to call from unmanaged code. They provided a sample on how to call the functions from unmanaged code and I should be able to sort out this, but once I have the "unmanaged loader" how I connect it to the reloaded injector? I am afraid it has come the time to think to the injector, sooner that I thought... Not sure on how to proceed. May be return to the posted sample and make it work with GH Injector and then, once done, If I am lucky enough, it should work out of the box even with CoreRT/NativeAOT. Easy to say, but I already know it will not...

IncPlusPlus commented 2 years ago

Hi @netcorefan1. Did you ever find a working library to get managed code injected and running within the context of an unmanaged process? I've tried so many different libraries and code examples and I've found nothing that works.

julianxhokaxhiu commented 2 years ago

Hello @netcorefan1 I'm also looking for an EH alternative especially that works across .NET 6.

I noticed you've got already an example project running there, would you mind sharing it? It could be a starting point for me to understand how to glue Reloaded into my own project, replacing EH.

Thank you in advance!

//EDIT: Nvm, I didn't notice this link: https://github.com/Reloaded-Project/Reloaded.Hooks/files/5915723/EasyHook_Vs_ReloadedHooks.zip thank you again!