stevemk14ebr / PolyHook_2_0

C++20, x86/x64 Hooking Libary v2.0
MIT License
1.58k stars 222 forks source link

A windows app that uses std::stringstream crashes when PolyHook_2.lib is included in the link. #162

Closed rjx-ray closed 1 year ago

rjx-ray commented 1 year ago

Background: I want to use PolyHook_2_0 with an existing openFrameworks app. I am using the overnight build of openFrameworks as referenced at https://forum.openframeworks.cc/t/note-nightly-builds-recommended-over-0-11-2/41220. This fully supports C++17. As a starting point I added PolyHook_2.lib to the linker inputs and let it load PolyHook_2.dll at start up. I didn't reference any PolyHook code at all but immediately got a crash which I traced back to the destructor of the ofLog class - https://github.com/openframeworks/openFrameworks/blob/master/libs/openFrameworks/utils/ofLog.h. The destructor somehow goes through to PolyHook_2.dll!_free_base() where the block is seen as invalid. I found that the crash did not happen if I removed the class variable std::stringstream message from the class.

Steps to reproduce In Visual Studio 22 create a new C++17 console application Change ConsoleApplication1.cpp code to be

#include <iostream>
#include <sstream>
#include <Windows.h>
class myLog {
public:
    std::stringstream unused;
    myLog(const std::string s) { std::cout << s << std::endl; }
};
int main() {
    myLog("Hello from MyLog");
    Sleep(100000);
}

Add PolyHook_2.lib to the linker inputs and copy PolyHook_2.dll to ConsoleApplication1\x64\Debug. Build and run and the app crashes as above

Any ideas why this is happening and what I could do to prevent the crash?

rjx-ray commented 1 year ago

I updated the app code in the above post to add the includes

stevemk14ebr commented 1 year ago

You should first check the scope of all your classes, things get unhooked in the destructors.

Then you should ensure your headers match the same version of the lib you're linking, header mismatches can cause weird behavior.

If the issue is still present then you'll need to debug on the assembly level or create a reproducible test case I can investigate.

rjx-ray commented 1 year ago

Thanks for looking at this, here's some more info and clarification

Firstly I simplified my example VS2022 C++ console app code down to

#include <iostream>
#include <sstream>
#include <Windows.h>

void foo() {
    std::stringstream unused;
}
int main() {
    foo();
    std::cout << "foo returned successfully" << std::endl;
    Sleep(100000);
}

This is the entire code for the example, It has no other source code modules. Without PolyHook_2.lib added to the linker inputs: it runs normally and outputs the message. With PolyHook_2.lib added to the linker inputs: it hits the check in PolyHook_2.dll!_free_base() where the block is seen as invalid.

I am not using PolyHook at all right now in this example, its just to demonstrate the issue in a simple reproducible manner. I am only including the three system headers.

I did a freash pull and a manual build of PolyHook2 exactly as described in the repository read me. Both PolyHook_2.lib and the application are built with Visual Studio toolset 143, using C++17 x64 and Windows SDK 10.0.19041.0.

The crash comes from the destructor of the stringstream on leaving foo(), the assembly code is:

00007FF69A7D349D  call        std::basic_stringstream<char,std::char_traits<char>,std::allocator<char> >::`vbase destructor' (07FF69A7D1136h) 

I haven't found a way to step into the destructor of the stringstream to examine its code, I'm unfamiliar with assembler level debugging.

If I change the type of the unused stringstream variable to be std::string I do not get the crash.

rjx-ray commented 1 year ago

Sorry closed by accident - finger trouble - now reopened

stevemk14ebr commented 1 year ago

Ok thanks for the additional information, that seems like enough for me to try and reproduce this. To confirm, when you observe the crash you're just adding polyhook to the console application as a library dependency via the linker, and not say injecting it as an external DLL, correct?

Edit: if you can upload a compiled executable of the above code (the crashing one w/ polyhook) and it's .pdb file I can debug this at the assembly level very quickly.

rjx-ray commented 1 year ago

Yes, that's exactly right, here's the files Debug.zip

stevemk14ebr commented 1 year ago

So not 100% sure yet what the exact reason for the crash, but the general problem is related to the way polyhook is linking in I think. Look at this:

image

The executable you generated is importing std:: functions related to ostreams and streambufs from polyhook rather than the runtime like it should. These statically linked polyhook implementations end up being called instead of the dynamic implementations in vcruntime. I guess it's possible there's some kind of mismatch between the version statically linked in polyhook and the vcruntime your app uses. The actual error is caused when HeapFree is invoked in one of the std:: destructors which seems to cause heap corruption.

I'd try the following if I were you (guesses):

  1. Turn on static linking for your app, I think this should cause your app to embed the runtime and not look at polyhook for runtime implementations.
  2. Compile your app with the same debug/release flags as the polyook lib you link, perhaps this is a situation of you compiling your app in debug but polyhook in release
  3. If all else fails you could wrap polyhook, then just declspec(export) the wrapper functions you need.
stevemk14ebr commented 1 year ago

Hi, I updated the example which should show how to link everything statically and should remove this issue. Try one of those examples instead.