Linux and 32 bit compile fixes

[bottiger1]

While attempting to use this library in Linux and 32bit, I ran into a quite a few issues. My build environment is Ubuntu 22.04 x64 which I would guess is probably the most common Linux build environment so I was rather surprised.

For some reason anything using ostream would segfault, so I have written alternatives which can be enabled with a cmake

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f1db217b09b in char32_t std::(anonymous namespace)::read_utf8_code_point<char>(std::(anonymous namespace)::range<char const, true>&, unsigned long)
#1  0x00007f1db217cf3f in std::codecvt_base::result std::(anonymous namespace)::utf16_in<char, char16_t>(std::(anonymous namespace)::range<char const, true>&, std::(anonymous namespace)::range<char16_t, true>&, unsigned long, std::codecvt_mode, std::(anonymous namespace)::surrogates) ()
#2  0x00007f1db217d0a4 in std::codecvt<char16_t, char, __mbstate_t>::do_in(__mbstate_t&, char const*, char const*, char const*&, char16_t*, char16_t*, char16_t*&) const ()
#3  0x00007f1db21df14a in std::ostream& std::ostream::_M_insert<unsigned long>(unsigned long) ()
#4  0x00007f1db2107c1d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > PLH::int_to_hex<unsigned long>(unsigned long) ()
#5  0x00007f1db20fff0e in PLH::x64Detour::hook()

Added some cmake options.

POLYHOOK_COMPILE_32 = add C and C++ flags to compile for 32 bits POLYHOOK_DISABLE_IOSTREAM = Replace iostream with alternative. It was crashing on my linux build for some reason. POLYHOOK_DISABLE_INSTRUCTION_PRINTING = Disable the printing of instructions in the debug log. Could make a difference when hooking hundreds of functions.

Add a bunch of ifdef in IHook.hpp to prevent linux compile errors.

Add an x86Detour constructor that accepts 32bit ints and upcasts them to 64bit behind the scenes.

[stevemk14ebr]

Can you split this into multiple PRs please. There are some features here I am willing to merge and others I am not. I recommend you use the vcpkg to manage the build, it is not necessary to change the cmake files here at all to choose the architecture and I do not wish to do that.

I would love to accept your changes to IHook to fix linux build issues, POLYHOOK_DISABLE_INSTRUCTION_PRINTING, and also the change you made to x86Detour.

Your iostream issue needs investigated more, if you are hook many things it's possible you're crashing due to memory corruption or something more specific to your application. I don't love these ifdefs all over to replace a standard output api and will not merge that.

I really appreciate your contributions! I hope we can work together to merge at least most of your work here.

[bottiger1]

Sorry I am unfamiliar with vcpkg, I thought it would be fine to just use cmake. If we can figure out how to fix the crash, then I'd prefer that of course.

I've spent an entire day trying to figure it out, and it doesn't crash when I don't use polyhook 2, and I've used this code for years without crashes, so it is very unlikely that the crashing is due to memory corruption.

It is a sourcemod shared library plugin that is loaded into a tf2 server. These are the link flags.

clang++ -m64 _home_tf2_extensions_sdk_sourcemod_public_smsdk_ext.o extension.o _home_tf2_extensions_exception_exception_public.o _home_tf2_extensions_sdk_sourcemod_public_CDetour_detours.o _home_tf2_extensions_sdk_sourcemod_public_asm_asm.o _home_tf2_extensions_sdk_sourcemod_public_libudis86_decode.o _home_tf2_extensions_sdk_sourcemod_public_libudis86_itab.o _home_tf2_extensions_sdk_sourcemod_public_libudis86_syn_att.o _home_tf2_extensions_sdk_sourcemod_public_libudis86_syn_intel.o _home_tf2_extensions_sdk_sourcemod_public_libudis86_syn.o _home_tf2_extensions_sdk_sourcemod_public_libudis86_udis86.o _home_tf2_extensions_sdk_sourcemod_public_amtl_compat_stdcxx.o libvstdlib_srv.so libtier0_srv.so -lm -lm -lgcc_eh -static-libstdc++ -L/home/tf2/extensions/sdk/PolyHook_2_0/_install/lib64 -lPolyHook_2 -lasmtk -lasmjit -lZydis -lZycore /home/tf2/extensions/sdk/hl2sdk-tf2/lib/linux64/mathlib.a /home/tf2/extensions/sdk/hl2sdk-tf2/lib/linux64/tier1.a -shared -o skialmods.ext.so

[stevemk14ebr]

How are you loading your extension into the process is it following all the C++ and CRT initialization steps

[bottiger1]

I don't believe something like CRT is needed on Linux.

The extension is loaded in a very normal way I believe, through dlopen (the equivalent of LoadLibrary) probably. It's done through the sourcemod https://github.com/alliedmodders/sourcemod/blob/e60c6724858ccd32e4df21cac5b8faee581ba329/core/logic/LibrarySys.cpp#L248

There is nothing crazy involved like manual mapping to hide from anti-cheat or the OS. This is just a mod and not a cheat.

[stevemk14ebr]

You'll have to investigate that crash more to find the root cause. It could be some kind of linking issue, or something more nefarious I can't know unless you provide a self contained reproducer. Does it crash if you don't hook anything and call the output API yourself

Are you interested in submitting separate PRS for the parts mentioned above?

[bottiger1]

I haven't tried just calling the output api.

I also wanted to say that I've ran this (before adding polyhook2) through valgrind and I think if there was some memory corruption going on, it would have showed up there, so I am quite certain this isn't an issue with my code. Maybe it is because the srcds server is using an outdated version of stdc++ runtime? who knows.

I probably should have ran this polyhook2 version on with valgrind but I've just switched to using the safetyhook library instead.

I'll maybe edit out the ostream replacement later, but I got no idea how to use vcpkg.

The reason I'm working on this is because TF2 64 bit came out and I'm trying to port everything to it before Valve randomly deletes the 32bit version one day.

[stevemk14ebr]

It's possible there's some kind of linking issue or dependency resolution issue with your .so and the target application. I simply don't know it's the first I've heard of this issue and without investigating the root cause yourself I can't help I'm sorry

[stevemk14ebr]

I'm going to close this PR for now, please open new PRS for each of the other changes you made. I would at least like to get the code changed you made to fix Linux builds in, without the cmake changes

[bottiger1]

Good news, I found out the reason for the crash is because I was statically compiling libstdc++ into the shared library with the flag "-static-libstdc++". When I removed it, the crashes stopped.

As long as an int bigger than 8 bytes is passed into cout, it causes this crash.

std::cout << (uint64_t)123;

Unfortunately I'm not sure if it will be safe to remove static libstdc++ as I sometimes have servers with different OS versions which means they have different libstdc++.

This is the only issue I've ever experienced with a static libstdc++. So I'm still wanting to use static libstdc++, so I am unable to use this library without the removal of iostream stuff.

[bottiger1]

Also I can't figure out how to move these cmake settings into vcpkg. I did some googling and it just said to put them in cmake?

I don't see any vcpkg configuration files in the github.