Option to use debug mode compilation with symbols for mod dll-s

[levitation]

When I am developing mods, sometimes I would like to attach my Visual Studio debugger to a process that has the mod loaded, and see what is happening inside.

Right now the mods are compiled with clang and with no symbols. Probably the code is optimised to release mode as well.

So I am wondering, if it would be possible to have a Windhawk configuration option where it invokes compiler and linker for my mod (or for all mods, if that is simpler) with the following properties:

• Debug symbols are enabled.
• Mod is compiled in debug mode, so that the code is less obfuscated.
• Maybe it is also necessary that compiler and linker is of Visual Studio tools instead of clang. I am unsure about that - I read that Visual Studio might support debugging clang compiled code, but have not tested it.

[m417z]

Have you tried using @compilerOptions? Windhawk uses a command similar to the following:

g++.exe -std=c++23 -O2 -shared -D<defines> ${engineLibPath} ${modSourcePath} -include windhawk_api.h -target ${bits === 64 ? 'x86_64-w64-mingw32' : 'i686-w64-mingw32'} -o ${compiledModPath} ${compilerOptions}

@compilerOptions are added at the end. I didn't try it, but perhaps specifying flags such as -O0 -g will override the previous -O2 flag.

[levitation]

Thank you! Yes, with these flags it is possible to debug the dll using Visual Studio Code.

In contrast, Visual Studio Community 2019 does not support the symbol format compiled for mingw. I tried compiling with --target=x86_64-pc-windows-msvc "--include-directory=C:/Program Files/Windhawk/Compiler/include/" but this resulted in compilation errors.

For debugging the mingw dll, I installed the following VS Code extensions:

• C/C++ for Visual Studio Code
• CodeLLDB

Then I use CodeLLDB as a debugger.

It nicely shows the source code, the variables, is able to set breakpoints and to step through the code.

My launch.json looks like following when Notepad is the example debug target:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "lldb",
            "request": "launch",
            "name": "Debug",
            "program": "C:\\Windows\\System32\\notepad.exe",
            "args": [],
            "cwd": "${workspaceFolder}"
        }
    ]
}

Currently I do not know how to attach to a running process with VS Code, maybe I will figure it out later.

[m417z]

I tried compiling with --target=x86_64-pc-windows-msvc "--include-directory=C:/Program Files/Windhawk/Compiler/include/" but this resulted in compilation errors.

We discussed it in Discord about a year ago:

OK, so I was able to hack something around. I did the following:

• Start from the new-mod template
• Add these metadata items:

  // @compilerOptions -luser32 -lcomdlg32 -target x86_64-pc-windows-msvc -I C:\include -v
  // @architecture    x86-64

• Add this to the code:

  #pragma comment(linker, "/EXPORT:_Z10Wh_ModInitv=?Wh_ModInit@@YAHXZ")
  #pragma comment(linker, "/EXPORT:_Z12LoadSettingsv=?Wh_ModSettingsChanged@@YAXXZ")
  #pragma comment(linker, "/EXPORT:_Z12Wh_ModUninitv=?Wh_ModUninit@@YAXXZ")
  #pragma comment(linker, "/EXPORT:InternalWhModPtr=?InternalWhModPtr@@3PEAXEA")

• Copy to C:\include: windhawk_*.h files from Windhawk's Compiler folder
• Copy to C:\Program Files\Windhawk\Compiler\lib\clang\15.0.0\lib\windows: Files from C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\lib\clang\16\lib\windows
• Add to path: C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64
• Add to path: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\Llvm\x64\bin

Not sure if it works with VS 2019 or with newer Windhawk versions, at the very least the paths have to be adjusted. It was just an experiment and in any case looks like you found a better solution.

For debugging the mingw dll, I installed the following VS Code extensions

Very nice! I thought about including a debugger with Windhawk, but then decided that it's not worth the extra disk space, as Windhawk takes a lot of disk space already, and most users probably won't be using it. I never got to actually check what it takes to make it work.

A tutorial for debugging mods would be a great addition to the wiki, here in Development tips or in a new, dedicated page. Would you like to write such a guide?

[levitation]

Thank you for the info about Microsoft Visual Studio! I will experiment with this at some time.

Yes, I would be happy to create a wiki page about mod debugging with VS Code.

Found a way to attach VS Code to an existing process:

1. Open launch.json
2. Click on bottom right "Add configuration...". A popup menu will open in the middle of configuration file.
3. Choose "CodeLLDB: Attach by Name" or "CodeLLDB: Attach to PID" options
4. In the newly added configuration block, update the "program" or "pid" field accordingly
5. Start debugger

As a sidenote, launching the processes via VS Code stopped working in my machine for some reason. So now I always use attaching instead. If needed, then I attach debugger first and then enable the mod only after that.

If you want the mod code to trigger a breakpoint, then you can use the following code

if (IsDebuggerPresent())
        DebugBreak();

IsDebuggerPresent() check is needed, else the program will crash without a debugger attached.

[m417z]

Yes, I would be happy to create a wiki page about mod debugging with VS Code.

Thanks! I temporarily set the wiki to be editable by anyone (it doesn't seem to be possible to give permission selectively). Let me know if that works for you, or if you prefer, you can write it in another place (like a gist) and I'll integrate it into the wiki.

[levitation]

Hello! I finished creating a new wiki page for mod debugging now.

How does it look to you, would you like me to change or add anything?

[m417z]

Very nice. Several notes:

• Using Visual Studio Code worked for me, but the guide lacks several steps, such as "When you first start debugging, choose CodeLLDB as a debugger" - how? Or - it complains about ${workspaceFolder} unless you open a workspace. Also, intellisense didn't work for the mod because it isn't able to find Windhawk's headers.
• Instead of using Visual Studio Code, I thought - why not just use Windhawk? I installed CodeLLDB (Ctrl+Shift+X opens the extensions view), then pressed Ctrl+Shift+P and typed "Debug: Add Configuration...", added "CodeLLDB: Launch", changed "program" path, pressed F5, and it just worked. Ctrl+Shift+D can be used to open the debug sidebar if it's closed. The upsides of using Windhawk: No need to install and configure an extra program, easier setup, intellisense already works. What do you think? Have you tried it? Have you encountered any downsides?
• Instead of if (IsDebuggerPresent()) DebugBreak();, you can use if (IsDebuggerPresent()) __debugbreak();, which will break in the mod itself instead of inside DebugBreak which is displayed as assembly code. And if you're using Windhawk to debug, it's probably easier to just set a breakpoint, the mod's code is always open.

[levitation]

Hey, yes, indeed the instruction can be updated with regards how to use VS Code in general. Unfortunately I am not expert in its use, therefore I wrote only what I remembered. It is possible I skipped some intuitive steps.

1. Because I did not use VS Code for development, then I did not need the headers, I needed to see the mod's existing code only and just as importantly - the variables. For the mod's code, VS Code found it automatically.
2. Sounds interesting as well as promising avenue for extending Windhawk. Seems like this is something you would know much more about than me. I did not know to try this. I will test it later at some point. Though perhaps I would not be the right person to write it as an instruction, considering that this is out of my expertise. Maybe there could be another page in wiki about how to use Windhawk itself for debugging.
3. Initially I also thought that debugbreak() would be better. But when trying out both, I found that DebugBreak() worked better in my machine. For me, it actually did not break into assembly code, but into my code. debugbreak() also breaked into my code, but with the shortcoming that it showed an useless exception popup about exception 0x80000003 or something similar. Maybe DebugBreak() uses some more clever method to achieve what it does. But if it worked differently in your machine, then maybe it is better to mention both approaches, so that people can choose based on how their machines behave.

[m417z]

I updated the guide and added a video recording for each step. https://github.com/ramensoftware/windhawk/wiki/Debugging-the-mods Let me know if you think anything is missing. I removed some of the content, such as the DebugBreak part, as I don't think it's necessary in this setup.