handler.exe console-window is visible in Release builds (on Windows)

[drywolf]

Hi, In the past I have been successfully using https://github.com/unidentifieddeveloper/crashpad.git to build crashpad for use in our projects. But since https://github.com/backtrace-labs/crashpad seems to be more active & supported, I wanted to try and use it instead for building crashpad.

The build went fine, but when I now use crashpad in one of our apps that have been built in a Release config ... then suddenly I see a handler.exe console window being opened alongside the regular application window. (for Debug builds I don't see this extra window)

We never saw such a handler.exe console window when we used crashpad binaries that were built from https://github.com/unidentifieddeveloper/crashpad.git

Do you have any guess why this would be happening in the backtrace-labs fork ? Thanks

[konraddysput]

@drywolf crashpad requires from developer to pass a path to the crashpad handler. do you pass a path to the exe file or maybe to the .com one? The .exe file shouldn't show the console.

[drywolf]

I am passing the path to the .exe "./handler.exe"

Where are the .com files located after the crashpad build ? Then I can double check that they are not mistakenly copied with a changed file-extension 🤔

[KrzaQ]

We don't build the .com file right now. I think fixing this issue will be as simple as adding /subsystem:windows to the compilation flags for Windows. I'll test it ASAP.

[drywolf]

@KrzaQ I see ... if I understood you correctly about what is the problem, then adding WIN32 to the add_executable() will do the same thing. (see https://cmake.org/cmake/help/latest/command/add_executable.html)

I will also test to see if this fixes it 👍

[KrzaQ]

I could not reproduce this issue myself, I tried running my test app from PS and from the Windows run dialog directly. Do I need a GUI app for this? Alternatively: what is the CL / VS version and the SDK you're building against?

[drywolf]

what is the CL / VS version and the SDK you're building against?

This is what the CMake configure log tells me:

  -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19043.
  -- The C compiler identification is MSVC 19.16.27045.0
  -- The CXX compiler identification is MSVC 19.16.27045.0

I am using VS 2019 & the v141 C++ Toolset currently.

Do I need a GUI app for this?

I first noticed the problem when I was using it in our GUI app ... I think I tested the same crashpad binaries before in a console app (Debug & Release builds) and I did not notice this problem there. I will check again and let you know ... but I guess it could be that this only happens for Windows GUI apps.

[drywolf]

I think what we just talked about makes all the difference ... by adding WIN32 to the add_executable(handler main.cc) call, I can get rid of the handler.exe console window 🥳

I also had a look at the https://github.com/unidentifieddeveloper/crashpad.git code, and they also use the WIN32 option in their add_executable() call ...

https://github.com/unidentifieddeveloper/crashpad/blob/699ff38124c43627f21f08a74a7ae2fa63efcd5e/CMakeLists.txt#L342-L344

whereas the backtrace-labs version does not ...

https://github.com/backtrace-labs/crashpad/blob/cca5f52037abd51399a1ad88344d8236028e297a/handler/CMakeLists.txt#L47

PS: I am still wondering why the handler.exe window then was only showing in Release builds, but not Debug builds 🤔

[drywolf]

I am still wondering why the handler.exe window then was only showing in Release builds, but not Debug builds

I just figured out why ... in our GUI application (the one that uses crashpad) ... we deliberately set /subsystem:console for our Debug builds, and /subsystem:windows for the Release builds. This is so we have a console window to read debug messages while developing the App in Debug builds, but for production Release builds, there should be no console window.

So, apparently if the host-application that wants to use crashpad, itself uses /subsystem:console then the handler.exe window won't show by default (I guess that is how microsoft decided it to be). But if the host-application uses /subsystem:windows and the crashpad-handler .exe was built with /subsystem:console ... then the handler.exe window will show up 🥴

Either way ... calling the add_executable() of the crashpad-handler target with the WIN32 option easily fixes this for all cases.

[KrzaQ]

I added a potential fix to the dev branch, I want to merge it on Monday with clear head 🙂