Incorrect subsystem on windows with VS2022

octopoulos commented 1 year ago

Just following the basic guide:

> mkdir ~/scratch
> cd ~/scratch
> git clone https://github.com/floooh/sokol-samples
> cd sokol-samples
> ./fips set config sapp-d3d11-win64-vstudio-debug
> ./fips build

=> error LNK2019: unresolved external symbol main referenced in function "int __cdecl invoke_main(void)" (?invoke_main@@YAHXZ)

The reason is that the main hijack is not working, because it's being overridden by the project subsystem, which is set to Console (instead of NotSet).

    <Link>
      ...
      <SubSystem>Console</SubSystem>
    </Link>

If I change the SubSystem to Windows or NotSet then it works, but it would be nice if the tool could do it automatically, of if the hijack could override that setting.

octopoulos commented 1 year ago

Note that a solution might just be to always define main( whenever WinMain( is defined. (in sokol_app.h)

floooh commented 1 year ago

This is weird, the subsystem should be selected via fips_begin_app(... windowed) (requires WinMain) vs fips_begin_app(... cmdline) (requires main). Let me check if I have a CMakeLists error somewhere...

...also the CI should actually fail in that case...

floooh commented 1 year ago

Hmm, I just tried the process from scratch and it works here.

Can you give a bit more context?

the output lines from cmake following === generating: sapp-d3d11-win64-vstudio-debug
the output lines around the actual error, e.g. which of the samples is failing
cmake and Visual Studio version
output of fips diag tools

floooh commented 1 year ago

For instance on my system:

=== generating: sapp-d3d11-win64-vstudio-debug
config file: E:/scratch/sokol-samples/fips-files/configs/sapp-d3d11-win64-vstudio-debug.yml
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  FIPS_AUTO_IMPORT="OFF"
  FIPS_CONFIG="sapp-d3d11-win64-vstudio-debug"
  FIPS_LOCAL_BUILD="OFF"
  SOKOL_USE_D3D11:BOOL="ON"
  USE_SOKOL_APP:BOOL="ON"

-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22000.0 to target Windows 6.2.9200.
-- The C compiler identification is MSVC 19.34.31935.0
-- The CXX compiler identification is MSVC 19.34.31935.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE: Debug
FIPS_ROOT_DIR: E:/scratch/fips
FIPS_PROJECT_DIR: E:/scratch/sokol-samples
FIPS_BUILD_DIR: E:/scratch/fips-build
FIPS_DEPLOY_DIR: E:/scratch/fips-deploy
FIPS_PROJECT_NAME: sokol-samples
FIPS_AUTO_IMPORT: OFF
FIPS_CONFIG: sapp-d3d11-win64-vstudio-debug
FIPS_PROJECT_BUILD_DIR: E:/scratch/fips-build/sokol-samples/sapp-d3d11-win64-vstudio-debug
FIPS_PROJECT_DEPLOY_DIR: E:/scratch/fips-deploy/sokol-samples/sapp-d3d11-win64-vstudio-debug
Host system: Windows
CMAKE_CXX_COMPILER_ID: MSVC
FIPS_MSVC: 1
FIPS_PLATFORM: WIN64
FIPS_PLATFORM_NAME: win64
-- Found Python: C:/Users/floh/scoop/apps/python/current/python.exe (found version "3.12.0") found components: Interpreter
-- Configuring done (4.5s)
-- Generating done (0.5s)
-- Build files have been written to: E:/scratch/fips-build/sokol-samples/sapp-d3d11-win64-vstudio-debug
cmake --build . --config Debug --
MSBuild version 17.4.0+18d5aef85 for .NET Framework

cmake version 3.27.1

Microsoft Visual Studio Community 2022 (64-bit) - Current Version 17.4.2

λ fips diag tools
=== tools:
git:    found
cmake:  found
cmake-gui:      found
make:   found
ninja:  found
javac:  found
java:   found
vscode: found
vscode-cpptools:        found
vscode-cmaketools:      found
vscode-codelldb:        found
wasm-dwarf-debugging:   OPTIONAL, NOT FOUND (https://marketplace.visualstudio.com/items?itemName=ms-vscode.wasm-dwarf-debugging)
clion:  OPTIONAL, NOT FOUND (used as IDE with clion configs)
http-server:    found
wasmtime:       OPTIONAL, NOT FOUND (required for running WASI executables)

floooh commented 1 year ago

Also, when I open Visual Studio (via fips open), and look at the properties of the clear-sapp sample, the subsystem is set to Windows:

(none of the samples under sapp/ should actually use the Console subsystem)

...looking at the generated .vcxproj files, all targets are set to SubSystem Windows (which is correct but seems to be different on your side):

Something must be wrong about the cmake generation step I think...

octopoulos commented 1 year ago

Ok redoing it from scratch: C:> mkdir scratch C:> cd scratch C:\scratch> git clone https://github.com/floooh/sokol-samples C:\scratch> cd sokol-samples C:\scratch\sokol-samples> ./fips set config sapp-d3d11-win64-vstudio-debug ... C:\scratch\sokol-samples> ./fips build ...

=== building: sapp-d3d11-win64-vstudio-debug
=== generating: sapp-d3d11-win64-vstudio-debug
config file: C:/scratch/sokol-samples/fips-files/configs/sapp-d3d11-win64-vstudio-debug.yml
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  FIPS_AUTO_IMPORT="OFF"
  FIPS_CONFIG="sapp-d3d11-win64-vstudio-debug"
  FIPS_LOCAL_BUILD="OFF"
  SOKOL_USE_D3D11:BOOL="ON"
  USE_SOKOL_APP:BOOL="ON"

-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19042.
-- The C compiler identification is MSVC 19.37.32824.0
-- The CXX compiler identification is MSVC 19.37.32824.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE: Debug
FIPS_ROOT_DIR: C:/scratch/fips
FIPS_PROJECT_DIR: C:/scratch/sokol-samples
FIPS_BUILD_DIR: C:/scratch/fips-build
FIPS_DEPLOY_DIR: C:/scratch/fips-deploy
FIPS_PROJECT_NAME: sokol-samples
FIPS_AUTO_IMPORT: OFF
FIPS_CONFIG: sapp-d3d11-win64-vstudio-debug
FIPS_PROJECT_BUILD_DIR: C:/scratch/fips-build/sokol-samples/sapp-d3d11-win64-vstudio-debug
FIPS_PROJECT_DEPLOY_DIR: C:/scratch/fips-deploy/sokol-samples/sapp-d3d11-win64-vstudio-debug
Host system: Windows
CMAKE_CXX_COMPILER_ID: MSVC
FIPS_MSVC: 1
FIPS_PLATFORM:
FIPS_PLATFORM_NAME:
-- Found Python: C:/Code/Python38/python.exe (found version "3.8.1") found components: Interpreter
-- Configuring done
-- Generating done
-- Build files have been written to: C:/scratch/fips-build/sokol-samples/sapp-d3d11-win64-vstudio-debug
cmake --build . --config Debug --
MSBuild version 17.7.2+d6990bcfa for .NET Framework

All the samples fail, ex: first 2 errors:

  ## sokol-shdc: C:/scratch/sokol-samples/sapp/noentry-sapp.glsl C:/scratch/fips-build/sokol-samples/sapp-d3d11-win64-vstudio-debug/sapp/noentry-sapp.gls
  l.h {'slang': 'hlsl4', 'compiler': 'MSVC'}
  Building Custom Rule C:/scratch/sokol-samples/sapp/CMakeLists.txt
  Building Custom Rule C:/scratch/sokol-samples/libs/sokol/CMakeLists.txt
  sokol.c
  sokol.vcxproj -> C:\scratch\fips-build\sokol-samples\sapp-d3d11-win64-vstudio-debug\libs\sokol\Debug\sokol.lib
  Building Custom Rule C:/scratch/sokol-samples/sapp/CMakeLists.txt
  arraytex-sapp.c
MSVCRTD.lib(exe_main.obj) : error LNK2019: unresolved external symbol main referenced in function "int __cdecl invoke_main(void)" (?invoke_main@@YAHXZ) [
C:\scratch\fips-build\sokol-samples\sapp-d3d11-win64-vstudio-debug\sapp\arraytex-sapp.vcxproj]
C:\scratch\fips-deploy\sokol-samples\sapp-d3d11-win64-vstudio-debug\arraytex-sapp.exe : fatal error LNK1120: 1 unresolved externals [C:\scratch\fips-buil
d\sokol-samples\sapp-d3d11-win64-vstudio-debug\sapp\arraytex-sapp.vcxproj]
  Building Custom Rule C:/scratch/fips-imgui/CMakeLists.txt
  imgui.cpp
  imgui_demo.cpp
  imgui_draw.cpp
  imgui_tables.cpp
  imgui_widgets.cpp
  Generating Code...
  imgui.vcxproj -> C:\scratch\fips-build\sokol-samples\sapp-d3d11-win64-vstudio-debug\fips-imgui_imgui\Debug\imgui.lib
  Building Custom Rule C:/scratch/sokol-samples/libs/dbgui/CMakeLists.txt
  dbgui.cc
  dbgui.vcxproj -> C:\scratch\fips-build\sokol-samples\sapp-d3d11-win64-vstudio-debug\libs\dbgui\Debug\dbgui.lib
  Building Custom Rule C:/scratch/sokol-samples/sapp/CMakeLists.txt
  arraytex-sapp.c
MSVCRTD.lib(exe_main.obj) : error LNK2019: unresolved external symbol main referenced in function "int __cdecl invoke_main(void)" (?invoke_main@@YAHXZ) [
C:\scratch\fips-build\sokol-samples\sapp-d3d11-win64-vstudio-debug\sapp\arraytex-sapp-ui.vcxproj]
C:\scratch\fips-deploy\sokol-samples\sapp-d3d11-win64-vstudio-debug\arraytex-sapp-ui.exe : fatal error LNK1120: 1 unresolved externals [C:\scratch\fips-b
uild\sokol-samples\sapp-d3d11-win64-vstudio-debug\sapp\arraytex-sapp-ui.vcxproj]
  Building Custom Rule C:/scratch/sokol-samples/libs/basisu/CMakeLists.txt
  sokol_basisu.cpp

octopoulos commented 1 year ago

PS C:\scratch\sokol-samples> cmake --version cmake version 3.25.0

=> Oh I see, it's not the latest one, I guess I can upgrade to 3.27, who knows maybe that will fix, I'll keep you updated. Edit: I tried with cmake 3.27.7 ... same thing.

Microsoft Visual Studio Community 2022 (64-bit) - Current Version 17.7.4

=> yours is showing stack reserve size, while mine shows nothing there. I'll check with the latest CMake. (edit: done)

By the way, I forgot to thank you for the prompt reply!

octopoulos commented 1 year ago

After comparing your file and mine, I see that the main difference is this, you have:

FIPS_PLATFORM: WIN64
FIPS_PLATFORM_NAME: win64

and I have:

FIPS_PLATFORM:
FIPS_PLATFORM_NAME:

That's probably the issue, right?

floooh commented 1 year ago

That's probably the issue, right?

Wow yeah, let me check how this is detected...

floooh commented 1 year ago

Hmm, still weird. When FIPS_PLATFORM isn't set it means it's not including this windows.cmake file here:

https://github.com/floooh/fips/blob/master/cmake-toolchains/windows.cmake

That should happen here when FIPS_HOST_WINDOWS is set:

https://github.com/floooh/fips/blob/36ae6b60a5d6d8c6a5661f8cf56b474c28d9bdaa/cmake/fips.cmake#L164-L172

...but that is definitely set because the message Host system: Windows shows up in your log:

https://github.com/floooh/fips/blob/36ae6b60a5d6d8c6a5661f8cf56b474c28d9bdaa/cmake/fips.cmake#L115-L118

Your Visual Studio seems to be a bit more recent... currently updating.

One more thing, are you building on PowerShell? If yes can you try on a "vanilla" cmd.exe (e.g. also not a "Visual Studio Command Prompt"). It should still work, but these are the only other differences I can think of.

I'll continue investigating tomorrow, it's pretty late now :)

floooh commented 1 year ago

Btw, can you also check this project:

https://github.com/floooh/pacman.c

This isn't using my fips cmake wrapper but a regular CMakeLists.txt instead. Build instructions are in the readme. If this also fails (which I don't expect), then it would be some incompatibility between cmake and the latest VS (unlikely though).

floooh commented 1 year ago

Hmm, updated to VS 17.7.5 and it's still working here, both on cmd.exe and in Powershell.

Running out of ideas...

floooh commented 1 year ago

Ok, one last thing before I hit the sack:

Can you add a couple of debug-messages to the fips directory that's been checked out next to sokol-samples:

...in fips/cmake/fips.cmake around line 164:

    if (NOT CMAKE_TOOLCHAIN_FILE)
        message("######### in NOT CMAKE_TOOLCHAIN_FILE")
        if (FIPS_HOST_WINDOWS)
            message("######### including windows.cmake")
            include(${FIPS_ROOT_DIR}/cmake-toolchains/windows.cmake)
        elseif (FIPS_HOST_OSX)
            message("######### including osx.cmake")
            include(${FIPS_ROOT_DIR}/cmake-toolchains/osx.cmake)
        else()
            message("######### including linux.cmake")
            include(${FIPS_ROOT_DIR}/cmake-toolchains/linux.cmake)
        endif()
    endif()

... then run fips gen in the sokol-samples directory. Here it looks like this:

FIPS_PROJECT_DEPLOY_DIR: E:/scratch/fips-deploy/sokol-samples/sapp-d3d11-win64-vstudio-debug
Host system: Windows
CMAKE_CXX_COMPILER_ID: MSVC
FIPS_MSVC: 1
######## in NOT CMAKE_TOOLCHAIN_FILE
######### including windows.cmake
FIPS_PLATFORM: WIN64
FIPS_PLATFORM_NAME: win64
-- Configuring done (0.5s)

I basically want to check if windows.cmake is actually included.

floooh commented 1 year ago

PS: do you have an environment variable CMAKE_TOOLCHAIN_FILE set by any chance? Because this would definitely confuse this piece of code.

floooh commented 1 year ago

Anyway logging out now :)

octopoulos commented 1 year ago

Hi, just letting you know that I will check everything tomorrow, I just had a full day trip. Note: I think the best way to fix would be to have CMakeLists.txt specify the subsystem:Windows. So, expect some news tomorrow.

octopoulos commented 1 year ago

Pacman works fine.

message("######### in NOT CMAKE_TOOLCHAIN_FILE")

Bingo!

I edited the fips.cmake to remove the toolchain check, and then it compiles fine on windows. I also tried emscripten a few days ago and that worked fine too.

floooh commented 1 year ago

message("######### in NOT CMAKE_TOOLCHAIN_FILE")

But this is entirely correct.

What's concerning is that there is no ######### including windows.cmake message (or is there?).

When the windows.cmake file is included, it should work because this should set the correct subsystems, otherwise such problems are expected.

floooh / sokol-samples

Incorrect subsystem on windows with VS2022 #126