WindowsToolchain and VCPkg don't work well together

[MarkSchofield]

This issue is a conversation on using WindowsToolchain and VCPkg at the same time. At the minute there's a couple of problems:

1. VCPkg support is added to a CMake build by setting CMAKE_TOOLCHAIN_FILE to [path to vcpkg]/scripts/buildsystems/vcpkg.cmake (as per the VCPkg 'Getting Started' documentation). CMAKE_TOOLCHAIN_FILE can only specify a single file and is treated specially by CMake - as discussed in the CMake code-injection documentation. If the VCPkg support is specified as the toolchain, then WindowsToolchain can't be specified as the toolchain.

   The cmake-toolchains(7) documentation explains the role of the toolchain file is to provide information on compiler/utility paths, and the VCPkg support doesn't seem to fulfil that role - https://github.com/microsoft/vcpkg/discussions/22675 is a conversation that covers this. As an alternative, CMake 3.24 added support for CMAKE_PROJECT_TOP_LEVEL_INCLUDES which "provides an injection point for things like configuring package managers, adding logic the user shares between projects (e.g. defining their own custom build types), and so on". VCPkg could be enabled through CMAKE_PROJECT_TOP_LEVEL_INCLUDES, and https://github.com/microsoft/vcpkg/discussions/26681 discusses that.

2. VCPkg adds a call to scripts/buildsystems/msbuild/applocal.ps1 to copy the DLL dependencies alongside the built binaries. But applocal.ps1 expects dumpbin, llvm-objdump or objdump to be found by PowerShell's Get-Command (here). If CMake/VCPkg is run from a Visual Studio Developer prompt, then dumpbin would be found, but since WindowsToolchain sets CMake properties to specify the compiler/utility paths, then the environment path doesn't contain dumpbin and applocal.ps1 fails:

   resolve: VCPkg\scripts\buildsystems\msbuild\applocal.ps1:162
   Line |
    162 |  resolve($targetBinary)
        |  ~~~~~~~~~~~~~~~~~~~~~~
        | Neither dumpbin, llvm-objdump nor objdump could be found. Can not take care of dll dependencies.

   Using the environment path to find utilities isn't very 'toolchain-y'. If applocal.ps1 could be told the command to use, or the path to search, then it could remove its dependency on the environment.

   It is possible to opt-out of VCPkg's applocal.ps1 functionality by setting VCPKG_APPLOCAL_DEPS to OFF, but if a port/triplet produces a DLL dependency it would need to be copied through some other means, but this would unblock header-only and static-library consumption.

[MarkSchofield]

Capturing some more notes:

1. On use of vcpkg.cmake as a toolchain:

   • Specifying scripts/buildsystems/vcpkg.cmake as a CMAKE_PROJECT_TOP_LEVEL_INCLUDES seems to work without problems. The VCPkg discussion - microsoft/vcpkg#22675 - talks about the 5 ports that have 'hacks' in vcpkg.cmake, and I appear to be able to consume them all with vcpkg.cmake configured through CMAKE_PROJECT_TOP_LEVEL_INCLUDES. Regardless, it would be great to get an answer to the questions raised by microsoft/vcpkg#22675 - the most recent comment from Mar 23 2021 seems to be an entirely reasonable ask of VCPkg;

     please either document how to accomplish vcpkg integration via find_package() or include() after project(), stating any resulting drawbacks or incompatibilities, or document the reasons why a toolchain file is absolutely necessary.

     CMAKE_PROJECT_TOP_LEVEL_INCLUDES requires CMake 3.24 and higher.

   • microsoft/vcpkg#26681 actually says that:

     IIUC vcpkg.cmake is not only code injection, but also actual toolchain setup. I wonder if it could be beneficial to separate these concerns?

     I'm not sure vcpkg.cmake really is a toolchain, though, is it? Configuring a project with --trace doesn't show any attempt by vcpkg.cmake to set a compiler or point to Visual Studio. The build succeeds by virtue of CMake's built-in awareness of Visual Studio as part of the Visual Studio Generator. If I'm reading the logs correctly. But it's entirely possible that I'm not.

2. On VCPkg's applocal.ps1 requirement of dumpbin, llvm-objdump or objdump being found by Get-Command:

   • microsoft/vcpkg#557 talks about this, but the issue is closed. There are comments saying that "Vcpkg internally handles the developer prompt for you", which suggests that there's potentially something that WindowsToolchain could do to have VCPkg configure the environment correctly..? I'd hate to duplicate work, though.

   • Setting VCPKG_APPLOCAL_DEPS to OFF disables the functionality provided by VCPkg to copy dependent DLL's. A more idiomatic CMake way of achieving the same functionality would be to leverage a TARGET_RUNTIME_DLLS generator expression. The following CMake snippet will copy dependencies for a target called CommandLine:

     if(WIN32)
         add_custom_command(TARGET CommandLine POST_BUILD
             COMMAND "${CMAKE_COMMAND};-E;$<IF:$<BOOL:$<TARGET_RUNTIME_DLLS:CommandLine>>,copy;$<TARGET_RUNTIME_DLLS:CommandLine>;$<TARGET_FILE_DIR:CommandLine>,true>"
             COMMAND_EXPAND_LISTS
         )
     endif()

     This seems to generally work, but it seems to depend on the implementation of different CMake packages.

     The TARGET_RUNTIME_DLLS generator expression requires CMake 3.21 and higher.

[gouarin]

Hii @MarkSchofield, I use https://github.com/aminya/project_options for C++ projects that uses WindowsToolchain for the windows part. Few days ago, I tried to submit a new port in vcpkg for one of my project (https://github.com/microsoft/vcpkg/pull/31051). It seems that vcpkg toolchain for package creation and WindowsToolchain don't play well together. WindowsToolchain tries to identify the CMAKE_SYSTEM_PROCESSOR and vcpkg toolchain too but they don't have the same behavior in the end (if I understood correctly).

see https://github.com/microsoft/vcpkg/blob/master/scripts/toolchains/windows.cmake#L7-L15

Do you know how to use the vcpkg toolchain even if I use your package in project_options. I'm not an expert, so maybe my question is not well defined... Let me know.

[aminya]

@gouarin thanks for the update. I can add an option in project_options to allow skipping the inclusion of the Windows toolchain. Then you could pass this option in the vcpkg port.

[gouarin]

Thanks @aminya. It would be very useful!

[aminya]

@gouarin I released v0.28.0, which fixes the parallel configure issues. Could you update your vcpkg PR?

[dg0yt]

There are a couple of problems when using WindowsToolchain and VCPkg together

Did anybody try to use vcpkg's canonical integration with external toolchain files, VCPKG_CHAINLOAD_TOOLCHAIN_FILE?

[MarkSchofield]

Thanks, @dg0yt! VCPKG_CHAINLOAD_TOOLCHAIN_FILE looks like a great solution. I'll update the documentation.

[MarkSchofield]

64 updates the documentation to talk about VCPKG_CHAINLOAD_TOOLCHAIN_FILE. I still have problems with dependency copying - applocal.ps1 still needs dumpbin, llvm-objdump or objdump to be found by PowerShell's Get-Command.

[dg0yt]

Thanks for taking the time to test.

applocal.ps1 still needs dumpbin, llvm-objdump or objdump to be found by PowerShell's Get-Command.

I know that the dumpbin dependency is to be removed - vcpkg.exe already parses DLL information for post-build checks. But IIUC it is not yet complete. I can't help with picking that path in another way - I don't have a local VS installation.

[gouarin]

@aminya thanks for the update. Do I have to give an option to disable the msvc toolchain ?

[gouarin]

Sorry, I misread your message. It seems that to solve my problem, I have to use VCPKG_CHAINLOAD_TOOLCHAIN_FILE. I will check that and give you feedback.

[aminya]

@gouarin I do not think you will even need that. I believe the error you were getting was related to some other issue that is fixed in the latest version of project_options.

Please let me know what actions are needed to be done for project_options. I'd like to fix the default behaviour. For example, if the solution is disabling WindowsToolchain for vcpkg, we need a way to automatically do that. https://github.com/aminya/project_options/blob/da045eb7f72d5a7f4fbedfc51054c105c2ac5565/src/VCEnvironment.cmake#L56

[gouarin]

@aminya if I understand correctly, I need to disable WindowsToolchain for vcpkg because it has its own defined here https://github.com/microsoft/vcpkg/blob/master/scripts/toolchains/windows.cmake

So it could be a good start to have this option and see if it's enough to be able to build vcpkg packages on windows. Thanks for your help.

[aminya]

project_options fixed this issue in v0.32.1 by not including the MSVC toolchain when CMAKE_TOOLCHAIN_FILE is already specified.

[bowie7070]

Would it solve the problem with dumpbin not being found if we set CMAKE_OBJDUMP to the full path to dumpbin and get vcpkg changed to also try the value of this variable alongside searching for dumpbin?

https://cmake.org/cmake/help/latest/command/file.html#variable:CMAKE_GET_RUNTIME_DEPENDENCIES_COMMAND

[bowie7070]

Here is an discussion on the vpkg github about specifying the location of dumpbin.

https://github.com/microsoft/vcpkg/discussions/17928