Multiple compilers will have ABI issues in manifest mode?

iXavierLiu commented 8 months ago

Environment

OS: Apple M3 Pro(macOS Sonoma 14.3.1)
Compiler: clang(Apple clang version 15.0.0), g++(g++-13 (Homebrew GCC 13.2.0) 13.2.0)

To Reproduce

Steps to reproduce the behavior:

Consider the following two scenarios:

INSTALL-CLANG with default compiler(clang), ABI is28823f79dc4bbccc7130554e4be16b5982020f9ac825f9cd3c07b94c6af54233

❯ ./vcpkg remove gtest
The following packages will be removed:
gtest:arm64-osx
Removing 1/1 gtest:arm64-osx
❯ ./vcpkg install gtest
Computing installation plan...
The following packages will be built and installed:
gtest:arm64-osx@1.14.0
Detecting compiler hash for triplet arm64-osx...
Restored 1 package(s) from /Users/xavier/.cache/vcpkg/archives in 45.3 ms. Use --debug to see more details.
Installing 1/1 gtest:arm64-osx@1.14.0...
Elapsed time to handle gtest:arm64-osx: 39.1 ms
gtest:arm64-osx package ABI: 28823f79dc4bbccc7130554e4be16b5982020f9ac825f9cd3c07b94c6af54233
Total install time: 39.2 ms
The package gtest is compatible with built-in CMake targets:

enable_testing()

find_package(GTest CONFIG REQUIRED)
target_link_libraries(main PRIVATE GTest::gtest GTest::gtest_main GTest::gmock GTest::gmock_main)

add_test(AllTestsInMain main)

INSTALL-GCC with gcc compiler, ABI is95dd47b37b611a080722c3c12cdce2392cd0bf0956fdc33017e4960f904c56f7 Pay attention to the first two lines.

❯ export CC=`where gcc-13`
❯ export CXX=`where g++-13`
❯ ./vcpkg remove gtest
The following packages will be removed:
gtest:arm64-osx
Removing 1/1 gtest:arm64-osx
❯ ./vcpkg install gtest
Computing installation plan...
The following packages will be built and installed:
gtest:arm64-osx@1.14.0
Detecting compiler hash for triplet arm64-osx...
Restored 1 package(s) from /Users/xavier/.cache/vcpkg/archives in 49.9 ms. Use --debug to see more details.
Installing 1/1 gtest:arm64-osx@1.14.0...
Elapsed time to handle gtest:arm64-osx: 39.1 ms
gtest:arm64-osx package ABI: 95dd47b37b611a080722c3c12cdce2392cd0bf0956fdc33017e4960f904c56f7
Total install time: 39.2 ms
The package gtest is compatible with built-in CMake targets:

enable_testing()

find_package(GTest CONFIG REQUIRED)
target_link_libraries(main PRIVATE GTest::gtest GTest::gtest_main GTest::gmock GTest::gmock_main)

add_test(AllTestsInMain main)

The ABIs of INSTALL-CLANG and INSTALL-GCC are obviously different because they are built using different compilers.

Then consider the following situation:

CMakeUserPresets.json has two presets, one named osx-gcc and the other named osx-clang. They only have different compilers.

{
"version": 2,
"configurePresets": [
{
  "name": "osx",
  "hidden": true,
  "binaryDir": "${sourceDir}/out/build/${presetName}",
  "generator": "Unix Makefiles",
  "cacheVariables": {
    "CMAKE_INSTALL_PREFIX": "${sourceDir}/out/install/${presetName}",
    "CMAKE_BUILD_TYPE": "Debug",
    "CMAKE_TOOLCHAIN_FILE": "/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake"
  }
},
{
  "name": "osx-gcc",
  "inherits": [
    "osx"
  ],
  "cacheVariables": {
    "CMAKE_C_COMPILER": "/opt/homebrew/bin/gcc-13",
    "CMAKE_CXX_COMPILER": "/opt/homebrew/bin/g++-13"
  }
},
{
  "name": "osx-clang",
  "inherits": [
    "osx"
  ],
  "cacheVariables": {
    "CMAKE_C_COMPILER": "/usr/bin/clang",
    "CMAKE_CXX_COMPILER": "/usr/bin/clang++"
  }
}
]
}

In classic mode, if osx-gcc preset uses the installation of INSTALL-GCC and osx-clang preset uses the installation of INSTALL-CLANG, there will be no ABI problems.

If osx-gcc preset uses installation of INSTALL-clang, it will cause ABI problems when building, and vice versa.

They worked well as expected.

main body

In manifest mode, vcpkg.json is very simple

{
  "dependencies": [
    "gtest"
  ]
}

Configuration using osx-clang preset

❯ rm -rf out
❯ cmake --preset=osx-clang
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  CMAKE_CXX_COMPILER="/usr/bin/clang++"
  CMAKE_C_COMPILER="/usr/bin/clang"
  CMAKE_INSTALL_PREFIX="/Users/xavier/workspace/test/out/install/osx-clang"
  CMAKE_TOOLCHAIN_FILE="/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake"

-- Running vcpkg install
...
Elapsed time to handle gtest:arm64-osx: 12.4 ms
gtest:arm64-osx package ABI: 97e2a6c6019c5922bc42ffc5f6b48bd397532cd195e4a9987452e6a8a8a1f0e2
Total install time: 14.9 ms
...

Configuration using osx-clang preset

❯ rm -rf out
❯ cmake --preset=osx-gcc
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  CMAKE_CXX_COMPILER="/opt/homebrew/bin/g++-13"
  CMAKE_C_COMPILER="/opt/homebrew/bin/gcc-13"
  CMAKE_INSTALL_PREFIX="/Users/xavier/workspace/test/out/install/osx-gcc"
  CMAKE_TOOLCHAIN_FILE="/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake"

-- Running vcpkg install
...
Elapsed time to handle gtest:arm64-osx: 11.8 ms
gtest:arm64-osx package ABI: 97e2a6c6019c5922bc42ffc5f6b48bd397532cd195e4a9987452e6a8a8a1f0e2
Total install time: 14.2 ms
...

As you can see, their ABIs are the same, although the specified compiler is different, where ABI of osx-clang is correct. errors will be reported when building osx-gcc.

It's not clear to me how vcpkg specifies a specific compiler in manifest mode, I tried specifying via CMAKE_CXX_COMPILER but it doesn't seem to work.

I tried looking up the problem through the documentation and google, but found nothing. Did I miss something?

autoantwort commented 8 months ago

If you use the CMake integration and set CMAKE_CXX_COMPILER, you only set the CMAKE_CXX_COMPILER of your project, not the c++ compiler used to compile vcpkg packages. You have to set the CC or CXX env vars or use different triplets with custom toolchains.

iXavierLiu commented 8 months ago

If you use the CMake integration and set CMAKE_CXX_COMPILER, you only set the CMAKE_CXX_COMPILER of your project, not the c++ compiler used to compile vcpkg packages. You have to set the CC or CXX env vars or use different triplets with custom toolchains.

In fact, I have tested it before, but in order to prevent the issue from being too lengthy, I did not put it up.

After I set the CC and CXX variables

❯ rm -rf out
❯ export CC=`where gcc-13`
❯ export CXX=`where g++-13`
❯ cmake --preset=osx-gcc
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  CMAKE_CXX_COMPILER="/opt/homebrew/bin/g++-13"
  CMAKE_C_COMPILER="/opt/homebrew/bin/gcc-13"
  CMAKE_INSTALL_PREFIX="/Users/xavier/workspace/test/out/install/osx-gcc"
  CMAKE_TOOLCHAIN_FILE="/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake"

-- Running vcpkg install
Fetching registry information from https://github.com/microsoft/vcpkg (HEAD)...
Detecting compiler hash for triplet arm64-osx...
error: while detecting compiler information:
The log file content at "/Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/stdout-arm64-osx.log" is:
-- Configuring arm64-osx-rel
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:112 (message):
    Command failed: /opt/homebrew/Cellar/cmake/3.28.3/bin/cmake /Users/xavier/workspace/vcpkg/scripts/detect_compiler -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/Users/xavier/workspace/vcpkg/packages/detect_compiler_arm64-osx -DCMAKE_MAKE_PROGRAM=/Users/xavier/workspace/vcpkg/downloads/tools/ninja/1.10.2-osx/ninja -DCMAKE_SYSTEM_NAME=Darwin -DBUILD_SHARED_LIBS=OFF -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=/Users/xavier/workspace/vcpkg/scripts/toolchains/osx.cmake -DVCPKG_TARGET_TRIPLET=arm64-osx -DVCPKG_SET_CHARSET_FLAG=ON -DVCPKG_PLATFORM_TOOLSET=external -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON -DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE -DCMAKE_VERBOSE_MAKEFILE=ON -DVCPKG_APPLOCAL_DEPS=OFF -DCMAKE_TOOLCHAIN_FILE=/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON -DVCPKG_CXX_FLAGS= -DVCPKG_CXX_FLAGS_RELEASE= -DVCPKG_CXX_FLAGS_DEBUG= -DVCPKG_C_FLAGS= -DVCPKG_C_FLAGS_RELEASE= -DVCPKG_C_FLAGS_DEBUG= -DVCPKG_CRT_LINKAGE=dynamic -DVCPKG_LINKER_FLAGS= -DVCPKG_LINKER_FLAGS_RELEASE= -DVCPKG_LINKER_FLAGS_DEBUG= -DVCPKG_TARGET_ARCHITECTURE=arm64 -DCMAKE_INSTALL_LIBDIR:STRING=lib -DCMAKE_INSTALL_BINDIR:STRING=bin -D_VCPKG_ROOT_DIR=/Users/xavier/workspace/vcpkg -DZ_VCPKG_ROOT_DIR=/Users/xavier/workspace/vcpkg -D_VCPKG_INSTALLED_DIR=/Users/xavier/workspace/test/out/build/osx-gcc/vcpkg_installed -DVCPKG_MANIFEST_INSTALL=OFF -DCMAKE_OSX_ARCHITECTURES=arm64
    Working Directory: /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/arm64-osx-rel
    Error code: 1
    See logs for more information:
      /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/config-arm64-osx-rel-CMakeCache.txt.log
      /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/config-arm64-osx-rel-out.log
      /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/config-arm64-osx-rel-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_cmake.cmake:344 (vcpkg_execute_required_process)
  scripts/detect_compiler/portfile.cmake:18 (vcpkg_configure_cmake)
  scripts/ports.cmake:170 (include)

error: vcpkg was unable to detect the active compiler's information. See above for the CMake failure output.
-- Running vcpkg install - failed
CMake Error at /Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake:899 (message):
  vcpkg install failed.  See logs for more information:
  /Users/xavier/workspace/test/out/build/osx-gcc/vcpkg-manifest-install.log
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/CMakeDetermineSystem.cmake:170 (include)
  CMakeLists.txt:3 (project)

CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
-- Configuring incomplete, errors occurred!

It seems that the make program is missing, so I added "CMAKE_MAKE_PROGRAM": "/usr/bin/make" to osx-gcc, then

❯ rm -rf out
❯ cmake --preset=osx-gcc
Preset CMake variables:

  CMAKE_BUILD_TYPE="Debug"
  CMAKE_CXX_COMPILER="/opt/homebrew/bin/g++-13"
  CMAKE_C_COMPILER="/opt/homebrew/bin/gcc-13"
  CMAKE_INSTALL_PREFIX="/Users/xavier/workspace/test/out/install/osx-gcc"
  CMAKE_MAKE_PROGRAM="/usr/bin/make"
  CMAKE_TOOLCHAIN_FILE="/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake"

-- Running vcpkg install
Fetching registry information from https://github.com/microsoft/vcpkg (HEAD)...
Detecting compiler hash for triplet arm64-osx...
error: while detecting compiler information:
The log file content at "/Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/stdout-arm64-osx.log" is:
-- Configuring arm64-osx-rel
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:112 (message):
    Command failed: /opt/homebrew/Cellar/cmake/3.28.3/bin/cmake /Users/xavier/workspace/vcpkg/scripts/detect_compiler -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/Users/xavier/workspace/vcpkg/packages/detect_compiler_arm64-osx -DCMAKE_MAKE_PROGRAM=/Users/xavier/workspace/vcpkg/downloads/tools/ninja/1.10.2-osx/ninja -DCMAKE_SYSTEM_NAME=Darwin -DBUILD_SHARED_LIBS=OFF -DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=/Users/xavier/workspace/vcpkg/scripts/toolchains/osx.cmake -DVCPKG_TARGET_TRIPLET=arm64-osx -DVCPKG_SET_CHARSET_FLAG=ON -DVCPKG_PLATFORM_TOOLSET=external -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON -DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON -DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE -DCMAKE_VERBOSE_MAKEFILE=ON -DVCPKG_APPLOCAL_DEPS=OFF -DCMAKE_TOOLCHAIN_FILE=/Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake -DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON -DVCPKG_CXX_FLAGS= -DVCPKG_CXX_FLAGS_RELEASE= -DVCPKG_CXX_FLAGS_DEBUG= -DVCPKG_C_FLAGS= -DVCPKG_C_FLAGS_RELEASE= -DVCPKG_C_FLAGS_DEBUG= -DVCPKG_CRT_LINKAGE=dynamic -DVCPKG_LINKER_FLAGS= -DVCPKG_LINKER_FLAGS_RELEASE= -DVCPKG_LINKER_FLAGS_DEBUG= -DVCPKG_TARGET_ARCHITECTURE=arm64 -DCMAKE_INSTALL_LIBDIR:STRING=lib -DCMAKE_INSTALL_BINDIR:STRING=bin -D_VCPKG_ROOT_DIR=/Users/xavier/workspace/vcpkg -DZ_VCPKG_ROOT_DIR=/Users/xavier/workspace/vcpkg -D_VCPKG_INSTALLED_DIR=/Users/xavier/workspace/test/out/build/osx-gcc/vcpkg_installed -DVCPKG_MANIFEST_INSTALL=OFF -DCMAKE_OSX_ARCHITECTURES=arm64
    Working Directory: /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/arm64-osx-rel
    Error code: 1
    See logs for more information:
      /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/config-arm64-osx-rel-CMakeCache.txt.log
      /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/config-arm64-osx-rel-out.log
      /Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/config-arm64-osx-rel-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_cmake.cmake:344 (vcpkg_execute_required_process)
  scripts/detect_compiler/portfile.cmake:18 (vcpkg_configure_cmake)
  scripts/ports.cmake:170 (include)

error: vcpkg was unable to detect the active compiler's information. See above for the CMake failure output.
-- Running vcpkg install - failed
CMake Error at /Users/xavier/workspace/vcpkg/scripts/buildsystems/vcpkg.cmake:899 (message):
  vcpkg install failed.  See logs for more information:
  /Users/xavier/workspace/test/out/build/osx-gcc/vcpkg-manifest-install.log
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/CMakeDetermineSystem.cmake:170 (include)
  CMakeLists.txt:3 (project)

-- Configuring incomplete, errors occurred!

config-arm64-osx-rel-err.log:

CMake Error:
  Running

   '/Users/xavier/workspace/vcpkg/downloads/tools/ninja/1.10.2-osx/ninja' '-C' '/Users/xavier/workspace/vcpkg/buildtrees/detect_compiler/arm64-osx-rel/CMakeFiles/CMakeScratch/TryCompile-j8qDDF' '-t' 'recompact'

  failed with:

   ninja: error: CMakeFiles/rules.ninja:21: lexing error

CMake Error at /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/CMakeDetermineCompilerABI.cmake:57 (try_compile):
  Failed to generate test project build system.
Call Stack (most recent call first):
  /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/CMakeTestCCompiler.cmake:26 (CMAKE_DETERMINE_COMPILER_ABI)
  CMakeLists.txt:11 (enable_language)

CMake Warning:
  Value of CMAKE_C_COMPILER_ARG1 contained a newline; truncating

This seems to be an internal error of vcpkg, so I don't think selecting the compiler in manifest mode via the CC and CXX variables is the correct approach.

Neumann-A commented 8 months ago

No. Set CC and CXX in the preset and don't use where.

github-actions[bot] commented 7 months ago

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 28 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

iXavierLiu commented 3 months ago

No. Set CC and CXX in the preset and don't use where.

This does fix the problem, but it is still confusing to set variables this way, because one would think vcpkg would read the default values for CC and CXX from bash, but it obviously does not. Anyway, thanks for your answer.

dg0yt commented 3 months ago

CC and CXX are environment variables. They influence both CMake projects, incl. vcpkg ports. CMAKE_C_COMPILER and CMAKE_CXX_COMPILER are CMake variables. They influence CMake projects. They are not passed through from the top-level project to vcpkg ports.

In fact, the environment variables may interfere with crossbuilds in undesired ways. where ... is just a distractor here. It is meant to be expanded by the shell before being passed into CMake. It won't work if it is passed literally into CMake.

microsoft / vcpkg