search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.02k stars 956 forks source link

[feature] Add a cmake variable through a profile #16021

Open Todiq opened 3 months ago

Todiq commented 3 months ago

What is your suggestion?

Hello. Thank you for taking the time to read this.

https://github.com/conan-io/conan/issues/3099 could be very handy for my use case, although workarounds exist. Please note that I prefer using conan build rather than conan install + cmake ..., making it easier to remember and apply on all operating systems.

In order to cross-compile from Linux to Windows using clang https://github.com/conan-io/conan/issues/15655, I need to properly set the environment for cmake.

A prerequisite is to set CMAKE_MT to llvm-mt.

This must be done before the build() section of the recipe. In fact, cmake first tries to check whether the compiler and linker work before compiling the project. Thus, one cannot simply write set(CMAKE_MT llvm-mt) in the root CMakeLists.txt and expect it to work (unless I missed something). It has to be done in the recipe, in the generate(self) method.

I am using python:3.9.18-slim as a base docker image (debian-based). I installed all my toolchain with these packets: clang-16 clang-tools-16 llvm-16 lld-16. It leads to a way slimer image than installing it through LLVM's shell script.

Unfortunately, it implies that all the tools are called name_of_the_tool-16 (I am aware this could be fixed through update-alternatives).

Until now, I wrote this line tc.variables["CMAKE_MT"] = "llvm-mt-16" in my generate(self) method, which worked fine to inject the var to cmake before the start of the build.

However, since I am currently transitioning compilers on Windows as well, this line breaks when trying to use msvc (clang works):

Log ```powershell Using lockfile: 'D:\core\conan.lock' ======== Input profiles ======== Profile host: [settings] arch=x86_64 build_type=Release compiler=msvc compiler.cppstd=14 compiler.runtime=dynamic compiler.runtime_type=Release compiler.version=193 os=Windows cast*/*:build_type=Release cast*/*:compiler.cppstd=17 [conf] cast*/*:tools.build:skip_test=False cast*/*:tools.cmake.cmaketoolchain:generator=Ninja Multi-Config cast*/*:tools.env.virtualenv:powershell=True cast*/*:user.cast:run_tests=False Profile build: [settings] arch=x86_64 build_type=Release compiler=msvc compiler.cppstd=14 compiler.runtime=dynamic compiler.runtime_type=Release compiler.version=193 os=Windows cast*/*:build_type=Release cast*/*:compiler.cppstd=17 [conf] cast*/*:tools.build:skip_test=False cast*/*:tools.cmake.cmaketoolchain:generator=Ninja Multi-Config cast*/*:tools.env.virtualenv:powershell=True cast*/*:user.cast:run_tests=False ======== Computing dependency graph ======== Graph root conanfile.py (castcore/1.1): D:\core\conanfile.py Requirements boost/1.81.0#753df9d4f149a5551335f47abe21de0b - Cache icu/72.1#d24b2d30acae3dcb788e139112bcd7c6 - Cache libiconv/1.17#73fefc1b696e069df90fd1d18aa63edd - Cache libxml2/2.11.6#41c14895baba105865cb22ecaf948115 - Cache libxslt/1.1.34#2bdc1f5fe5db1df40f8465e485a19ee4 - Cache pugixml/1.13#f615c1fcec55122b2e177d17061276e7 - Cache xerces-c/3.2.4#c52d1d1bb36d1377471ec1e551a6598a - Cache Test requirements gtest/1.13.0#8a0bc5b3e159ed45de97260c2bff65b5 - Cache Build requirements b2/4.10.1#a41eecb267963ddf518e3c177eae60e1 - Cache cmake/3.29.0#a055de871f63a7904aa3dcb9f3c61242 - Cache msys2/cci.latest#5a31efa2bde593541fd5ac3bcc50c01c - Cache ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21 - Cache ======== Computing necessary packages ======== Requirements boost/1.81.0#753df9d4f149a5551335f47abe21de0b:e03785e977988d57a84487e4306a26c9018cabf0#f65d418cddf6cc771a803fde82e12dd9 - Cache icu/72.1#d24b2d30acae3dcb788e139112bcd7c6:9f8165f8d74e26bf104abb2a5005a80d1101b198#3cb5da99fe9d01c55d1c5dbf13f35112 - Cache libiconv/1.17#73fefc1b696e069df90fd1d18aa63edd:7bfde258ff4f62f75668d0896dbddedaa7480a0f#9ef92719f5c05dca2f0dbb46f50d3f8d - Cache libxml2/2.11.6#41c14895baba105865cb22ecaf948115:a6cdd368afb04c4d9e5918696d661f8f5249a029#d4a7bc8cb16ca7e7411db8804a765032 - Cache libxslt/1.1.34#2bdc1f5fe5db1df40f8465e485a19ee4:4e7168dde525f5dbaf65a8b8f6b3280b6a7e87d7#a632a2a87954026a2d962070cbad7aa6 - Cache pugixml/1.13#f615c1fcec55122b2e177d17061276e7:da39a3ee5e6b4b0d3255bfef95601890afd80709#ae530054ed7ac99330514c13a5bfde3a - Cache xerces-c/3.2.4#c52d1d1bb36d1377471ec1e551a6598a:41c4e6767851fe7db56310649912b6259451a38f#0831576b3c913f5e3d1294aa2f05ee41 - Cache Test requirements gtest/1.13.0#8a0bc5b3e159ed45de97260c2bff65b5:9940e217936a0f218f90473d62bf1d4db55efaa7#f44bf478a18701d76b310497997d7b3b - Cache Build requirements cmake/3.29.0#a055de871f63a7904aa3dcb9f3c61242:522dcea5982a3f8a5b624c16477e47195da2f84f#27973db6c7b443ab1b970571b73d0818 - Cache ninja/1.11.1#77587f8c8318662ac8e5a7867eb4be21:723257509aee8a72faf021920c2874abc738e029#74ebd8f35f54015dcbf2b18a6adbd023 - Cache Skipped binaries b2/4.10.1, msys2/cci.latest ======== Installing packages ======== ======== Installing packages ======== cmake/3.29.0: Already installed! (1 of 10) cmake/3.29.0: Appending PATH environment variable: D:\conan\p\cmake8c0a252b8ebf8\p\bin gtest/1.13.0: Already installed! (2 of 10) ninja/1.11.1: Already installed! (3 of 10) pugixml/1.13: Already installed! (4 of 10) xerces-c/3.2.4: Already installed! (5 of 10) boost/1.81.0: Already installed! (6 of 10) boost/1.81.0: Disabled magic autolinking (smart and magic decisions) icu/72.1: Already installed! (7 of 10) libiconv/1.17: Already installed! (8 of 10) libxml2/2.11.6: Already installed! (9 of 10) libxml2/2.11.6: Appending PATH environment variable: D:\conan\p\libxm07501da66e843\p\bin libxslt/1.1.34: Already installed! (10 of 10) WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: WARN: deprecated: 'env_info' used in: libxslt/1.1.34, libxml2/2.11.6, libiconv/1.17, boost/1.81.0, icu/72.1, cmake/3.29.0 WARN: deprecated: 'cpp_info.names' used in: gtest/1.13.0, libxml2/2.11.6, libiconv/1.17, boost/1.81.0, icu/72.1, libxslt/1.1.34 WARN: deprecated: 'cpp_info.filenames' used in: libxml2/2.11.6, boost/1.81.0 WARN: deprecated: 'user_info' used in: boost/1.81.0 WARN: deprecated: 'cpp_info.build_modules' used in: libxml2/2.11.6 ======== Finalizing install (deploy, generators) ======== conanfile.py (castcore/1.1): Calling generate() conanfile.py (castcore/1.1): Generators folder: D:\core\build\windows-msvc-193-x86_64\generators conanfile.py (castcore/1.1): CMakeToolchain generated: conan_toolchain.cmake conanfile.py (castcore/1.1): Preset 'conan-windows-msvc-193-x86_64' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-windows-msvc-193-x86_64' if using CMake>=3.23 conanfile.py (castcore/1.1): If your CMake version is not compatible with CMakePresets (<3.23) call cmake like: 'cmake -G "Ninja Multi-Config" -DCMAKE_TOOLCHAIN_FILE=D:\core\build\windows-msvc-193-x86_64\generators\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW' conanfile.py (castcore/1.1): CMakeToolchain generated: CMakePresets.json conanfile.py (castcore/1.1): CMakeToolchain generated: ..\..\..\CMakeUserPresets.json conanfile.py (castcore/1.1): CMakeDeps necessary find_package() and targets for your CMakeLists.txt find_package(Boost) find_package(ICU) find_package(LibXslt) find_package(libxml2) find_package(pugixml) find_package(XercesC) find_package(GTest) target_link_libraries(... boost::boost icu::icu libxslt::libxslt LibXml2::LibXml2 pugixml::pugixml XercesC::XercesC gtest::gtest) conanfile.py (castcore/1.1): Generating aggregated env files conanfile.py (castcore/1.1): Generated aggregated env files: ['conanbuild.bat', 'conanbuild.ps1', 'conanrun.ps1'] ======== Calling build() ======== conanfile.py (castcore/1.1): Calling build() conanfile.py (castcore/1.1): Running CMake.configure() conanfile.py (castcore/1.1): RUN: cmake -G "Ninja Multi-Config" -DCMAKE_TOOLCHAIN_FILE="D:/core/build/windows-msvc-193-x86_64/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="D:/core" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" "D:\core" conanvcvars.bat: Activating environment Visual Studio 17 - amd64 - winsdk_version=None - vcvars_ver=14.3 [vcvarsall.bat] Environment initialized for: 'x64' -- Using Conan toolchain: D:/core/build/windows-msvc-193-x86_64/generators/conan_toolchain.cmake -- Conan toolchain: C++ Standard 17 with extensions OFF -- Conan toolchain: Setting BUILD_SHARED_LIBS = ON -- The CXX compiler identification is MSVC 19.39.33523.0 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - failed -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - broken CMake Error at D:/conan/p/cmake8c0a252b8ebf8/p/share/cmake-3.29/Modules/CMakeTestCXXCompiler.cmake:60 (message): The C++ compiler "C:/Program Files/Microsoft Visual Studio/2022/Professional/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe" is not able to compile a simple test program. It fails with the following output: Change Dir: 'D:/core/build/windows-msvc-193-x86_64/CMakeFiles/CMakeScratch/TryCompile-g45zpv' Run Build Command(s): D:/conan/p/ninjae2ad385cd85df/p/bin/ninja.exe -v cmTC_ee6f6 [1/2] C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\cl.exe /nologo /TP -DCMAKE_INTDIR=\"Debug\" /DWIN32 /D_WINDOWS /EHsc /Zi /Ob0 /Od /RTC1 -std:c++17 /showIncludes /FoCMakeFiles\cmTC_ee6f6.dir\Debug\testCXXCompiler.cxx.obj /FdCMakeFiles\cmTC_ee6f6.dir\Debug\ /FS -c D:\core\build\windows-msvc-193-x86_64\CMakeFiles\CMakeScratch\TryCompile-g45zpv\testCXXCompiler.cxx [2/2] C:\Windows\system32\cmd.exe /C "cd . && D:\conan\p\cmake8c0a252b8ebf8\p\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_ee6f6.dir\Debug --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=llvm-mt-16 --manifests -- C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo CMakeFiles\cmTC_ee6f6.dir\Debug\testCXXCompiler.cxx.obj /out:Debug\cmTC_ee6f6.exe /implib:Debug\cmTC_ee6f6.lib /pdb:Debug\cmTC_ee6f6.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ." FAILED: Debug/cmTC_ee6f6.exe C:\Windows\system32\cmd.exe /C "cd . && D:\conan\p\cmake8c0a252b8ebf8\p\bin\cmake.exe -E vs_link_exe --intdir=CMakeFiles\cmTC_ee6f6.dir\Debug --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=llvm-mt-16 --manifests -- C:\PROGRA~1\MICROS~2\2022\PROFES~1\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\link.exe /nologo CMakeFiles\cmTC_ee6f6.dir\Debug\testCXXCompiler.cxx.obj /out:Debug\cmTC_ee6f6.exe /implib:Debug\cmTC_ee6f6.lib /pdb:Debug\cmTC_ee6f6.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:console kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ." MT: command "llvm-mt-16 /nologo /manifest CMakeFiles\cmTC_ee6f6.dir\Debug/intermediate.manifest /out:CMakeFiles\cmTC_ee6f6.dir\Debug/embed.manifest /notify_update" failed (exit code 0x0) with the following output: no such file or directory ninja: build stopped: subcommand failed. CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:3 (project) -- Configuring incomplete, errors occurred! ERROR: conanfile.py (castcore/1.1): Error in build() method, line 99 cmake.configure() ConanException: Error 1 while executing ```

While I could have a if condition in my conanfile.py, I would find it way simpler to inject the variable in the host profile (targeting Windows) on my linux build machine, having no traces in the recipe. What do you think?

Have you read the CONTRIBUTING guide?

memsharded commented 3 months ago

Hi @Todiq

I think you would be interesting in injecting CMake toolchains from profiles, this is your use case isn't it?

Please check this example: https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/inject_cmake_variables.html, it can be used to define any CMake variable from profiles.

Todiq commented 3 months ago

@memsharded,

From what I read, one can only load a .cmake file containing the variables. Isn’t there any way to directly write them in the profile, avoiding to manage another file?

memsharded commented 3 months ago

From what I read, one can only load a .cmake file containing the variables. Isn’t there any way to directly write them in the profile, avoiding to manage another file?

No, I am afraid there isn't. The reason is that often it is not enough to define a variable. Many times, it is necessary, specially for toolchain files to make those variables CACHE variables. But not always. Inventing a syntax for supporting the different scenarios would be very artificial. Not to say more advanced cases where conditional logic based on CMake inputs is necessary, that is impossible from the profile. So passing the cmake file covers all cases.

Plus something equivalent is also done for other build systems like Meson, also passing files.

There are utilities in the profiles to be able to reference the current profile_dir folder, etc. to make easier and relocatable using the extra .cmake files, and at the end of the day, the files are managed with conan config install automatically, so not a big issue to have a .cmake more.

Todiq commented 3 months ago

@memsharded,

I just followed the steps from the documentation link you provided, but I am now getting an error that I had not encountered until now:

CMake Error at /opt/conan/p/cmakee300763ec076c/p/share/cmake-3.29/Modules/CMakeTestCXXCompiler.cmake:60 (message):
  The C++ compiler

    "/usr/bin/clang-cl-16"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/workspace/core/build/windows-msvc-193-x86_64/Release/CMakeFiles/CMakeScratch/TryCompile-Sq5pQn'

    Run Build Command(s): /opt/conan/p/ninja6fed3c8400c5b/p/bin/ninja -v cmTC_757c9
    [1/2] /usr/bin/clang-cl-16   -fcolor-diagnostics -fuse-ld=lld -target x86_64-pc-windows-msvc /winsdkdir /opt/win-sdk/sdk /vctoolsdir /opt/win-sdk/crt -flto -march=x86-64-v3  -std:c++17 -o CMakeFiles/cmTC_757c9.dir/testCXXCompiler.cxx.o -c /workspace/core/build/windows-msvc-193-x86_64/Release/CMakeFiles/CMakeScratch/TryCompile-Sq5pQn/testCXXCompiler.cxx
    FAILED: CMakeFiles/cmTC_757c9.dir/testCXXCompiler.cxx.o
    /usr/bin/clang-cl-16   -fcolor-diagnostics -fuse-ld=lld -target x86_64-pc-windows-msvc /winsdkdir /opt/win-sdk/sdk /vctoolsdir /opt/win-sdk/crt -flto -march=x86-64-v3  -std:c++17 -o CMakeFiles/cmTC_757c9.dir/testCXXCompiler.cxx.o -c /workspace/core/build/windows-msvc-193-x86_64/Release/CMakeFiles/CMakeScratch/TryCompile-Sq5pQn/testCXXCompiler.cxx
    clang: error: no input files
    ninja: build stopped: subcommand failed.

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)

-- Configuring incomplete, errors occurred!

ERROR: conanfile.py (castcore/1.1): Error in build() method, line 98
        cmake.configure()
        ConanException: Error 1 while executing

myvars.cmake is simply the following:

set(CMAKE_MT "llvm-mt-16")

and my host profile:

[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.version=193
os=Windows

cast*/*:build_type=Release
cast*/*:compiler.cppstd=17

[conf]
tools.cmake.cmaketoolchain:generator=Ninja
tools.build:cxxflags=["-fcolor-diagnostics"]
tools.build:cxxflags+=["-fuse-ld=lld"]
tools.build:cxxflags+=["-target x86_64-pc-windows-msvc"]
tools.build:cxxflags+=["/winsdkdir /opt/win-sdk/sdk /vctoolsdir /opt/win-sdk/crt"]
tools.build:cxxflags+=["-flto -march=x86-64-v3"]
tools.build:compiler_executables={"c": "clang-cl-16", "cpp": "clang-cl-16", "rc": "llvm-rc-16"}
tools.cmake.cmaketoolchain:user_toolchain+={{profile_dir}}/myvars.cmake

[buildenv]
LDFLAGS=/winsdkdir:/opt/win-sdk/sdk /vctoolsdir:/opt/win-sdk/crt

Everything worked when defining CMAKE_MT in the recipe

memsharded commented 3 months ago

This sounds weird. Can you please try a couple of things:

Todiq commented 3 months ago

The message in myvars.cmake confirms it is called.

Adding a message in the CMakeLists.txt does not do anything, since the error comes before the actual build part. It happens during the compiler checks.

memsharded commented 3 months ago

Adding a message in the CMakeLists.txt does not do anything, since the error comes before the actual build part. It happens during the compiler checks.

Maybe add the message() before the project() call? I am not sure why there would be any difference in having it defined in the recipe and in the profile, both are defining in the same toolchain approach.

Todiq commented 3 months ago

Adding a message() before the project() call indeed works. As well as writing set(CMAKE_MT "llvm-mt-16") before project() as well. It only breaks when loading myvars.cmake file through the profile

memsharded commented 3 months ago

It might be something specific about this variable? I cannot find it in the CMake docs: https://cmake.org/cmake/help/latest/search.html?q=CMAKE_MT shows no result? Maybe there is some other logic around that variable that it is not standard?

It would be good to have a reproducible case that we can try on our side.

sykhro commented 3 months ago

Do you actually need MT? We often get away with /MANIFEST:NO. llvm-mt is nowhere near MT.exe anyway

memsharded commented 2 months ago

Hi @Todiq

any update here? The variable CMAKE_MT doesn't seem a CMake variable, I can't find anything regarding it

Todiq commented 2 months ago

Thanks for the ping @memsharded,

Indeed, CMAKE_MT is not mentionned anywhere, even though it exists and is usable for cross-build scenarios.

However, I could clearly face the issue that the variable was not loaded through myvars.cmake. I will try to create a minimum reproducible case. On my own project, passing /MANIFEST:NO was enough.

memsharded commented 2 months ago

Thanks for the feedback. Cleaning the staled tag, and looking forward your minimum reproducible case, thanks!

danielnilsson9 commented 1 month ago

When adding a user_toolchain from config the CMakeToolchain generator no longer sets CMake variables required for cross compiling.

See: https://docs.conan.io/2/reference/tools/cmake/cmaketoolchain.html#cross-building

Related code: https://github.com/conan-io/conan/blob/develop2/conan/tools/cmake/toolchain/blocks.py#L1033

This might be related to your issue.

Edit: Support for defining extra CMake variables though config was recently introduced in version 2.4.0: https://github.com/conan-io/conan/pull/16242