search

conan-io / cmake-conan

CMake wrapper for conan C and C++ package manager
MIT License
820 stars 250 forks source link

[suggest][dev2] Run conan install in a subprocess when injection. (to solve CAE) #488

Open hwhsu1231 opened 1 year ago

hwhsu1231 commented 1 year ago

Suggestion

There still exists the CAE problem of CMakeToolchain in the Dependency Provider, which causes some limitations:

However, I wondered why not adopt the techinuqe I mentioned in the cmake-conan-setup before:

Use a subproccess to run conan install command to generate conan_toolchain.cmake when injection.

The outline of the workflow may look like this:

  1. _When injection with CMAKE_PROJECT_TOP_LEVEL_INCLUDES:_

    1. Generate a temperary CMakeLists.txt through file(WRITE) command.
    2. _Call conan_install() inside the generated CMakeLists.txt._
    3. _Configure the generated CMakeLists.txt through execute_process() command._
    4. _Include conan_toolchain.cmake after conan install successes._

  2. _When find_package() invokes conan_provide_dependency():_

    1. _Append ${CONAN_OUTPUT_FOLDER} into CMAKE_PREFIX_PATH._
    2. _Append ${CONAN_OUTPUT_FOLDER} into CMAKE_MODULE_PATH._
    3. _Call find_package(${ARGN} BYPASS_PROVIDER)._

Demonstration

The following project is a simple prototype, in which I only considered the situation using MSVC compiler with Ninja generator:

  1. Extract the following archived file.

    cmake-conan-provider.zip

  2. Call vcvarsall.bat x64 to initialize the MSVC env:

    vcvarsall.bat x64

  3. Run the following cmake command to configure the porject:

    cmake --preset win32-msvc16-x64-ninja-debug

Logs

Click to expand logs ```cmd D:\Test\cmake-conan-provider>vcvarsall.bat x64 ********************************************************************** ** Visual Studio 2019 Developer Command Prompt v16.11.23 ** Copyright (c) 2021 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' D:\Test\cmake-conan-provider>cmake --preset win32-msvc16-x64-ninja-debug Preset CMake variables: CMAKE_BUILD_TYPE="Debug" CMAKE_CXX_COMPILER="cl.exe" CMAKE_C_COMPILER="cl.exe" CMAKE_PROJECT_TOP_LEVEL_INCLUDES="D:/Test/cmake-conan-provider/conan_provider.cmake" ======== Input profiles ======== Profile host: [settings] arch=x86_64 build_type=Debug compiler=msvc compiler.cppstd=14 compiler.runtime=dynamic compiler.runtime_type=Debug compiler.version=192 os=Windows [conf] tools.cmake.cmaketoolchain:generator=Ninja Profile build: [settings] arch=x86_64 build_type=Release compiler=msvc compiler.cppstd=14 compiler.runtime=dynamic compiler.runtime_type=Release compiler.version=193 os=Windows ======== Computing dependency graph ======== Graph root conanfile.py: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan\conanfile.py Requirements fmt/8.1.1#e63eae266b23cf3d7f3e0dfe33d1f7e9 - Cache ======== Computing necessary packages ======== Requirements fmt/8.1.1#e63eae266b23cf3d7f3e0dfe33d1f7e9:a6b1a9b5962f3f69ca4f26a80d8d82e9ba1fabe7#f1762e8333d32db4225378e23e3b4070 - Cache ======== Installing packages ======== fmt/8.1.1: Already installed! (1 of 1) WARN: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: WARN: 'cpp_info.names' used in: fmt/8.1.1 ======== Finalizing install (deploy, generators) ======== conanfile.py: Writing generators to D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan conanfile.py: Generator 'CMakeToolchain' calling 'generate()' conanfile.py: CMakeToolchain generated: conan_toolchain.cmake conanfile.py: Preset 'conan-debug' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-debug' conanfile.py: If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake -G Ninja -DCMAKE_TOOLCHAIN_FILE=D:\Test\cmake-conan-provider\build\win32-msvc16-x64-ninja-debug\conan\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Debug' conanfile.py: CMakeToolchain generated: CMakePresets.json conanfile.py: Generator 'CMakeDeps' calling 'generate()' conanfile.py: Generating aggregated env files conanfile.py: Generated aggregated env files: ['conanbuild.bat', 'conanrun.bat'] Install finished succesfully -- Using Conan toolchain: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan/conan_toolchain.cmake -- Conan toolchain: C++ Standard 14 with extensions OFF -- The C compiler identification is MSVC 19.29.30147.0 -- The CXX compiler identification is MSVC 19.29.30147.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done ========== cmake-conan-provider ========== -- Conan: Component target declared 'fmt::fmt' -- Configuring done -- Generating done -- Build files have been written to: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug ```

As we can see in the above logs, the conan_toolchain.cmake file is included and fmt-config.cmake file is find_packaged successfully. Although I only considered using MSVC compiler with Ninja generator currently, it proves that this scheme is feasible at least. What do you think, Conan Team?

-- Using Conan toolchain: D:/Test/cmake-conan-provider/build/win32-msvc16-x64-ninja-debug/conan/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 14 with extensions OFF
...
...
...
========== cmake-conan-provider ==========
-- Conan: Component target declared 'fmt::fmt'
hwhsu1231 commented 1 year ago

@memsharded

This scheme should be able to solve the situation you mentioned in: https://github.com/conan-io/cmake-conan/issues/480#issuecomment-1473822193, right?

hwhsu1231 commented 1 year ago

The generated directory structure will look like this:

image

hwhsu1231 commented 1 year ago

And I noticed that it won't generate a CMakeUserPresets.json in the root source directory. Therefore, it might be able to solve https://github.com/conan-io/cmake-conan/issues/484 as well.

memsharded commented 1 year ago

The approach described cannot detect correctly CMake current compiler and configuration.

I) Generate a temperary CMakeLists.txt through file(WRITE) command. II) Call conan_install() inside the generated CMakeLists.txt.

When conan_install() is executed at that point, it cannot detect the current CMake configuration, because it has not been configured yet. One of the main reasons of existence of cmake-conan is being able to get the current CMake configuration and install dependencies for that configuration, and this is not possible if you execute conan install before the toolchain is processed. Thus, this is not a viable approach.

hwhsu1231 commented 1 year ago

The approach described cannot detect correctly CMake current compiler and configuration.

I) Generate a temperary CMakeLists.txt through file(WRITE) command. II) Call conan_install() inside the generated CMakeLists.txt.

When conan_install() is executed at that point, it cannot detect the current CMake configuration, because it has not been configured yet. One of the main reasons of existence of cmake-conan is being able to get the current CMake configuration and install dependencies for that configuration, and this is not possible if you execute conan install before the toolchain is processed. Thus, this is not a viable approach.

@memsharded - Are you sure? Well...

Recently, QtCreator updated its auto-setup.cmake on 10.0 branch:

As we can see, Qt Team uses nearly the same method as mine to integrate conan_support.cmake with its auto-setup.cmake. It seems that they treat conan_support.cmake as an upgrade of conan.cmake.

  1. Generate a temperary CMakeLists.txt by file(WRITE) command:

    https://github.com/qt-creator/qt-creator/blob/5633533e0d485de15aa9ab03bf8a1d6229364558/src/share/3rdparty/package-manager/auto-setup.cmake#L85-L117

  2. Configure the generated CMakeLists.txt by execute_process() command:

    https://github.com/qt-creator/qt-creator/blob/5633533e0d485de15aa9ab03bf8a1d6229364558/src/share/3rdparty/package-manager/auto-setup.cmake#L119-L127

Since my cmake-conan-setup project is inspired by the design of auto-setup.cmake of QtCreator, and I haven't met any fatal error so far, I'm curious about what you said. Are you saying that Qt Team's idea is simply a mistake?

memsharded commented 1 year ago

Their approach is to provide a profile: -pr \"${CMAKE_BINARY_DIR}/conan-dependencies/conan_host_profile, but this is not the main use case that users expect from cmake-conan, and that requires discipline and user alignment, which might be ok for qt beause they control the configuration, but not for the majority of Conan cmake-conan users.

cc/ @cristianadam

hwhsu1231 commented 1 year ago

But their conan_host_profile is generated by the detect_host_profile() function provided by conan_support.cmake?

https://github.com/qt-creator/qt-creator/blob/5633533e0d485de15aa9ab03bf8a1d6229364558/src/share/3rdparty/package-manager/auto-setup.cmake#L94

cristianadam commented 1 year ago

I did the same things as the conan_provide_dependency function https://github.com/conan-io/cmake-conan/blob/develop2/conan_support.cmake#L164-L173

@hwhsu1231 thank you for looking in to Qt Creator's source code. I do hope that it works for you, if not feel free to open bug reports at http://bugreports.qt.io/

memsharded commented 1 year ago

The key of running function(conan_provide_dependency package_name) that contains inside the detect_host_profile(${CMAKE_BINARY_DIR}/conan_host_profile) is that it runs when a find_package() is found, which happens much later than the project() has been called and also much later that the user CMake input configuration (command line arguments like -DCMAKE_CXX_COMPILER or user provided toolchains) has been processed.

Trying to detect the profile before that, as @hwhsu1231 suggests, doesn't work, and results in the wrong profile being detected and passed to conan install, so it is not a valid alternative.

hwhsu1231 commented 1 year ago

@memsharded

I still don't get it. Why will the wrong profile be detected? Can you provide an example?

In my opinion, if we can pass exactly the same CMake CLI arguments as the main-process is using to the sub-process, then it should be able to generate the correct profile in the sub-process. What do I miss?

hwhsu1231 commented 1 year ago

Excuse me, @cristianadam.

Do you understand what @memsharded was talking about? I think the integration with cmake-conan that QtCreator proposed is great and feasible. However, @memsharded said that this only works with Qt, but doesn't work with the majority of Conan cmake-conan users.

Why?

hwhsu1231 commented 1 year ago

If I understand correctly, what @memsharded insisted is that:

But what I'm saying is that:

hwhsu1231 commented 1 year ago

@memsharded @jcar87 @czoido @prince-chrismc

Hello Conan Team. Can you give some responses, please?

memsharded commented 1 year ago

Sorry, we are busy, things take time and there are many ongoing parallel priorities, no need to rush the team, we already have this in our agendas, please be patience.