[bug] `CMakePresets` relative paths break out of build CMake usage

[nyibbang]

Describe the bug

Since Conan 2.3.0 and merge of the https://github.com/conan-io/conan/pull/16015 feature pull request, the CMakePresets.json file uses relative paths to the CMake toolchain file.

I'm not sure what the reasoning for this change was, but it introduced a regression when using CMake with presets and an out of build configuration (with -S -B arguments), and CMake now fails to find the toolchain file.

How to reproduce it

Here is a small example of a project that reproduces this issue with conan 2.3.0

conanfile.txt

[requires]
zlib/1.2.11

[generators]
CMakeToolchain
CMakeDeps

[layout]
cmake_layout

CMakeLists.txt

cmake_minimum_required(VERSION 3.28)
project(ConanRegression)

find_package(ZLIB)

build.sh

mkdir cmake-build && cd cmake-build
conan install ..
cmake -S .. -B . --preset conan-release

Result

When we execute the build.sh script, we get the following CMake error:

Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  CMAKE_POLICY_DEFAULT_CMP0091="NEW"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="generators/conan_toolchain.cmake"

CMake Error at /<cmake-path>/Modules/CMakeDetermineSystem.cmake:152 (message):
  Could not find toolchain file: generators/conan_toolchain.cmake
Call Stack (most recent call first):
  CMakeLists.txt:2 (project)

CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!

[memsharded]

Hi @nyibbang

Thanks for your report.

It seems there would be some inconsistency in this point:

cmake -S .. -B . --preset conan-release

This is defining a different build folder than the one that the cmake_layout is defining, which would be a build folder, and that is causing the error.

If you type

cmake -S ..  --preset conan-release

It will work.

This is assumed as a pre-condition for cmake_layout. If you want to change the build folder, is also possible, but it is needed to specify it in the conan install command with tools.cmake.cmake_layout:build_folder configuration :

conan install .. -c tools.cmake.cmake_layout:build_folder="<path/to>/cmake-build"
cmake -S .. -B . --preset conan-release

This will work. That means, that the conan install build folder and the cmake build folder must be the same.

[nyibbang]

Well, this is unfortunate.

My use case is this one: I have a C++ project that uses pybind11 to create a Python module, and I want to build that project as a wheel. I use CMake + Conan to build this project.

I have 2 workflows for the build system:

1. I want to build the project for development, testing and debugging. No Python wheel is built, only the binaries so that I can run my C++ and Python unit tests. In that case, I want to use the cmake_layout.
2. I want to build the project for distribution, so I need to build Python wheels. I build a wheel for each version of Python 3.7+ and I even cross build for ARM64. So what I do is that I create a conan install configuration once for each platform where I build each dependency from sources (because I use a manylinux environment which often comes with older GLIBC versions that the ones used to make prebuilt binaries available on conancenter) and I then reuse this installation for every version of Python I want to build for. This ensures that I have to build the dependencies only once (as they do not depend on the version of Python), and then I build my project from these dependencies many times.

Now if that preconditions exists for cmake_layout, I cannot use it in my second use case. And indeed, since I upgraded from Conan 2.3+, my build started failing because of the change in behavior I described.

Would there be a way to disable the cmake_layout with a configuration option from the conan install command line ?

[memsharded]

Now if that preconditions exists for cmake_layout, I cannot use it in my second use case. And indeed, since I upgraded from Conan 2.3+, my build started failing because of the change in behavior I described.

I am not sure I understand the reasons why this won't work for this use case. If you can please clarify why the -c tools.cmake.cmake_layout:build_folder won't work, that would help. Note that there are also even more parametrization for the build folders, using the -c tools.cmake.cmake_layout:build_folder_vars it is possible to parameterize the build folder for different settings and options (and soon constants too), so multiple builds for multiple configurations (like different architectures) will be automatically managed. If glibc is a setting in your case, then it can automatically map to different folders at conan install time.

[nyibbang]

The reason it wouldn't work is that I use conan install call once per platform to generate a CMake toolchain and FindDeps.cmake files and also build dependencies from sources into the Conan cache. Each platform/architecture is a new build environment so I don´t need to use build_folder_vars for that, since my Conan build folders are not shared through the different platforms (environments are handled by the CI tool that I use, which are docker containers).

However my Conan build folders are shared through multiple versions of Python. My dependencies do not depend on Python, so I do not need to rebuild them (I can build them once for the platform and that's it). I suppose I could have cmake_layout generate a list of build folder for each version of Python, by tweaking build_folder_vars to include the version of Python somehow. But that's not very satisfying to me because I did not want to handle this through my conanfile. In this project, I only use Conan to fetch and build my dependencies, and then everything is built through CMake that is directly called through a setuptools module with the correct arguments. This module is itself called once for each of the Python version I want to build for and everything is built in isolation (although they can access the Conan cache and my project Conan build folder).

I think I need to find a way to build and install dependencies in the cache but ignore anything that gets generated in my build folder (conan install --build=* ...), and then recreate a new build each time the setuptools module builds my project for a specific version of Python (conan install ...).

[memsharded]

I think I need to find a way to build and install dependencies in the cache but ignore anything that gets generated in my build folder (conan install --build=* ...), and then recreate a new build each time the setuptools module builds my project for a specific version of Python (conan install ...).

Yes, I might still be missing something, but in general there are 2 distinct flows here, and the flow of extracting artifacts from Conan packages to use them in other technology such as .deb debian packages or creating a Windows installer is by first having the Conan packages there, then doing something like conan install --requires=mypkg/1.0 --deployer=mydeployer.

Because when doing those wheels it is not only necessary the current binaries, but also if the transitive dependencies have shared libraries, it is necessary to "collect" them, extract them from the Conan cache and put them inside the wheel (in user folder). This is typically done with deployers or maybe even with a Conan custom command that further automates the creation of the installer.

This way the problem of the package creation, local development, etc is decoupled from the "deployment" flow. Maybe for this case, you might need to do some changes, but something like this could make sense:

• Move the "consumer" conanfile.txt to a conanfile.py that allows to create the "final" binary artifact that you are building, maybe a shared library?
• Create that consumer package as yet another package
• Use a conan install --requires=mypkg/version --deployer=mydeployer to automate the collection of compiled libraries into the folder that you want. For this step, instead of a conan install you could create your own conan myorg:create_wheel --requires=mypkg/version custom command, if you have some python automation after the deployment, it depends.
• Then the build of the wheel happens from there (no Conan or cmake really involved, because the C/C++ binaries were already built)

I don't know if this makes sense for your use case, just some ideas from the usage we see by other users.

[nyibbang]

Because when doing those wheels it is not only necessary the current binaries, but also if the transitive dependencies have shared libraries, it is necessary to "collect" them, extract them from the Conan cache and put them inside the wheel (in user folder). This is typically done with deployers or maybe even with a Conan custom command that further automates the creation of the installer.

My goal was to stay as much as possible close to the Python workflow. This step in my case is handled by Python auditwheel repair utility. It reads the shared library in the wheel and copies the dependencies from the Conan cache.

Once dependencies were built with Conan, I could build my wheel just by running pip build -c ... and some args to specify the CMake toolchain file generated by Conan CMakeToolchain generator. This is really convenient because everything is then handled by the setuptools module that I was talking about (scikit-build-core) that interfaces with CMake and neatly places all libraries where they belong in the wheel archive, while generating all the metadata from the CMake install.

And because that workflow was really close to a more simple wheel project, I could use other tools such as cibuildwheel that automates the creation of the wheels for each platform and Python version that I target.

I would prefer staying in that workflow if possible, so I'll try to find a way to work out with this change in Conan. I'll probably just ignore the presets and directly target the toolchain file.