search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.17k stars 974 forks source link

[question] preexistend toolchain file #15927

Open schifazl opened 6 months ago

schifazl commented 6 months ago

What is your question?

Disclaimer: I'm still a newbie both in CMake and Conan.

I'm working on migrating an embedded project to CMake + Conan.

In the first stage, I have created the CMakeLists, CMakeUserPresets, and gcc-toolchain.cmake files (without using Conan).

The gcc-toolchain files contain everything that's compiler-related and could be extracted from the project: the compiler to be used and the compiler flags when compiling and linking. The idea is to create one toolchain file for each compiler we're using and choose the more suitable one on a per-project basis.

Then I tried to extract some ARM headers (CMSIS5 stuff) in a Conan package and consume it from the main project. I was able to create it, but not to consume it, as the compiler isn't finding the header files (I'm using find_package() and target_link_libraries(), but the path isn't passed when invoking GCC).

But maybe before that, the problem is the Conan vs. custom toolchain files:

So already having a toolchain file, I can't use the one generated by Conan. But well... since in the conanfile.py we can choose to not generate the Conan toolchain file, I assumed that it isn't strictly necessary... or it is? The documentation states that the Conan toolchain file can be not created, so I assume that it isn't mandatory, yet I can't find in the documentation the info on how to do things without the Conan toolchain file. This confuses me a lot.

I won't write regarding the other things I tried to do, since I didn't really know what I was doing. For the moment, I'll just ask if someone can point me in the right direction regarding the toolchain files.

To summarize: can I use a Conan dependency without the Conan toolchain file? If not, is there a way to put "somewhere" my actual toolchain file so that it can be used with Conan?

I'm sorry for the confused post, it reflects the confusion I have in my head 😅

Have you read the CONTRIBUTING guide?

jcar87 commented 6 months ago

Hi @schifazl - thank you for asking this question.

To summarize: can I use a Conan dependency without the Conan toolchain file? If not, is there a way to put "somewhere" my actual toolchain file so that it can be used with Conan?

You can actually do both, although I would recommend the method where the Conan-generated conan_toolchain.cmake, chain-loads your custom toolchain.

You can use the tools.cmake.cmaketoolchain:user_toolchain conf to specify a custom, use-provided toolchain - then conan_toolchain.cmake will include this directly. You can specify this either:

Let me know if this helps or if you need any more details.

Depending on the complexity of your custom gcc-toolchain.cmake, you may be able to specify the same information in the Conan profile directly, with the added advatange that Conan is able to propagate that information when building things in other build systems as well (e.g. autotools, meson) - whereas a custom gcc-toolchain.cmake only works with CMake. Although wether you would benefit from this depends on your needs :)

schifazl commented 5 months ago

Thanks and sorry for the late reply, but I had other stories to close and just now returned on this issue 😅

I'm trying to use the [conf] method (eventually I think that I'll move the toolchain to a separate Conan recipe as per your third suggestion, but one thing at a time), but I'm having difficulties:

  1. In the conf file I put tools.cmake.cmaketoolchain:user_toolchain=["gcc-toolchain.cmake"]
  2. I launch conan install . --output-folder=build --build=missing --profile=./conan.conf ; cmake --preset=release
  3. Conan generates the build directory with the conan_toolchain.cmake file containing the include("gcc-toolchain.cmake") line
  4. Obviously CMake doesn't find the provided toolchain file (because it's in the root directory, not in the build subdirectory) and it complains.

I tried to change the conf line in various ways, like ../gcc-toolchain.cmake or prepending it with various CMake (and Conan) standard variables, like CMAKE_SOURCE_DIR, or the jinja2 method with absolutely no luck. The only way to make it work is by providing the absolute path to the gcc-toolchain.cmake file.

Is this intended behavior or it's just me that I'm a total noob both in Conan and in CMake? 😅

memsharded commented 5 months ago

Hi @schifazl

Indeed In the conf file I put tools.cmake.cmaketoolchain:user_toolchain=["gcc-toolchain.cmake"] with a relative path will not work. It is necessary to provide the full path.

In profile jinja syntax, specially when the conf is provided in a profile, with the profile_dir variable, see https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/inject_cmake_variables.html

[conf]
tools.cmake.cmaketoolchain:user_toolchain+={{profile_dir}}/myvars.cmake

In general, adding things to profile files and managing the profiles with conan config install would be recommended, easier to maintain, reproduce and share than relying on passing many command line arguments.

schifazl commented 5 months ago

Thanks, it worked! 😄

In general, adding things to profile files and managing the profiles with conan config install would be recommended, easier to maintain, reproduce and share than relying on passing many command line arguments.

I'm not sure here what are you suggesting... replacing my toolchain file with variables in the conan.conf files, or replacing the --output-folder=build --build=missing parameters?

memsharded commented 5 months ago

I'm not sure here what are you suggesting... replacing my toolchain file with variables in the conan.conf files, or replacing the --output-folder=build --build=missing parameters?

I am suggesting not passing conf in the command line like -c tools.cmake.cmaketoolchain:user_toolchain=... and probably not adding that to the global.conf file either.

That kind of configuration can be added to the [conf] section of a profile file. Note that the name conan.conf for a profile, like you suggested above in --profile=./conan.conf is not recommended, as the .conf extension is other concept. I'd recommend using profiles with .txt or .profile or even without extension.

The the --output-folder=build can be ommitted if using layout() (or [layout] in conanfile.txt), making the user command simpler and shorter to type.

schifazl commented 5 months ago

Great thanks, I applied your suggestions!

I had some trouble compiling my project using a header-only Conan package (made by me).

Using the recommended method for Conan 2, the compilation doesn't work (the header dir is not added to the compiler command with the -I switch:

find_package("cmsis5")
target_link_libraries(${PROJECT_NAME} PUBLIC cmsis5::cmsis5)

It now works by adding these lines in the CMakeLists.txt file:

find_package("cmsis5")
include_directories(${cmsis5_INCLUDE_DIRS})
target_link_libraries(${PROJECT_NAME} ${cmsis5_LIBRARIES})

But... This is from Conan 1.64 guide. Is it still OK?

Edit: It's enough to use

find_package("cmsis5")
include_directories(${cmsis5_INCLUDE_DIRS})
memsharded commented 5 months ago

Hi @schifazl

include_directories(${cmsis5_INCLUDE_DIRS})

No, this shouldn't be necessary, and even if it is a header-only library, the recommended way is target_link_libraries(${PROJECT_NAME} PUBLIC cmsis5::cmsis5), and it should be enough.

I'd say it is possible that you are missing some detail in your recipe, like in Conan 2 package_type = "header-library" is strongly recommended, or maybe there is something else missing in the recipe. If you want to share it, we could have a look.

schifazl commented 5 months ago

Sure, this is the library recipe:

from conan import ConanFile
from conan.tools.files import copy

class cmsis5Recipe(ConanFile):
    name = "cmsis5"
    version = "0.1.0"
    package_type = "header-library"

    # Optional metadata
    license = "<Put the package license here>"
    author = "<Put your name here> <And your email here>"
    url = "<Package recipe repository url here, for issues about the package>"
    description = "<Description of cmsis-5 package here>"
    topics = ("<Put some tag here>", "<here>", "<and here>")

    # No settings/options are necessary, this is header only
    exports_sources = "include/*"
    # We can avoid copying the sources to the build folder in the cache
    no_copy_source = True

    def package(self):
        # This will also copy the "include" folder
        copy(self, "*.h", self.source_folder, self.package_folder)

    def package_info(self):
        # For header-only packages, libdirs and bindirs are not used
        # so it's necessary to set those as empty.
        self.cpp_info.bindirs = []
        self.cpp_info.libdirs = []

        # Generate both MyFileNameConfig.cmake and FindMyFileName.cmake
        self.cpp_info.set_property("cmake_find_mode", "both")

And this is the recipe of the main project:

from conan import ConanFile

class S300Recipe(ConanFile):
    # Binary configuration
    settings = "compiler", "build_type", "arch"
    generators = "CMakeDeps", "CMakeToolchain"

    def requirements(self):
        self.requires("cmsis5/0.1.0")

    def layout(self):
        self.folders.build = "build"
        self.folders.generators = "build"
memsharded commented 5 months ago

Uhm, I am not sure what is happening. Maybe it is worth adding in the consumer the package_type = "application" or library/shared-library/static-library, but I am not sure that would be the main issue. Maybe the settings = "os" that is missing could also be.

I have tried to put it as a full example, that can be tested with conan create . and it seems to work: example.zip Maybe if you can put a fully reproducible example including the CMakeLists.txt and simple source files, that could clarify it.

schifazl commented 5 months ago

I prepared a test with just the main .c file and the dependency (conanfile buried in ra\arm\CMSIS_5\CMSIS\Core ). It doesn't link, but I think this is not important for the purpose of this issue.

I create the dependency with conan create .\ra\arm\CMSIS_5\CMSIS\Core --profile:all .\conan.profile and the main project with remove-item build -recurse ; conan install . --profile:all=./conan.profile ; cmake --preset=release ; cmake --build --preset=release

You'll have to change the TOOLCHAIN_PREFIX variable in the CMakePresets file, I presume.

conan-minimal-example.zip

memsharded commented 5 months ago

I am trying to reproduce, but I don't have this compiler installed "TOOLCHAIN_PREFIX": "C:/Program Files (x86)/Arm GNU Toolchain arm-none-eabi/12.2 mpacbti-rel1", so this would be difficult. Isn't it possible to reproduce it without such specific compiler? It might be possible to reproduce it with the standard msvc compiler?

Adding full logs to the repo that prints the full output and error might help.

A quick possible hint is that when using a specific generator as Ninja Multi-Config it is necessary to tell Conan, with -c tools.cmake.cmaketoolchain:generator="Ninja Multi-Config", so things are aligned.

schifazl commented 5 months ago

I am trying to reproduce, but I don't have this compiler installed "TOOLCHAIN_PREFIX": "C:/Program Files (x86)/Arm GNU Toolchain arm-none-eabi/12.2 mpacbti-rel1", so this would be difficult. Isn't it possible to reproduce it without such specific compiler? It might be possible to reproduce it with the standard msvc compiler?

I don't know 😅 I have to try, never used msvc for embedded work...

Adding full logs to the repo that prints the full output and error might help.

Successful compilation (when using target_include_directories):

PS C:\work\src\myproject> remove-item build -recurse ; conan install . --profile:all=./conan.profile ; cmake --preset=release ; cmake --build --preset=release

======== Input profiles ========
Profile host:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

Profile build:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

======== Computing dependency graph ========
Graph root
    conanfile.py: C:\work\src\myproject\conanfile.py
Requirements
    cmsis5/0.1.0#c5711ac2ca3283714557f93b4bd51fa7 - Cache

======== Computing necessary packages ========
Requirements
    cmsis5/0.1.0#c5711ac2ca3283714557f93b4bd51fa7:da39a3ee5e6b4b0d3255bfef95601890afd80709#c3c71f623eb728469d6afcc66453dfaa - Cache

======== Installing packages ========
cmsis5/0.1.0: Already installed! (1 of 1)

======== Finalizing install (deploy, generators) ========
conanfile.py: Writing generators to C:\work\src\myproject\build
conanfile.py: Generator 'CMakeDeps' calling 'generate()'
conanfile.py: CMakeDeps necessary find_package() and targets for your CMakeLists.txt
    find_package(cmsis5)
    target_link_libraries(... cmsis5::cmsis5)
conanfile.py: Generator 'CMakeToolchain' calling 'generate()'
conanfile.py: CMakeToolchain generated: conan_toolchain.cmake
conanfile.py: Preset 'conan-default' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-default' if using CMake>=3.23
conanfile.py: If your CMake version is not compatible with CMakePresets (<3.23) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=C:\work\src\myproject\build\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW'
conanfile.py: CMakeToolchain generated: CMakePresets.json
conanfile.py: CMakeToolchain generated: ..\CMakeUserPresets.json
conanfile.py: Generating aggregated env files
conanfile.py: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully
Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="conan_toolchain.cmake"
  TOOLCHAIN_PREFIX="C:/Program Files (x86)/Arm GNU Toolchain arm-none-eabi/12.2 mpacbti-rel1"

-- Using Conan toolchain: C:/work/src/devops/s300-ufo/build/conan_toolchain.cmake
-- The C compiler identification is GNU 12.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Arm GNU Toolchain arm-none-eabi/12.2 mpacbti-rel1/bin/arm-none-eabi-gcc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Conan: Target declared 'cmsis5::cmsis5'
-- Found cmsis5: 0.1.0 (found version "0.1.0")
-- Configuring done (5.3s)
-- Generating done (0.2s)
-- Build files have been written to: C:/work/src/devops/s300-ufo/build
[18/43] Building C object CMakeFiles/myproject.dir/Debug/src/view-state-manager.c.obj
C:/work/src/devops/s300-ufo/src/view-state-manager.c: In function 'ViewStateUpdate':
C:/work/src/devops/s300-ufo/src/view-state-manager.c:182:25: warning: assignment discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers]
  182 |                 actions = ViewGetActionTable(globalViewState.nextView);
      |                         ^
[43/43] Linking C executable Debug\myproject.out

Unsuccessful compilation (when using target_link_libraries):

======== Input profiles ========
Profile host:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

Profile build:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

======== Computing dependency graph ========
Graph root
    conanfile.py: C:\work\src\myproject\conanfile.py
Requirements
    cmsis5/0.1.0#c5711ac2ca3283714557f93b4bd51fa7 - Cache

======== Computing necessary packages ========
Requirements
    cmsis5/0.1.0#c5711ac2ca3283714557f93b4bd51fa7:da39a3ee5e6b4b0d3255bfef95601890afd80709#c3c71f623eb728469d6afcc66453dfaa - Cache

======== Installing packages ========
cmsis5/0.1.0: Already installed! (1 of 1)

======== Finalizing install (deploy, generators) ========
conanfile.py: Writing generators to C:\work\src\myproject\build
conanfile.py: Generator 'CMakeDeps' calling 'generate()'
conanfile.py: CMakeDeps necessary find_package() and targets for your CMakeLists.txt
    find_package(cmsis5)
    target_link_libraries(... cmsis5::cmsis5)
conanfile.py: Generator 'CMakeToolchain' calling 'generate()'
conanfile.py: CMakeToolchain generated: conan_toolchain.cmake
conanfile.py: Preset 'conan-default' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-default' if using CMake>=3.23
conanfile.py: If your CMake version is not compatible with CMakePresets (<3.23) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=C:\work\src\myproject\build\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW'
conanfile.py: CMakeToolchain generated: CMakePresets.json
conanfile.py: CMakeToolchain generated: ..\CMakeUserPresets.json
conanfile.py: Generating aggregated env files
conanfile.py: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully
Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="conan_toolchain.cmake"
  TOOLCHAIN_PREFIX="C:/Program Files (x86)/Arm GNU Toolchain arm-none-eabi/12.2 mpacbti-rel1"

-- Using Conan toolchain: C:/work/src/myproject/build/conan_toolchain.cmake
-- The C compiler identification is GNU 12.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Arm GNU Toolchain arm-none-eabi/12.2 mpacbti-rel1/bin/arm-none-eabi-gcc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Conan: Target declared 'cmsis5::cmsis5'
-- Found cmsis5: 0.1.0 (found version "0.1.0")
-- Configuring done (1.7s)
-- Generating done (0.1s)
-- Build files have been written to: C:/work/src/myproject/build
[1/43] Building C object CMakeFiles/myproject.dir/Debug/src/appmain.c.obj
FAILED: CMakeFiles/myproject.dir/Debug/src/appmain.c.obj
C:\PROGRA~2\ARMGNU~1\12977F~1.2MP\bin\AR19DD~1.EXE -D_GLIBCXX_USE_CXX11_ABI=0 -DCMAKE_INTDIR=\"Debug\" -IC:/work/src/myproject/src -IC:/work/src/myproject/ra/fsp/inc -IC:/work/src/myproject/ra/fsp/inc/api -IC:/work/src/myproject/ra/fsp/inc/instances -IC:/work/src/myproject/ra_gen -IC:/work/src/myproject/ra_cfg/fsp_cfg/bsp -IC:/work/src/myproject/ra_cfg/fsp_cfg -mcpu=cortex-m23 -mthumb -O0 -fmessage-length=0 -fsigned-char -ffunction-sections -fdata-sections -Wunused -Wuninitialized -Wall -Wextra -Wmissing-declarations -Wconversion -Wpointer-arith -Wshadow -Wlogical-op -Waggregate-return -Wfloat-equal -g -gdwarf-4 -D_RENESAS_RA_ -D_RA_CORE=CM23 -std=c99 -MMD -MP  -Og -g -MD -MT CMakeFiles/myproject.dir/Debug/src/appmain.c.obj -MF CMakeFiles\myproject.dir\Debug\src\appmain.c.obj.d -o CMakeFiles/myproject.dir/Debug/src/appmain.c.obj -c C:/work/src/myproject/src/appmain.c
In file included from C:/work/src/myproject/ra/fsp/inc/api/bsp_api.h:47,
                 from C:/work/src/myproject/ra_gen/hal_data.h:5,
                 from C:/work/src/myproject/src/hal.h:11,
                 from C:/work/src/myproject/src/appmain.c:1:
c:\work\src\myproject\ra\fsp\src\bsp\cmsis\device\renesas\include\renesas.h:42:11: fatal error: cmsis_compiler.h: No such file or directory
   42 |  #include "cmsis_compiler.h"
      |           ^~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

A quick possible hint is that when using a specific generator as Ninja Multi-Config it is necessary to tell Conan, with -c tools.cmake.cmaketoolchain:generator="Ninja Multi-Config", so things are aligned.

This gives me:

======== Input profiles ========
Profile host:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:generator=Ninja Multi-Config
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

Profile build:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

======== Computing dependency graph ========
Graph root
    conanfile.py: C:\work\src\myproject\conanfile.py
Requirements
    cmsis5/0.1.0#c5711ac2ca3283714557f93b4bd51fa7 - Cache

======== Computing necessary packages ========
Requirements
    cmsis5/0.1.0#c5711ac2ca3283714557f93b4bd51fa7:da39a3ee5e6b4b0d3255bfef95601890afd80709#c3c71f623eb728469d6afcc66453dfaa - Cache

======== Installing packages ========
cmsis5/0.1.0: Already installed! (1 of 1)

======== Finalizing install (deploy, generators) ========
conanfile.py: Writing generators to C:\work\src\myproject\build
conanfile.py: Generator 'CMakeDeps' calling 'generate()'
conanfile.py: CMakeDeps necessary find_package() and targets for your CMakeLists.txt
    find_package(cmsis5)
    target_link_libraries(... cmsis5::cmsis5)
conanfile.py: Generator 'CMakeToolchain' calling 'generate()'
conanfile.py: CMakeToolchain generated: conan_toolchain.cmake
conanfile.py: Preset 'conan-default' added to CMakePresets.json. Invoke it manually using 'cmake --preset conan-default' if using CMake>=3.23
conanfile.py: If your CMake version is not compatible with CMakePresets (<3.23) call cmake like: 'cmake <path> -G "Ninja Multi-Config" -DCMAKE_TOOLCHAIN_FILE=C:\work\src\myproject\build\conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW'
conanfile.py: CMakeToolchain generated: CMakePresets.json
conanfile.py: CMakeToolchain generated: ..\CMakeUserPresets.json
conanfile.py: Generating aggregated env files
conanfile.py: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully
CMake Error: Could not read presets from C:/work/src/myproject:
"configuration" expected a string, got:
"configuration" expected a string, got:
CMake Error: Could not read presets from C:/work/src/myproject:
"configuration" expected a string, got:
"configuration" expected a string, got:
memsharded commented 5 months ago

Some important thing:

Profile build:
[settings]
arch=armv8.3
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=13.2
os=baremetal
[conf]
tools.cmake.cmaketoolchain:user_toolchain=['C:\\work\\src\\myproject/gcc-toolchain.cmake']

This is an incorrect "build" profile. The build profile will be your current machine, it will be os=Windows/Linux/Macos in most cases, not baremetal. And same with arch and other settings, they must align with your current machine, typically something like the conan profile detect

schifazl commented 5 months ago

OK, that was just a recent addition when doing my tests, I will remove it.

I read this page: https://docs.conan.io/2/tutorial/consuming_packages/cross_building_with_conan.html

If I understood correctly, this wouldn't affect the usability of the created package? I mean: if I cross-compile my source code on my CI environment (which uses Linux), it will still be usable from the developer's Windows machine, because the build profile doesn't affect the package id. It will just be used when deciding when using the requirements and build_requirements from the conanfile.py recipe. Is this correct?

memsharded commented 5 months ago

If I understood correctly, this wouldn't affect the usability of the created package?

It might easily break the build and don't build correctly. Yes, the package_id is independent of the "build" context in general, but the binary might not be correct if for some reason the "build" context is not correctly configured.